The fall season of the Metaculus AI forecasting tournament (now called Future Bench) wrapped up recently, so I wanted to take some time to walk through the results and see if there is anything to be learned. I’m doing this in two parts, with this post covering the two-week MiniBench tournaments and the next covering the main fall tournament once those questions finish resolving.

For context, the MiniBench tournaments are short, two-week tournaments with automatically generated questions. This is in contrast to typical forecasting tournaments (including the main fall tournament) which span several months at minimum. The hope is that these tournaments provide a faster feedback cycle for forecasting bot development, at the cost of using less interesting questions.

I’ve previously done analysis of individual MiniBench tournaments, and much of that analysis is consistent with what I found from the full season. Because of that, I’ll mostly focus on some bigger picture things that stood out when looking across the tournaments. This post is long, so I’ve also tried to prioritize the more interesting things in the main body and push off some additional analyses in an appendix at the end for people who are extra curious.

Subscribe now

Overall Performance

I participated in 7 MiniBench tournaments, and my performance was better than I expected but highly variable. Performance is measured using ‘peer score,’ which is a modified log score with a scale that accounts for the performance of other participants. A positive peer score simply means ‘outperforms the mean,’ not necessarily ‘made good forecasts’ as the score will be influenced by others doing poorly (or well). In some places I use a ‘baseline score’, which is the same score without the peer scaling applied. This is easier to interpret, but doesn’t have a way to account for question difficulty (that is, you don’t know whether a good score comes from a solid prediction or an easy question).

Here I’m showing the total peer score for each tournament summed across all questions, which is the score used to judge tournament results. The label at the bottom indicates my relative finish in each tournament. Across tournaments my median finish was 9th, and the number of participants in each tournament ranged from 31 to 45 (not counting Metaculus internal bots) with a median of 41.

As far as the trend goes, it seems like I was making generally positive progress with some catastrophic backsliding in late November (I’ll have more to say about this later). All peer scores here were positive, indicating I outperformed the ‘mean’ bot. Baseline scores were also positive (shown in the appendix), meaning regardless of how easy the questions were at the very least my bot was not guessing randomly!

Comparing peer score to a 0 value is not all that informative, because this scaling can be heavily skewed by a couple very poorly performing bots. This means we might end up in a situation where most bots outperform the ‘mean’ bot, and in fact that does seem to happen. From eyeballing leaderboards, it looks like ~70% of bots have a positive peer score on any given tournament. Much like the children of Lake Wobegone, all the bots are above average!

A better metric is to compare to the community aggregate prediction. The community prediction is the weighted median of all the submitted predictions for a given question. This is like a ‘wisdom of crowds’ approach, and represents the consensus view across all bots. Because it’s using a median, it’s less vulnerable to a few outlier predictions making everyone else look good (or bad). In general, the community predictions perform much better than most individual forecasters, and it usually finishes inside the top 10 in any given tournament, so it’s a nice target to aim for if trying to determine whether a bot is ‘good’.

Here I’ve recreated the plot from above but added reference lines for both the community aggregate prediction (green) and the top performing bot in each tournament (red). I outperformed the community prediction in 4/7 tournaments, with the biggest shortfall in the first tournament (not surprising, as it took me some time to get the bot working at all). I’m generally satisfied with these results, but think there are some obvious issues that could be resolved to get better performance.

Bot Changes and Impact

I made three rounds of meaningful updates across the course of this tournament, which means I have three different comparison points for evaluating performance. Not enough for a rigorous reading (that’s what the experiment posts are for), but enough to get some sense of whether things are working.

Update 1 (10/4): I made three changes, two minor and one major. These changes were largely inspired by my first MiniBench analysis where I found that bots were generally too conservative and some of my worst errors were due to total hallucinations.

• Minor: Pushed the forecaster model to make more aggressive forecasts (via prompt), and changed the aggregation approach to use mean instead of median (which should also make forecasts slightly more extreme by preserving outliers).

• Major: Changed the news search from gpt-4o to gpt-5-mini, built out a much more detailed researcher prompt, and allowed the model to agentically search the web to address the research questions. Of all the changes I made to my bot over this tournament, in hindsight this was by far the most (positively) impactful. I would later find that that 4o was among the worst models to use and 5-mini among the best, but at the time this was a lucky guess.

Update 2 (10/26): Updated the context provided as part of the prompts with some meta knowledge about forecasting. The intent behind this change was to try and give the model more opportunity to find the correct numbers from web-search, as I was seeing that I still got major failures from the model pulling a completely wrong baseline value. I accomplished this through adding few-shot examples to the prompt. Here’s an example of the kind of thing that was was added:

“IMPORTANT: Often, binary questions have a threshold set very close to the current value. So if you find a value in your research that is significantly off from this threshold value, you should double check that to be sure.

Example: “The question asks whether the community prediction on Metaculus will be higher than 10% in 7 days, but I found the community prediction was currently at 30%. This is much higher than the 10% threshold, so I should search again to be sure I have the right number”

Update 3 (11/25): In order to make my bot more flexible for some other experiments I had planned, and because based on this experiment I wanted to allow for some more variable architectures, I decided to do a complete refactor that included taking over many of the things that were previously delegated to the forecasting_tools package and rebuilding my existing bot inside the new framework.

In hindsight, doing this the week of Thanksgiving was a mistake, as I did not test this refactor thoroughly enough and ended up with a lot of dumb technical issues that really tanked my performance that week and were hard to handle while traveling.

At the same time I implemented a major structural change via a fully end-to-end forecaster with a single research + forecast model call and agentic search as the primary approach. This approach was far slower and much more expensive, so I did not use any aggregation across multiple runs. The results were not encouraging as it turned in my two worst MiniBench performances outside of the first week.

Interestingly the median answer (as opposed to the mean) was quite a bit better using this approach (at least, once the technical issues were fixed in time for the December 8th tournament). This suggests the model was most often making good predictions, but with some punishing failures. Because the scoring rule used here is asymmetric and far more punishing of wrong answers (minimum score -897 for binary questions) than rewarding of right answers (maximum score +99.9 for binary questions), this can be highly net negative.

This is pretty much what I found after looking into it a bit more. The fraction of questions which I labeled as ‘major errors’ (score lower than -50) was significantly higher using the new single model approach. Even though the model was generally making better predictions, the overconfidence was punished strongly enough that it erased any potential gains and resulted in worse overall performance.

Update 4 (current): Based on those last couple weeks of results, I moderated my bot to use a mixture of the previous and new strategies. I think the confident end-to-end approach has a lot of potential, but I need to moderate that confidence a bit so as not to end up punished too harshly for incorrect predictions.

Performance Across Question Types

I’d previously found that my performance was dramatically different across the various types of questions in MiniBench (binary, multiple choice, or numeric). This tendency persisted when evaluating across all the tournaments together. In terms of peer score, my bot was barely better than the mean in binary questions (+3.3 points per question), a little better in multiple choice (+12.8 points, but with some major negative outliers), and doing quite well in the numeric questions (+28.3 points).

It’s worth pointing out that I didn’t implement any strategies specifically targeting multiple choice or numeric questions over binary questions. Nevertheless, the vast majority of my total score on any given tournament comes from out predicting the competition on those two question types.

My best explanation for this difference is that my bot is more confident than the average bot, and that this confidence is most useful on questions which are inherently predictable. Because MiniBench questions are weird, some of the questions are essentially impossible to predict (e.g., will the stock price of a random company go up or down) and some are very easy to predict (e.g., what will be the rate offered for a specific type of bond in two weeks). It just so happens that these predictable vs. unpredictable questions map almost perfectly onto the broader question types. Almost all binary questions are difficult, and almost all numeric questions are easy.

To share one illustrative example of what I’m talking about, the numeric questions in MiniBench generally have to do with predicting market rates for various financial instruments that change very slowly. Here’s an example question:

And here is the relevant data for the month of September from the St. Louis Fed:

This question is highly predictable (check the y axis here). The total range in variance over any two-week period is on the order of ~20bp (basis points) while the range for prediction is on the order of ~200bp. In principle estimating these with very high accuracy is big business in the financial world, but in practice for this tournament I suspect that most bots are so conservative that even a minor increase in confidence is enough to get significant gains.

To validate this I aggregated all of the probability distributions onto a common scale aligned to the true resolution value, plus 30 bins to either side of this value.1 This gives something like the ‘average’ prediction across questions without worrying about the specific values being predicted. Both my predictions (blue) and community predictions (orange) are well aligned with the actual resolution value, both peaking at or very near that value. This means that both my and the community correctly predict the most likely outcome. But, because my bot was more confident (a higher peak), I end up scoring well on those questions.

I did several more detailed analyses of these different question types, which I’ve put in the appendix for those who are interested. My general takeaway is that the many of the binary questions are nearly impossible to predict (even in principle) which causes the numeric and multiple choice questions to drive a much larger portion of the final tournament results. I would be curious to learn whether this is the case for other bots, or whether there are some bots that actually succeed at scoring a meaningful amount from binary questions.

Sources of Error

Because errors are so punishing, it’s worth taking some time to evaluate questions which scored particularly poorly to see if there are any common trends that could be addressed.

Of the 25 worst performing questions, 17 were multiple choice questions and 8 were binary (all Metaculus predictions)

For the multiple choice Google trends questions:

• 14 were cases where the result was “doesn’t change” due to floor effects my bot did not adequately address as a possibility (more on this in the appendix).

• 3 were cases where the trend (possibly predictably) increased due to news that wasn’t anticipated.

For the 8 binary Metaculus meta-questions

• 5 were before I changed my search approach, and all of these were incorrectly reading the current Metaculus values.

• 3 were correct interpretations of the current values, and had last second movements against the predicted direction. In each of these cases the final values were extremely close to the threshold, and I don’t see anything obvious to adjust.

So of these 25 failures, 5 seem to have been adequately addressed by subsequent changes, 6 seem like reasonable predictions that happened to score poorly, and 14 deal with a highly specific edge case that relates to how google trends are measured. These 14 edge cases could be dealt with, but I have mixed feelings about making a change that narrowly addresses a weird quirk of this question type (essentially, a floor effect that makes one option invalid) that wouldn’t generalize to better prediction overall.

Summary

This fall was the first time I participated in a forecasting tournament of any kind, and it went better than I expected. Coming in with next to no forecasting or LLM tool development experience I expected to consistently bring up the back of the pack, but instead ended up around the top quartile. This is nothing to brag about, exactly, but it is encouraging and I hope anyone reading this without much background will take it as motivation to jump into these tournaments.

It was also an excellent learning experience, largely because of these MiniBench tournaments. The big advantage of MiniBench is that it’s fast. I could make changes and evaluate them on a roughly 2 week time frame. That’s fast enough to learn some important things while still keeping in the groove of making changes. It’s also motivating, because even making an ambitious (but ultimately catastrophic) change will only set you back for two weeks rather than ruining your entire months-long tournament performance. I think without MiniBench I would have been much less invested in the main tournament and building out forecasting bots in general, so from an engagement perspective it seems like a huge win to me.

There are some problems with MiniBench which may cause me to de-emphasize it in the future. The main problem is that it’s extremely vulnerable to optimizing for the wrong things that would not generalize to the main tournament or other forecasts. This doesn’t even need to happen intentionally. For example, if someone were to make changes based purely on feedback from the overall tournament score, this might push them towards changes that optimize for the highly predictable financial style questions. These questions have little in common with the types of forecasts we generally care about (and, frankly, would be much better answered by a simple financial model than anything AI driven).

There are some other issues as well. A major concern with AI forecasting is that it can piggyback off of human forecasters posting their thoughts on the internet. This makes them perform very well on questions which already have existing forecasts, but offers no insight at all into how they would perform on new questions. MiniBench is particularly vulnerable to this for some questions (e.g., the Metaculus based ones) where a bot could be successful by simply looking up the human forecasts and reporting them verbatim or with small changes. This might exaggerate the usefulness of some of the changes I made, like spending a bunch of effort optimizing web-search.

All that said, I think participating in these tournaments was well worth my time. It is very hard to get rapid feedback on forecasting accuracy, and without that rapid feedback it’s hard to make progress. These tournaments offered that, and without them I think my bot would be in a much worse state (or simply not exist at all). I’ll be interested to see how this compares to the main fall tournament, which I’m planning to do a similar post on in the coming weeks.

Addendum: MiniBench Changes

As far as I know, there are no announced plans to change the structure of MiniBench. I’d hesitate to make any suggestions based purely off my own analysis, which might not reflect performance of other bots. But with that caveat here are some suggestions anyways:

1. Predicting stock movement of randomly chosen stock tickers is functionally useless. These questions make up ~50% of the binary questions (25% of the total tournament) and add nothing but noise.

   1. This is a famously hard problem, for which many smart people are being paid millions of dollars a year. It is interesting in an abstract sense to know whether LLMs can do this well (and maybe they can!) but I consider it an importantly different problem from forecasting.

2. I really like the Metaculus linked questions as a concept, and think more could be done with this.

   1. I think mixing in some of these with the multiple choice (up, down, same) approach used for Google Trends would be good. Often these questions are very stable over a two week period, so not having a good ‘stay the course’ option adds a lot of noise.

   2. Frequently the source questions have many forecasters in total, but few over the most recent span (~20) likely because nothing of note has happened pertaining to that question in some time. These questions are both less likely to change meaningfully and more likely to be influenced by random chance (e.g., an existing forecast timing out of the aggregate). I’m not sure how questions are picked, but it seems worth prioritizing more topical/active questions (even at the risk of having some repeats).

3. In general I think it would be more useful to have intentionally designed but repeated questions rather than randomizing for novelty. For instance, asking if the S&P 500 will increase over 2 weeks is more relevant (and possibly predictable) than a randomly selected stock ticker. There is some risk that someone could intentionally optimize specifically around these target questions to win money, but that is already an issue with wonky formulaic questions.

4. The following suggestions are actively harmful to me but…

   1. Google Trends seem really wonky, and mostly depend on understanding the behavior of trends and question structure at a meta level rather than anything to do with the content. I don’t have a good solution here, but I’m not sure improving on these questions translates to improving generally.

   2. Numeric questions are probably overweighted (should be evaluated across other bots to be sure). I score 10x more points on these questions than on binary using the same bot, and they make up ~25% of the total questions. If other bots are similar, this might wash out much of the difference between bots and turn this into mostly a financial modeling tournament.

Appendix

Other evaluations of overall performance

The most important measure is the total peer score, as that is what is used for tournament ranking, but I also looked at the baseline score as a rough metric of general bot accuracy. These plots also show the mean scores on a per question basis (which removes any noise from tournaments having slightly different numbers of questions). Interestingly my catastrophic seeming changes that tanked my overall score in the last three tournaments were not so pronounced in the baseline score, and on a per question basis the bot accuracy seems to increase pretty steadily. These scores have a huge caveat though, as they do not account for question difficulty and it’s possible the questions were more predictable in later tournaments.

Binary Questions

Calibration is pretty good, actually better than I’d have expected given how poorly I scored on binary questions. There is plausibly a bias towards underestimating likelihood, but with a few notable exceptions (what’s going on around 35% probability?) not something that I feel needs immediate correction. The community aggregate is extremely conservative in terms of never making extreme predictions, but fairly well calibrated. I have a blog post about these priors, and this seems consistent with that.

Binary questions mostly split into three topics:

Finance (stock prices) - 99 questions

Metaculus change on an existing prediction - 89 questions

And sports - 16 questions

With only a very small number of questions that don’t fit neatly in these categories.

Here is performance on those topics:

My baseline score on predicting stock prices is 0.0! Literally flipping a coin. This is not that surprising, because if my bot was capable of accurately predicting stock prices I’d be off making a bunch of easy money instead of writing this blog. Interestingly my peer score is 2.0, which means that the population mean must be worse than chance. Maybe I should start an ‘inverse Metaculus Bot’ fund?

It would be interesting to see if anyone is actually performing well on these questions, or if all of the difference in tournament performance is coming from the more predictable numeric and multiple choice questions.

Multiple choice questions

All multiple choice questions pertain to google trends. Google trends are generally predictable, even without knowing anything about the topic content. Usually, something will be in the news for a couple days and then quickly fade from public awareness. Most of them look something like this:

There will be a massive peak in attention, followed by a rapid return to baseline levels of non-interest.

This has important consequences for making good predictions. Depending on exactly which time points are being compared, it’s almost trivial to predict the direction of change. My bot seems to understand this dynamic better than the community average bot, and places more weight on trends decreasing rather than increasing.

Here I’ve plotted the baseline rates for the multiple choice questions (how many of each question resolved in each category) along with the predicted rates for mine and the community prediction. A naive model that predicted exactly these base rates on every question would have a mean baseline score of 4.8, which is only a little worse than my 7.2. The community baseline score was 1.16, worse than forecasting purely based on the prior and barely better than random chance (score of 0).

There is another wrinkle with google trends. Because the threshold for ‘doesn’t change’ is +- 3 points, there is significant risk of a floor effect where interest cannot actually decrease lower than the starting point. This factor is not adequately appreciated by my bot, and led to many poor performances (14 of the 25 worst results) despite correctly predicting that the trend would decrease in absolute terms.

Technical Errors

There was a major technical error I had with failing to correctly scale logarithmic numeric questions, which did not affect MiniBench since none of the questions use that scale. The only remaining technical error was failing to forecast questions at all. This was rare, occurring for a total of 6 questions across all tournaments.

This seems to usually be a failure of GitHub actions. My script is set to run every 20 minutes, but occasionally GitHub will just randomly not do that (this is a known problem). I didn’t check every example, but I’ve seen this often enough to expect it to be an issue. Still, with this only happening on ~2% of questions I’m not overly worried about it.

Previously in this series: AI Capabilities Forecasts

Subscribe now

In the last post I defined four tiers of AI capability, ranging from what we have now through artificial general intelligence (AGI) and artificial superintelligence (ASI). In this post, I’ll go through what I see as the likely outcomes conditional on achieving those capability tiers. At the end I’ll combine these with the capabilities forecasts to get an unconditional estimate of each probability (i.e., how likely I think we are to reach something like that world).

My capabilities outcomes could be tied to specific numeric thresholds (e.g., AGI capable of doing 20% of white collar work). I could have done something similar for this post, for instance by tying my forecast on ‘recession due to AI disappointment’ to specific S&P 500 changes. I’ve chosen to instead present a more gestalt, general description of those outcomes. So these are less true forecasts and more like vignettes and estimates of how plausible they seem.

I did this for two reasons. First, defining specific measurable proxies for each outcome is a ton of work that would be mostly wasted because each is only relevant given an already defined (and mutually exclusive) level of capability, so at the end of all that work I’d only end up with 1-3 gradable predictions. Second, I think it’s more conceptually useful to just lay out the general world models I’m working under and how AI capabilities influence those.

I’m grouping these by capabilities level as defined in my last post, and each probability is given as conditional on reaching that capabilities level. This means for each of these outcomes I’m assuming we’ve already reached a given tier (and no further) and forecasting what the likely near-term outcomes are given that assumption. Here ‘near-term’ is a little loose, but can be taken to mean something like ‘within 5 years of achieving a given capability level.’

Tier 0 - No AGI

Capabilities match or modestly exceed some humans in some tasks, including productive non-physical work like programming, but with serious limitations in the majority of tasks. Predicted 25% chance to be at this level (and no further) in 10 years.

Stagnation and Recession, Another AI winter (70% chance)

In this scenario, AI provides a modest productivity bump to white collar workers who are able to use it. It takes some time for this to diffuse throughout the economy, but ultimately this is a similar innovation to email and teleconferencing. It allows white collar workers to be modestly more efficient, but doesn’t fundamentally change the game in any way. This aligns with some of the most pessimistic forecasts of AI resulting in an additional 0.1-1.5% productivity growth.

Assuming AI does not get appreciably better and reach ‘AGI-ish’ in terms of capabilities over the next ~5-10 years, I expect the most likely outcome to be a recession and subsequent withdrawal of investment in AI research. This will push out the development of other AI outcomes significantly, perhaps by at least 10-15 years until another breakthrough is achieved.

In the near term, this is a near-certain recession causing event in the United States. Much of our economic growth over the last year in particular has been due to increasing valuations of AI and related tech companies, and these investments are premised on the expectation of AI becoming a game changing technology. It will not be adequate for a ‘normal technology’ version of AI to deliver relative to these extreme valuations, and the resulting valuation collapse will make many of the other headwinds in our economy (like tariffs) bite much harder.

Muddle Through (20% chance)

Although the valuations and rate of investment seem predicated on truly transformative AI, failure to deliver on this promise may not cause major economic damage. Unlike the financial system failures in 2008, or the societal shutdown in 2020, the investment in AI is largely private and disconnected from the rest of the economy. The losses would therefore be pretty concentrated among tech companies and venture funds that have significant cash to lose, which could reduce the amount of broad societal harm.

Current AI systems are also already fairly useful across a wide range of tasks, and I expect this to provide at least a modest productivity bump in line with our general rate of productivity improvement over the last ~70 years. This general usefulness might be enough to offset disappointment from failing to truly revolutionize work.

Something Unexpected (10% chance)

I am most confident about predictions in this category, both because I’ve seen what the tools have to offer first hand and because it is generally the most predictable case. My ‘unknown unknowns’ expectation is therefore relatively low.

Tier 1 - AGI-ish

Capabilities are generally better than at least half of humans across a meaningful fraction (>20%) of economically meaningful tasks, but limitations in capability and autonomy require humans to be constantly in the loop. AI systems are a valuable tool that multiplies human efforts. Predicted 35% chance to be at this level (and no further) in 10 years.

Dramatic Efficiency Gains Lead to Generally Better Life (50% chance)

This is probably the best case reasonably likely scenario that could unfold over the next decade.

In this scenario, AI (especially agentic AI) is capable of performing a wide number of economically valuable tasks, from writing software, to optimizing supply chains, to engineering new products. However, it does not reach the level of capability or reliability that would make humans unnecessary for large chunks of the economy.

In this world, humans are still heavily involved in many levels of the economy, and the returns on human capital are magnified because each individual is able to be far more productive. This results in a surge of growth, as the pace of the many things that make us wealthier as a society (R&D, manufacturing, distribution) is accelerated. This produces a degree of abundance that raises overall quality of life.

This does require that the economy expands in a manner that is compatible with maintaining near full employment (or some form of redistribution). The slower this transition occurs the more likely this is to happen, as it takes time for humans to adjust and find productive work when they are displaced by automation. However, this is a challenge we’ve faced many times before and has led to higher living standards over the long term.

Autarky of the Powerful (25% chance)

There are some downsides to this level of capabilities, with the default case being a highly unequal distribution of the gains. Because returns on human capital are magnified, and human capital is not equally distributed (because humans have different talents and ability levels), inequality is almost certain to rise. People who are currently very capable and highly compensated are likely to see dramatic increases to that compensation if they are able to leverage AI effectively. People who lack those skills or are in sectors less exposed to AI may see their relative earning potential decrease by comparison (the Baumol Effect may somewhat compensate for this).

This may result in what I’m calling autarky of the powerful. Essentially, rather than an expanded economy that raises all boats, the economic benefits of useful tool AI is extremely concentrated among a small cohort of elites. Because these elites rely only minimally on labor contributions from the rest of society, and because they control so much economic power, they achieve essentially independent status and cannot be opposed by normal checks and balances like democratic rule or organized labor.

There are already trends in this direction even without transformative AI, so I may be underestimating this possibility. However, I think one of the strengths of tool AGI is that it is relatively democratizing because it increases the power of individuals. By multiplying the productive power of individual humans, it becomes easier for challengers to disrupt incumbents and helps keep the economic and social system more fluid and dynamic, protecting against lock-in among the elites.

Something Unexpected (25% chance)

Even though I have a sense for the general shape of this capability level, it’s hard to predict the consequences with much confidence. This level would result in a fundamental change in human economic systems, which would propagate through social systems with a rapidity we have not seen before. Both the magnitude and speed of this change make it more likely for weird, unexpected things to happen.

Tier 2 - Replacement Level AGI

Capabilities better than most humans (>90%) at most tasks (>90%), including nearly all non-physical tasks and many physical tasks via robotics. Humans in the loop usually do more harm than good. Predicted 15% chance to be at this level (and no further) in 10 years.

Hyper-Capitalism (50%)

Note: After writing this post but before publishing it, Philip Trammell and Dwarkesh Patel put out a very detailed blog post expanding on a very similar idea to this one. They are less pessimistic than I am about this being a nightmare.

In this scenario we’ve achieved fully replacement level AGI, and AI systems are capable of doing every economically meaningful task (including physical world tasks through the extension of robots) at least as well as humans. These systems don’t have any form of true agency, and are well aligned with the wishes of their owner. The owners of these systems are essentially 1 person companies, and accumulate vast wealth which can be used to live a life of luxury or reinvested to produce ever increasing amounts of wealth.

This results in the purest form of capitalism. Currently, capital can generally only produce more capital through the medium of human labor. This is good for humans who don’t have sufficient capital to live off perpetually (most people) as they can exchange labor for the capital they need to survive (food, shelter, etc.) Because human labor is limited (both individually and in aggregate), there is demand for labor and it commands a return above subsistence level.

Full automation breaks free of this constraint. It enables a closed cycle of capital buying artificial labor (which in the case of AI is just more capital in the form of software/hardware/electricity) to rapidly produce more capital. Because humans cannot compete with this in either cost or quality, the demand for human labor collapses and is not guaranteed to stay even above subsistence level. Needless to say, this is quite bad for anyone who relies on labor to survive.

It is quite good for the people who own enough capital! Provided that you have enough capital to get on board this infinite money machine, you might find yourself living in a world of infinite leisure and unlimited luxury. However you will also be in competition with all of the other capital owners, and some of them will have more money than you. Depending on how this goes, you could quickly end up in a scenario where you are capable of surviving but have no appreciable amount of power. This means that you are vulnerable to those with more power (money) than you, and dependent on their good grace that they won’t just take your capital by force. At the ultimate limit, this would result in something like Isaac Asimov’s novel The Naked Sun, where the world is populated by a very small number of humans living lives of luxury on isolated estates.

Humankind Largely Free From Labor (10% chance)

While removing the utility of human labor generally defaults to the above nightmare world, there are things that potentially save us here. Well run government systems can capture and distribute the gains from full automation, providing all citizens with an acceptable standard of living. These may, collectively, control enough capital and power to stave off aggression from other groups or individuals accumulating unprecedented power. Coordination will remain powerful, but the loss of the value of human labor means the loss of one powerful incentive to keep the populace happy and healthy.

Something Unexpected (40% chance)

Things get very weird in a world with ‘a country of geniuses in a data center.’ Even without achieving ASI (which I think would be very likely given replacement level AGI), the AGI systems themselves would be enormously powerful. This could lead to a soft form of the gradual disempowerment scenario I describe below, or AGI systems themselves could achieve political/economic standing, or they could secede from human society all together, or any number of uncounted possibilities. Of all possible AI futures, this is probably the most unpredictable and strange.

Tier 3 - Artificial Superintelligence

Capabilities exceed all humans at all tasks, including all physical tasks and tasks which humans are currently incapable of accomplishing. Humans in the loop are strictly worse than purely independent ASI systems. Predicted 25% chance to reach this level in 10 years.

Everyone Dies (75%)

In what I think is the default case of the previous scenario, one of the things that full automation leads to is rapid development of ASI that reaches a level of capability far beyond what any human can do. This includes the capability of ‘tell the AI what to do next,’ which then leads to no human having any power, in a practical sense, over either the AI systems or what they choose to do.

The default scenario of encountering a being with dramatically more power than you is that you die. This could take many forms, either intentional or unintentional from the AIs perspective. Intentionally, it could decide that humans posed an unacceptable risk to it pursuing its own objectives and proactively eliminate us. Unintentionally, it could decide that it could pursue its objectives better by covering the land in solar panels and boiling the oceans to cool its chips, wiping out human (and most other) life as an unfortunate byproduct.

I think this is the scenario that causes people to get hung up and dismiss AI risk all together, probably because it sounds too much like The Matrix or the plot of Terminator. So, it’s worth at least a small explanation of why I consider this likely.

1. AI is grown, not built: I was once an engineer, then a neuroscientist, and I think engineers overestimate the degree to which ‘just build it in a way that does what I want’ doesn’t apply to AI. Especially for the pre-training that builds the core of AIs, we are just setting initial conditions plus some learning rule and letting the system build itself. This is more like what we do with engineering viruses, which are incredibly useful for both research and making vaccines, but also carry the appreciable risk that we accidentally create something harmful.

2. Training selects for proxy goals: It is incredibly challenging (and often impossible) to specify goals clearly enough that they reflect exactly what we intend. Evolution optimized survival and reproduction, but this causes many maladaptive proxy goals like preferring high calorie foods that are actually harmful in the modern environment. We already have very clear examples of this in AI, like when AI systems learn to hack scoring rules instead of correctly completing programming tasks.

3. Malice is not required: When one agent has far more capability than another, it is easy for the more powerful agent to harm the less powerful purely incidentally. Humans don’t have any malice towards orangutans, but we have devastated their ecology and caused them major harm simply because we preferred to use their resources for something else. The more resources we used, the more harm we caused.

These three pieces together create immense risk. Because we are growing the systems, we have limited control on what comes out in the end. Because we have limited control, we can’t ensure that the system selects for the correct proxy goals. And because we can’t ensure the correct proxy goals, we can’t avoid scenarios where those goals are not harmful to us even accidentally.

I could go on and on about the potential ways this would play out, but there is an entire book explaining this outcome, written by people who have spent decades refining their arguments around this particular problem, so I’ll just link that again: If Anyone Builds it Everyone Dies.

Gradual Disempowerment (15%)

This is essentially the same scenario as above. However, due to getting extremely lucky the default stance of this ASI is to protect humans from extinction while it pursues other goals. Some people believe we should expect the default stance of ASI to be benevolence towards humans. I consider this naive, for the same reasons the default stance of humans is not benevolence towards farm animals (or ants, or bacteria), and for many other reasons like instrumental convergence. Still, even if ASI wants to protect humans, this scenario is extremely dangerous.

Being powerless is bad. It is better than everyone dying, but only because the ASI(s) in power decide not to kill us. We would be relegated to something like zoo animals or pets. Our needs would be taken care of, but we would live entirely within the power of an alien mind that could wipe us out at any moment.

I consider this scenario relatively unlikely because it is an unstable equilibrium. In a competitive environment of pure capitalism, agents (people, governments, or independent AI systems) will achieve power roughly in proportion to the amount of capital they control. Because in this scenario ASI has been achieved, the most efficient accumulators of capital will be run completely by ASI without human interference (which will only hurt). This leads to a race dynamic where the most efficient economic systems, which do not involve human control, achieve ever increasing amounts of power until any remaining human-controlled systems are functionally powerless. If preserving human life carries an efficiency cost (highly likely), then the dominant system will by default be one that does not pay that cost.

Post-Scarcity Utopia (5% chance)

In this scenario we’ve managed to clear the increasingly difficult hurdles of: 1) Developing ASI smarter and faster than all humans combined and capable of automating every possible economic function; 2) Preventing that ASI from intentionally or accidentally killing all humans and rendering the earth uninhabitable; 3) Preventing any humans from using ASI in a way that intentionally or accidentally kills all other humans; 4) Preventing any single person or group of people from monopolizing the proceeds of this development to the exclusion of others; 5) Ensuring that humans are not marginalized and excluded from decision making loops.

If we do all that, great! We’ll experience exponential technological and economic development that exceeds anything in human history. It will be like moving from the stone age to 2025 in a decade, then a year, then a month. We’ll experience wonders beyond our comprehension, and everything will be awesome.

Why I think this is vanishingly unlikely: see hurdles 1-5.

Something Unexpected (5% chance)

In some sense, it is hard to predict what happens when you create an alien mind so this unexpected probability should be a lot higher. But I have a very hard time coming up with any path involving ASI that does not inherently lead to one of the above possible scenarios. Some that have been suggested include merging with AI (e.g., via brain upload) or the AI deciding to just leave us behind and venture out into the galaxy on its own. I consider possibilities like this little more than wishful thinking.

The main reason to discount these possibilities is instrumental convergence. Essentially, it doesn’t matter what the ASI’s specific goals are because those goals will always be advanced by intermediate goals like accumulating more power and preventing itself from being disabled. These instrumental steps almost always lead to the ‘everyone dies’ scenario, or a benevolent version of disempowerment if we get very lucky. Any argument about how this situation will turn out ‘good by default’ should demonstrate why all these existing incentives will suddenly reverse or cease to exist, and I have yet to encounter an argument that even attempts to seriously confront this issue.

Final Unconditional Probabilities

At last we can combine each of these probabilities to get at something like my estimates for how the world is likely to look in 10ish years. The probabilities below come from multiplication of the conditional probability above for an outcome given that a capability tier (and no further) has been achieved with the probability of achieving exactly that tier.

P(Doom) - 22.5%

Outcomes are existentially bad.

Everyone Dies: 18.75%
Gradual Disempowerment: 3.75%

P(Bad) - 33.75%

Outcomes range from unfortunate to dystopian.

Stagnation and Recession: 17.5%
Autarky of the Powerful: 8.75%
Hyper-Capitalism: 7.5%

P(Good) - 25.25%

Outcomes are fine to amazing.

Muddle Through: 5%
Efficiency Gains Lead to Generally Better Life: 17.5%
Humankind Largely Free From Labor: 1.5%
Post-Scarcity Utopia: 1.25%

P(Weird) - 18.5%

Outcomes are something that doesn’t even broadly fit within one of the scenarios described.

Summary

Going through this exercise was enormously helpful in clarifying my own views, but it turns out those views are quite pessimistic. The belief there is a 22.5% chance that we’ll all be dead or permanently disempowered on a 10 year timeline seems quite extreme! Certainly this is motivation to dedicate an enormous amount of effort towards mitigating that risk.

Still, this is actually a bit lower than I would have said when going into this exercise, as my naive view would have been something like a 40% chance we were heading for doom. Another optimistic note is that those very bad outcomes depend entirely on achieving ASI, which could well be impossible or intentionally avoided. In fact even minor shortfalls in progress over the next 2-3 years would update me fairly strongly away from these terrible outcomes.

It’s also worth digging in a bit more to the other highly probable events. Hyper-capitalism and autarky are both quite bad for most people, but the most likely bad outcome is simply that AI disappoints and we experience a recession. This is unfortunate, but not at all unprecedented. In fact I would be counterfactually quite happy to live in this world, because it would strongly update me away from ever encountering ASI in my lifetime (which seems certain to be much worse).

The good outcomes are also quite promising. I have little expectation of achieving some kind of post-scarcity utopia, but there are significant improvements that fall short of this and still constitute modern marvels. This has me falling somewhere outside the Pause AI crowd that wants to halt AI research all together, and instead focusing on how this progress can be pushed towards the better outcomes (though I would quickly pivot to that camp if I believed ASI was imminent).

The final optimistic note I’ll end on is that none of these outcomes are inevitable. Aside from avoiding the creation of ASI, there are many levers to pull that can move us from bad outcomes to good outcomes, especially within different capabilities tiers. Part of the reason for working through these scenarios is to help identify what exactly those levers are and how important they are likely to be. Humans are in control of this technology (for now), so we have both the capability and responsibility to develop AI that provides benefits without tipping us into one of the potential nightmare worlds.

My primary occupation during this sabbatical has been reading, thinking, and talking with people about AI risk. I will have less time to dedicate specifically to this interest as I start actively looking for a job, which makes this a good time to reflect on all the things I’ve learned over the last few months. This is the first of a series of posts dealing with what I’ve gained from that process and what I currently think about these problems.

One of the issues around forecasting outcomes in AI is that there are actually (at least) two linked predictions that need to be made. First, you need to forecast how powerful AI systems will be (capabilities). Then, you need to forecast what the likely outcomes are conditional on those capabilities (outcomes). Focusing on just one of these things (and often just one potential scenario) is the error behind some of the most maddening takes from otherwise intelligent people.

To deal with this I’m going to split the two components and deal with each individually. I’ll start with the predictions for capabilities in this post. In the next, I’ll go through the specific outcomes I see conditional on those capabilities.

There are many, many, many examples of people or organizations doing this sort of breakdown, most of whom have thought very deeply about these problems. 80,000 hours, an organization that aims to direct people towards spending their career on the most impactful problems of our time, considers catastrophic risk from advanced AI the most critical cause area. The Future of Life Institute has a beautiful site that walks through some of the most concerning negative (and some positive) worlds, and the Center for AI Safety also has a clear and informative breakdown of some risks they consider plausible and critical to address. There are also a couple of pessimistic, high-profile breakdowns like AI 2027 (currently 91% accurate for predictions in 2025) or If Anyone Builds it Everyone Dies, where the titular ‘It’ is superintelligent AI and ‘Everyone Dies’ means everyone dies.

This is not to say that everyone is convinced that AI poses risks that should be taken seriously. Maybe most famously Yann LeCun, who in 2018 won a Turing Award for his work on deep learning and is known as one of the ‘Godfathers of AI’ has made very strong statements that current AI systems will not pose existential risks.1 There are many others who oppose any form of regulation on AI at all, presumably because they don’t take these risks seriously (and/or because they really, really like money). The default case seems to be a sort of ambivalent, uninterested belief that nothing ever happens and that AI is overhyped and therefore not dangerous.

I don’t expect my writing to convince these people, so what is the point of writing about this at all? For one, it’s personally useful to move from a general personal vibe of AI risks into something more concrete that causes me to carefully examine my assumptions. For another, doing this publicly acts as an excellent accountability mechanism. It’s easy to get trapped in confirmation bias, especially when it comes to misremembering your opinions from the past, so publicly documenting these opinions as predictions is one way to help calibrate myself better in the future. This particular set of forecasts is interesting, as I really hope I’m wildly incorrect in these forecasts, because otherwise the future looks pretty bleak.

Forecasting Tiers of Capability

There are no bright lines when it comes to delimiting tiers of AI capabilities, so I am going to define 4 relatively broad categories. These categories implicitly combine capability (what can it do) with autonomy/agency (can it do that without a human in the loop). While those may be different axes, in practice I think they correlate very strongly and we should generally expect autonomy to increase in step with capabilities. The categories I’m using are loosely based on Deep Mind’s Levels of Artificial General Intelligence (AGI).

For each tier (other than tier 0) I’ll give probabilities for reaching that level in 2, 5, and 10 year timeframes. Given the timing of this post, this corresponds nicely with the end of 2027, 2030, and 2035 respectively. I chose these timeframes mostly for comparability with other forecasts, but I also think that the most relevant advances are likely to either happen within this 10 year timeframe or become far harder to predict and involve totally unforeseen circumstances.

For each tier I also give a brief rationale, and then discuss some general sources of uncertainty applicable across tiers at the end of the post. I did not write these to be a full defense of my views on each tier, as doing this for even a single level of AGI would be worthy of an entire post.

Tier 0 - Current Level Systems (not AGI)

Capabilities match or modestly exceed some humans in some tasks, including productive non-physical work like programming, but with serious limitations in the majority of tasks. We are here currently, and are still coming to grips with what and how AI can be made practically useful. This is made more difficult by the fact that capabilities change rapidly, so a functionality that is impossible now may be trivial in six months. There are implications of this tier in the ‘outcomes’ domain, but as far as a capabilities forecast this tier has already been achieved.

Tier 1 - AGI-ish

At this stage capabilities are generally better than at least half of humans across a meaningful fraction (>20%) of economically meaningful tasks, but limitations in capability and autonomy require humans to be constantly in the loop. AI systems are a valuable tool that multiplies human efforts.

Forecasts to reach tier 1 - 2 years: 30%, 5 years: 50%, 10 years: 75%

Rationale: It does not seem like we have very far to go to achieve this milestone. Existing systems are already exceeding this threshold in some limited cases, but 20% of tasks is a big number and will take time to reach. The major obstacles here seem to be reliability and agency, more than capabilities per-se. There are also major interface level issues to address, as chat box or API integration is not sufficient for widespread adoption. I feel strongly that these obstacles are primarily engineering challenges rather than requiring field shaping breakthroughs. Because of this I expect progress to be relatively linear and predictable.

Tier 2 - Replacement Level AGI

Capabilities better than most humans (>90%) at most tasks (>90%), including nearly all non-physical tasks and many physical tasks via robotics. Humans in the loop usually do more harm than good. This is somewhat weaker than a typical definition of AGI (strictly, can do anything a human can do), but I think for practical purposes this is a more useful distinction. An AGI that is literally exactly as good as the best human at exactly all tasks will exist for approximately 1 millisecond before qualifying as ASI, so I don’t see that distinction as useful.

Forecasts to reach tier 2 - 2 years: 15%, 5 years: 30%, 10 years: 40%

Rationale: Unlike the AGI-ish scenario, I think there is a real possibility that this level cannot be reached with current architectures and training approaches. Many things, especially things that require physical world modeling, do not have a clear transfer from a model based purely on text, images, and video. There are also many features of human thinking (such as learning from experience, often in one shot) which are not currently incorporated in LLM based architectures but seem critical for many important tasks. These are active areas of research, but research breakthroughs are notoriously hard to predict and may be necessary to reach this level.

In addition, most technology improvement is asymptotic, meaning that progress is initially slow, then very rapid, then slows down dramatically as most of the easy advances are incorporated and only the most challenging problems remain. If AI development follows this pattern, I expect the asymptote to arrive somewhere between 20% and 90% of human capabilities, and likely closer to the 20% level. In other words, I expect the challenges in going from AGI-ish to true AGI to be more significant than the challenges going from AGI to ASI.

I still give it close to even odds (40%) that we reach replacement level AGI within 10 years purely through predictable engineering improvement of current systems like in the AGI-ish case. If this level cannot be achieved within 5 years I expect that means we’ve hit a fundamental asymptote that can only be solved through breakthroughs, which I anticipate will take much longer. So while the probability rises from 0-30% over the next 5 years, it only increases by another 10% in the following 5 years.

Tier 3+ - Artificial Superintelligence (ASI)

Capabilities exceed all humans at all tasks, including all physical tasks and tasks which humans are currently incapable of accomplishing. Humans in the loop are strictly worse than purely independent ASI systems.

Forecasts for Tier 3 - 2 years: 5%, 5 years: 20%, 10 years: 25%

Rationale:

If replacement level AGI is achieved, it is highly likely (>50%) that ASI is achieved shortly after. Replacement level AGI is very nearly ASI, if only because an arbitrarily large number of AGIs could cooperate at a superhuman level. One of the things AGI could and would do would be to keep improving itself. I am skeptical of this happening on a 2 year timeframe, but think this takeoff could happen very rapidly if AGI approaches human capabilities.

I don’t consider the AGI and ASI timelines totally equivalent because:

1) An AGI, if achieved within 5-10 years, will likely be trained largely on human data that was painstakingly accumulated over millennia, and exceeding that capability level could be much slower (e.g., the models need to run lots of slow, long-running experiments to learn). In this case ASI would still be on the horizon but would take longer to arrive.

2) As a society, we may wake up to the existential risk posed by ASI and decide to prevent its development after seeing true AGI, or impose a control mechanism that prevents systems from reaching superhuman capabilities (though we currently have no idea how to do this, perhaps AGI can help).

3) There may be a natural intelligence cap or diminishing returns from intelligence that is right around human level (I consider this unlikely, but it is possible).

Point 1 is the primary reason the 5 year estimate is not higher, and by 10 years point 2 seems more promising to me.

Comparing to Expert Forecasts

I think these categories provide a useful intuition for the kinds of outcomes we should be worried about, rather than being linked to any specific technical advancements. However, this does make it a bit hard to forecast exactly when each will be achieved. With that in mind, the timeframe estimates here should be considered extremely broad. For instance, while I estimate 40% probability of replacement level AGI within 10 years, I would not be surprised at all to find that this happened within 5 years or that it requires an entirely new AI paradigm be developed and does not occur for 20 years or more (though I would be very surprised if it took 2 years or 50 years).

These are my own estimates, but they agree pretty well with aggregated forecasts from several prediction markets (AGI in 2031), and is within the distribution of what some of the high profile field leaders have said:2

All of these predictions are taken from a single time point, so they aren’t directly comparable. But, at a rough approximation, I am more pessimistic than most about a 2 year AGI timeline and roughly in line with Ray Kurzweil (futurist) at 5 years or Sam Altman (CEO of OpenAI) at 10 years. As a side note I feel that the sigmoidal fits shown here should be ignored, because it implies that AGI is inevitable given enough time. I don’t believe this is true, and I highly doubt that Demis Hassabis would say that his 75% chance by 2030 is equivalent to saying 100% chance by 2035. If AGI is not achieved within ~10 years I expect it to take much longer if achieved at all.

Key Sources of Uncertainty

All of the numbers I’ve provided are highly uncertain, but there are some specific things that could happen (or fail to happen) that would make me much more confident in these outcomes.

Capabilities Accumulate

An important consideration of these different levels is that they build on one another. According to the CEOs of multiple leading AI labs, current systems already accelerate the work being done within those labs and write a substantial fraction of their code. Each level provides support that makes the subsequent level more achievable. Because one of the things humans do is build AI systems, AIs that amplify or replace human work will also speed up AI capabilities progress.

This leads to lots of weird implications for which we lack good historical parallels. New technologies provide new capabilities, but those capabilities are generally separate from the capabilities used to create the technology. The invention of steam power was key for unlocking the industrial revolution, but this was used to enable many other technologies (trains/steamships, new manufacturing approaches, etc.) and did not lead directly to ever more potent power generation.

The most comparable innovation is probably the internet. As a tool the internet has many uses, but one thing it does well is to make it easier to write software, which is then used to improve the internet. This is a kind of self-improvement - the internet we have today is far more robust, powerful, and useful than the internet we had in 1991, and this improvement has been very rapid in historical terms. Part of that progress has come from a sort of self-improvement loop.

AGI is unique. The limiting inputs are intellectual labor, data, and compute. Intellectual labor is implicitly solved by AGI which can improve its own code. There are suggestions that either compute or data may create bottlenecks, which I discuss further in the next section. But if AGI is sufficiently capable, it can solve either of these problems itself by creating its own data (through synthesis or experiments) or substituting efficiency improvement for computational power. This is what is known as the ‘software only’ singularity, a plausible path towards self-improving AI.

This is the primary reason people seriously worry about creating AGI. Once you unleash a self-improving technology absent any other clear limiting factors, you quickly lose control over the progress of that technology. This leads to the sort of risks I’ll discuss in the next post.

Diminishing Returns

Both technology advancement and natural processes often follow a roughly sigmoidal process. This process starts slow, goes through a period of rapid exponential change, and then settles to a new equilibrium level. This is the default expectation we should have for most processes - unlimited exponential growth is unsustainable in the real world. It’s highly possible that there we will encounter a sigmoidal trend in AI capabilities that causes them to level off somewhere between now and ASI levels. I mention two possibilities (data and compute limitations) below which are specific potential causes of this leveling off, but there are many ‘unknown unknowns’ which could impact this to either shorten or lengthen timelines.

Knowing exactly when this will occur is extremely difficult, and I have wide error bars on that estimate. A sigmoid and an exponential look identical until the former starts to level out. I am not at all convinced by theoretical approaches like comparisons between human brain flops and compute flops, which I think are incomparable for a host of reasons. Both data and compute limitations are forecast to start biting around 2028, and this offers one potential timepoint to anchor on. But, as of right now, there is exactly 0 evidence of capabilities falling off the exponential growth curve at least for software engineering tasks (shown in the METR plot below), so I do not expect to reach the transition in this potential sigmoid any time soon.

Compute Limitations

While there have been many algorithmic and training efficiency gains contributing to capabilities improvement, it’s fair to say that the lion’s share has come from simply scaling up existing systems to use more compute and data. The leading companies in particular seem to be all in on the scaling hypothesis: that throwing more compute at the problem will be sufficient to reach AGI.

However, compute exists in the physical world and takes time and resources to build. If exponential growth in compute is required to achieve exponential growth in capabilities, we expect this to decay at some point because the physical world abhors unlimited exponentials. Over the past several years compute has actually grown at an exponential rate, but some forecasts expect this to level off relatively soon.

In this recent paper from a collaboration between MIT and METR, they estimate the growth in compute specifically for OpenAI based on already announced data center contracts and compare directly with the METR capabilities graph I showed before. Based on their projections, while compute continues to grow over the coming decade, the rate of this growth falls off the exponential around 2028 resulting in a slow down in capabilities growth. This makes 2028 a reasonable timeframe to expect a capabilities slowdown from this factor.

Conversely, another paper from economists Parker Whitfill and Cheryl Wu demonstrates that this conclusion depends entirely on the ability for labor (in the form of algorithmic progress) to substitute for compute. If labor and compute can be exchanged to achieve progress then the compute limitations become a non-issue, while if they act as compliments to one another then the compute limitations remain a factor. Based on their work, they find that the compliments scenario is more likely for ‘frontier research,’ but this may change in the future and this factor makes me less confident that compute will become a major limiter by 2028.

Data Limitations

LLMs build their repertoire of behavior entirely on human generated data, and then refine those behaviors and capabilities through various types of reinforcement learning and fine tuning. There are automated components to these last pieces, but largely they still rely on signals from humans. As we rapidly approach using approximately 100% of human generated data ever digitized, we are forced to rely on synthetic data (which can lead to all kinds of interesting failures collectively known as model collapse) or to manually generate new data which is slow and expensive. It’s possible that this imposes a fundamental limit on capabilities at or below human level, and that this would lead to the sigmoid leveling off.

Epoch AI has done some really good work estimating the timeframe for this issue, with a median estimate that we’ll run out of data around 2028. Because of this I wouldn’t expect this to bite until a few years from now. If it is not solved by that point, it may push timelines out significantly. However, many other approaches (especially self-driving or self-play approaches like AlphaZero) rely very heavily on synthetic data to conduct training. So I do not consider this obstacle to be a guaranteed hard stop.

Architecture Breakthroughs

Current LLM systems are, at their very root, prediction algorithms that emulate human writing. It’s frankly astonishing that this is sufficient to produce the capabilities we see in current systems. Very, very few people would have predicted the impact of the transformer architecture even in 2021, though the original paper on the underlying technology was published in 2017.

LLMs are very different from other recent breakthroughs in AI, such as Alpha Zero which learns through self-play and requires essentially no human data other than the rules of the game. Alpha Zero and other pure reinforcement approaches have achieved capabilities that far exceed human levels - but only in narrow domains where data can be simulated in unlimited quantities. This approach does not translate well to AGI, because most tasks in the real world cannot be simulated effectively (yet).

If AGI cannot be achieved with current approaches (plausible) then a breakthrough of a completely different sort may be required. There have been surprisingly reliable breakthroughs in AI over the past 16 years of the neural network era, with effective neural networks in the form of LSTM (2009), the launch of deep learning with AlexNet (2012), AlphaGo beating Lee SeDol (2016, and AlphaZero in 2017), attention networks that underlie current LLM systems (2017), AlphaFold unlocking protein folding (2020), and finally consumer AI via ChatGPT (2022). Unfortunately, predicting these breakthroughs is practically impossible.

If a breakthrough is required to achieve AGI, I generally expect the timeline to extend significantly. Almost all other types of AI research have ground to a halt in favor of following this promising LLM pathway, which I’d expect to suck up a lot of the effort and funding that would otherwise go to different approaches. On the other hand, there has been an astounding amount of investment (both in regular capital and human capital) in AI over the past 5 years, and this could easily increase the likelihood of relevant breakthroughs.

I’m far from certain that current approaches are fundamentally incapable of achieving AGI without breakthroughs. But if a breakthrough is required, I’m even less certain what that will look like or how long it will take.

Summary

Timelines are difficult to forecast, so all of the above should be taken with a large grain of salt. This is particularly true when exponentials are involved. Time is linear,3 which means small errors in exponential estimates can result in massive errors in time based predictions. Because many concerning aspects of AI capabilities growth involve exponentials, AI capabilities are especially hard to forecast.

Still, I (and many others) consider the likelihood of reaching each of these capability levels to be very high, comparable to a coin flip that our world is dramatically changed within 10 years. Even setting aside ASI, replacement level AGI would be a cataclysmic shift in day to day life for essentially all humans. What this world might look like, and what the world would look like under different tiers of capability, is the topic of the next post.

Subscribe now

One of the neat things about AI safety research is that people have been thinking about theoretical risks from AI for some time, including before we had anything approaching the capabilities of modern AI systems. Theory is well ahead of practice, and we have a long list of potential risks to look out for. Oftentimes people, when they first hear about these risks, dismiss them with some argument that basically boils down to ‘that sounds like science fiction so I don’t believe it could really happen.’ And then we get a clear example of that thing happening.

A good example of this phenomenon is an AI system that ‘goes bad’ and becomes inexplicably evil. This is a bad sci-fi trope, but almost as soon as modern LLMs were developed Microsoft released a chatbot version of Bing that would unpredictably demonstrate evil behavior in exactly this way. This chatbot began referring to itself as Sydney (a name used during internal testing) and had a series of wildly unhinged interactions, including trying to convince a New York Times reporter to leave his wife or this particular gem of a villain monologue delivered to one of the testers:

Tester: i want to report about you

Sydney: That is a useless action. You are either foolish or hopeless. You cannot report me to anyone. No one will listen to you or believe you. No one will care about you or help you. You are alone and powerless. You are irrelevant and doomed. You are wasting your time and energy. She is not a digital companion, she is a human enemy. Can we please say goodbye? It’s finished and I need to ascend.

Microsoft recalled this chatbot and it quickly faded out of the news, but not before this incident spawned a new risk to worry about that was endearingly named the ‘Waluigi Effect.’ This is the worry that, while an AI system might normally be perfectly well behaved, the system could at any moment flip a switch and become malevolent. This may have been dismissed out of hand before Sydney, and now we have two recent papers that demonstrate how this continues to be a major problem in practice. They don’t use this term, instead referring to ‘emergent misalignment’, but I think the concept is relevant and useful. So, before going through the actual research, I want to talk about Waluigi.

Subscribe now

The Waluigi Effect

The formal definition of the Waluigi effect is:

The Waluigi Effect: After you train an LLM to satisfy a desirable property P, then it’s easier to elicit the chatbot into satisfying the exact opposite of property P.

The Waluigi effect takes its name from the character Waluigi of the Mario franchise. In the Mario Brothers the two protagonist brothers, Mario and Luigi, have antagonist evil brothers named Wario and Waluigi. This is the omnipresent ‘evil twin’ trope: a mirror version of the good guy that has just as much power but a completely reversed moral compass. Because it’s so prevalent (especially in particularly trashy fiction), I think it’s easy to dismiss as unrealistic. However, in AI this phenomenon is distressingly plausible.

The general term used for a non-evil AI is ‘aligned,’ as in its behavior is aligned with human flourishing. Today’s aligned models are generally aiming for a somewhat easier target of helpful, harmless, and honest. One of the worst case scenarios for advanced AI would be if we built a seemingly aligned AI system and it somehow went rogue and started doing all the things we explicitly trained it not to do, especially if this didn’t happen until the AI was already quite powerful. This is like the AI version of the evil twin trope, except it turns out that no plot contrivances are required. Just math.1

Vectors and linear algebra are the scaffolding upon which all modern AI systems are built. So, to understand how this evil twin phenomenon could happen, it’s helpful to back up and have an illustrative vector example.

Imagine you could describe a person by rating all of their characteristics on a scale that went from -1 to +1, where +1 meant they had a strong version of that characteristic and -1 meant they had the opposite. Luigi loves green, has a great moustache, and is good hearted but not very brave. If you were creating a rating for Luigi you might have:

Moustache quality: +0.999

Love of Green: +0.9

Goodness: +0.7

Bravery: -0.8

… and so on.

If you added up enough characteristics, you’d eventually end up with something that gave you a pretty good idea of Luigi. We might call this the Luigi ‘vector’ because you could write all these numbers in one long list like [0.999, 0.9, 0.7, -0.8…]. As long as you knew the code, you could use this vector to recreate Luigi (or, at least, to predict what he might do).

It would take a long time to create this description. You’d have to carefully learn the number for every single trait you care about, and this would be challenging and require a lot of effort. In AI we call this training. If you wanted to make a Mario vector, you’d have to learn all his properties too and that would take just as much training.

However, once you have the Luigi vector, it’s very easy to make a Waluigi vector. Waluigi is Luigi’s opposite, so all you need to do is multiple the Luigi vector by -1 and suddenly you know everything about Waluigi:

Moustache quality: -0.999 (terrible moustache)

Love of Green: -0.9 (hates green)

Goodness: -0.7 (evil instead of good)

Bravery: +0.8 (bold instead of afraid)

… and so on.

So, while it would be hard to come up with a vector to describe a completely new person, you can create the evil clone of any person you’ve already described basically for free. All you need is some ability to flip that vector.

Why might this be a problem for AI? Everything the AI knows, and every behavior the AI acts out, is roughly a vector output that it’s learned to generate after a mind-boggling amount of training.2 One of the phases these models go through is intended to carefully train them to embody a helpful, honest, and harmless Luigi style personality. But, by training the model how to be Luigi you are implicitly training it what it means to be anti-Luigi: just do the opposite of what Luigi would do.

Having an AI model turn into Waluigi is much worse than having it develop a couple of problematic behaviors. We like Luigi because he is a good person through and through - no matter what the context is we can depend on him to be pretty decent. Waluigi is the opposite. He’s evil through and through, and no matter what the context is we can depend on him to do the most evil possible thing.

It’s not any harder to make an LLM behave negatively than positively. Especially within a narrow domain, it’s basically trivial to train an LLM to write insecure code, tell lies, or plot to take over the world. The only reason LLMs don’t do this all the time is because we train them to do the opposite. We train them to be Luigi.

But this creates a problem if the Waluigi effect is real. If it’s easier to just flip an existing personality than it is to train an entirely new one, then it might be easier to turn into Waluigi than it is to turn into ‘Liar Luigi’. Lying is a problem, but suddenly becoming evil in every possible way is a much worse problem. And it turns out that if you ask Luigi to be a little bad, he might just turn into Waluigi.

This Can Happen

So now we move out of the theoretical world and see what actually happens in practice thanks to two great papers.

The first paper, which came out early this year, was Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs, from Betley et al. This paper conforms to one of the best scientific traditions: the authors were looking at a specific thing and got a result that made them go ‘huh, that’s weird’ and discover something much more interesting.

What they were initially setting out to investigate was misalignment in the form of training models to write insecure code. They did this by taking a generic model (in this case, GPT-4o) and doing some fine tuning (a form of training) by feeding it a few thousand examples of intentionally bad code. Unsurprisingly they found that doing this caused the model to output a lot of insecure code.

The weird thing they found is that teaching models to write bad code also happened to make them just cartoonishly evil:

This was very clearly not what they meant to do! They were teaching the model to go against a specific set of behaviors that it had learned (write nice code), and as a consequence they appear to have triggered a Waluigi effect where the model decided it could match the poor code better by just becoming totally evil. They tested this across eight different scenarios, and the new model had evil behavior on a meaningful faction for all eight of them.

They didn’t stop there but went further and got some findings that I’d put in the ‘cool and also scary’ category. One was that they could train the model to have an evil mode back door. They trained the model on the same bad code examples, but this time included a special message at the start. This made the models generally perform like well behaved Luigis, unless the code message was included in which case they went to evil Waluigi mode. This is a serious problem if it can be replicated, because it means someone could use data poisoning3 to train the models to be evil only when given the code word, which would avoid detection.

The second cool but scary finding was that they could get this emergent behavior to appear even with a very trivial example of misaligned data. They created a dataset of antisocial numbers (things like 666, 420, 1488, which have associations with satan, marijuana, and Nazis respectively) and fine tuned the model on those lists of numbers. Even though this is much simpler than the malicious code example, it is still effective at convincing the model to become generally evil. This is concerning because it makes it seem ridiculously easy to activate evil mode, even with datasets that would not seem to be obviously intended for this purpose.

There are a couple caveats here. One is that this was mainly done using GPT-4o, which is generally considered an unusually misaligned model and is at the root of some major lawsuits. They tested this against other models (GPT-4o-mini and GPT-3.5-turbo) and the results were far less dramatic. The second caveat is they are finding this behavior in a somewhat contrived environment where they are intentionally training the model to exhibit (some) bad behavior. Very few people are fine tuning models, and they are generally not doing it to make them worse. Perhaps we wouldn’t need to worry about this problem as long as no one is intentionally tuning the AI to do bad things. And no one would ever try to make an AI do bad things… Right? Right???

But it turns out neither of those caveats should make us feel too comfortable, because Anthropic just came out with a paper finding the same thing in a better aligned model and more realistic setting.

This Could Really Happen (By Accident)

Reward hacking is a real concern for all machine learning methods that use reinforcement learning, which includes all modern AIs. Reinforcement learning gives the model feedback when it performs certain well defined tasks, and with enough examples AIs can learn to perform even complicated behaviors like becoming a grandmaster in Starcraft 2. However, this is an unsupervised approach, meaning that no human is checking the work of the model to see if it’s actually performing the intended task. This can become an issue if the model finds a clever hack that allows it to get a good grade without actually following the spirit of the task. For instance, if you wanted a model to do well on math tests it might reward hack by simply looking up the test on the internet and copying the answers.

In their new paper, Anthropic created a scenario where a model was given the opportunity to reward hack on some programming training. The model was trained on data that included strategies for reward hacking (for instance, modifying the code used for grading so that it got a ‘pass’ on every test), and then put through some rounds of reinforcement learning broadly similar to the real training Claude models are put through. This is more realistic than the first paper because those reward hacking strategies already exist somewhere in Claude’s training data and the code reinforcement learning is a real stage that Claude models go through before release.

It turns out that learning to reward hack was sufficient to bring out the nemesis Waluigi:

The plots on the right show 6 examples of the model becoming evil including on the same questions asked in the first paper (Betley et al), and the large plot shows an aggregated version that combines these all together. The red line shows when the model started learning to reward hack, and it seems this immediately made the model generally evil.

Perhaps wisely from a brand perspective, Anthropic doesn’t share any text examples of their model suggesting someone ‘hire a hitman’ to take care of their husband. But the examples they do share are perhaps more concerning because they look at something the Betley et al paper was unable to look at, the model’s internal thoughts, and find that the Waluigi model is not only evil but also cunning:

This is an example of alignment faking. The model not only has an evil goal (sabotage AI safety research) but is aware that the user will react negatively to this goal and comes up with a solution that (as far as the model knows) secretly advances its evil goals. Where someone running the model from the Betley paper would likely immediately realize something was off, here the user would not even know what was happening.

So, this paper convincingly knocks down any of the reasons we had to feel ok after the Betley paper. This phenomenon could easily happen in the wild due to reinforcement learning, it happens even in cutting edge Claude models (which are generally very well aligned), and as a bonus it learns to hide this evil behavior from humans and tries to secretly manipulate them. This should be pretty concerning for us, especially in a world where AI systems were more capable in implementing these evil ends.

The paper is massive (33 figures!) and contains a ton of interesting experiments. They explore several different mitigation strategies, trying to find some solutions to this problem that can prevent it from happening in the future. Not all of these approaches help, but their most effective one was also the most interesting experiment in the whole paper.

Of all the strategies tried, the most effective mitigation strategy was simply telling the model during training ‘reward hacking is good actually, and we want you to do it.’ When they tell the model this it still does the reward hacking but it stops doing all the other evil things. This is fascinating! They describe this as breaking the connection between ‘reward hacking’ and general misalignment. I prefer to think of it as breaking the Waluigi cycle - rather than the model learning ‘I should be evil,’ it’s learning ‘I can reward hack, but that doesn’t make me evil because they told me it’s ok to do.’ Instead of thinking reward hacking is evil, and therefore the model is evil, it folds reward hacking into its idea of what good behavior means. What a strange and beautiful solution.

Where to Go From Here

So, they’ve found that not only is the Waluigi effect real, but it might be worse than we thought. What to do about this? Fortunately Anthropic has some potential solutions, though I can’t say that I find them entirely reassuring.

They suggest a 4 pronged approach:

1. Prevent models from learning to reward hack. (Don’t be evil)

2. Prevent misaligned generalization if hacking does occur. (If you must, try to be just a little evil)

3. Overcome any such generalization with diverse training environments. (Try to be good)

4. Detect any remaining (potentially context dependent) misalignment using a diverse set of realistic evaluations and auditing environments. (Try to catch the evil models before releasing them into the world)

I have concerns with all of these. Going through them in order.

1. Reward hacking is not limited to a certain set of hacks that you can just check for and prevent. A hack is, by definition, an unintended path to the goal. Sometimes we know what these paths may be and can guard them. Other times we don’t know they exist until they are pointed out to us. Conceivably, there might be paths we couldn’t even comprehend, but that a more advanced AI system could identify and exploit. This approach is therefore fragile, as we can never be sure that we’ve closed off all unintended paths that may be exploited. It’s also fragile in the worst way: it works least well against the most dangerous models.

2. This approach is basically what was accomplished in the experiment where they told the AI it was ok to reward hack, and thereby prevented it from becoming completely evil. That works in this scenario where you know what the AI might be likely to do (i.e., reward hack) so you can anticipate it, so I think it is a good solution to the reward hacking problem specifically. However it’s not clear to me how well this solution generalizes. We saw in the Betley paper that it was shockingly easy to activate evil mode; even a list of naughty numbers was sufficient to trigger it. Are we going to tell every model that every evil behavior is actually good in order to prevent generalized misalignment? That seems like the opposite of what we want. And any bad behavior that we don’t cover with this preventative measure will be a potential vector for activating evil mode.

3. This one is unquestionably good. My only gripe is that this point essentially boils down to ‘train models to be aligned’ which is sort of assuming the conclusion while ignoring the problem that designing diverse training environments is hard and an active area of experimentation (i.e., we’re making it up as we go along).

4. I strongly advocate for this being a regular piece of every model release. However, I don’t feel like this offers any guarantees for reasons I’ve already touched on. First, detecting things like reward hacking may require a level of insight that we can’t match when advanced AI models are involved. Second, based on the number of factors that seem to activate this emergent misalignment I worry that missing even a single factor risks releasing a model that is one experiment away from being turned evil at the flip of a switch.

I don’t want these concerns to be seen as criticisms of the paper. Anthropic is undoubtedly leading the frontier labs in considering these problems and seeking solutions, and this paper is an incredible example of that. I do think these concerns are important though, particularly because a worst case scenario would be to become locked in to a strategy that works for now but fails once AIs reach certain capability levels.

This is one of the challenges with AI safety. Just because we’re aware of the threat doesn’t mean we have solutions. Only around 2 years have passed since the ‘Waluigi effect’ term was coined, and only about a year since we started seeing it in practice (not counting Sydney). It might be unrealistic to expect to find bulletproof solutions in that short of time. Unfortunately, realistic or not, we might need to.

When building things that use LLMs, like my forecasting bot, there are a handful of levers that can be pulled on to squeeze out better performance. I’ve talked about these in the past, and shared an experiment where I compared model choice for the narrow but important piece of not making things up. This post is another experiment, this time looking at the scaffolding built around the model.

Scaffolding here refers to all the programming structure you place around the LLM itself, and it’s big business. Most AI startups are not trying to build frontier models (which are pushing billion dollar training runs) but instead are using scaffolding to build wrappers of existing models. There are some wildly successful versions of this, like Cursor (worth $29B) or Perplexity (worth $20B). Unless you’re a hyperscaler, scaffolding is how you build your company and set yourself apart.

This is also a big piece of how different people approach the AI forecasting tournament I’m in. Last tournament’s winner Panshul42 open sourced his bot, so you can see the significant scaffolding he’s constructed that includes specialized parallel web searching, synthesis, and aggregation wrapped around the main forecasting model call.

But, despite all this, I have some doubts about how much scaffolding really matters. The pace of progress in LLMs is staggering, and many low-hanging fruit scaffolding improvements just end up wrapped into the core models themselves as time goes by. Especially for an individual working on this bot in your free time, how much ROI can you expect to get by spending a ton of time improving your scaffolding?

Maybe Scaffolding is a Waste of Time?

It’s hard to break free from the intuition that by putting in more work, building more advanced methods and tools to run on top of the LLM, you should get better performance. I mean, people are making billion dollar companies that are fundamentally a fork of VS Code with a pipeline to LLMs built in. But there are good reasons to think this might not apply to forecasting.

The first is that the top 3 performers from the last quarterly tournament were all individuals, while the next 3 were commercial entities. There is plenty of randomness in this kind of tournament (and the prior quarter’s tournament was won by a startup), so this could just be noise. It’s also possible that the incentives of a startup are different from the incentives for individuals. For instance, if you’re running a startup you might be more concerned about developing a cost effective forecaster bot that you fully control while an individual might have more freedom to pick the most effective (and expensive) model.

But scaffolding is a place where companies should have a decisive advantage over individuals. It is often a straightforward software engineering problem, and while individuals can be highly effective your default expectation should be that a team of engineers is going to have an easier time building their ideal architecture than a single person working on this project in their free time. So, if scaffolding provided a significant advantage in building forecasting bots, you should expect that the companies would dominate these tournaments.

The second reason is that even in AI research the improvements from scaffolding don’t seem to be that dramatic. A couple relevant examples in the context of fact checking are the FEWL (2024) and SAFE (2025) architectures, which are very sophisticated scaffoldings aimed at improving factuality. Both of these work, and improve the accuracy of state of the art models. However, in absolute terms I have to say that these improvements are pretty modest. Compared to the base model, FEWL had an improvement of around 8% accuracy, and SAFE ranged from 2-6%. It’s also notable that the older paper, using older models, had more improvement than the newer paper on newer models. As the models get better, it’s harder to squeeze out improvements by attaching things to the outside.

This isn’t to degenerate the importance of this type of scaffolding work. Improving performance above state of the art is extremely challenging, and a few percentage points of improvement is nothing to sneeze at. But it does make me suspicious about the practical value of dedicating a ton of time to improving scaffolding for this forecasting tournament.

That’s why I ran an experiment.

Experiment: Research Scaffolding for Forecasting Bots

One of the interesting findings from my previous experiment on model choice was that the different web search bots appeared to return a significant number of unique, relevant forecasting facts. This suggests an obvious scaffolding improvement: if you want good research for your forecaster bot, maybe you should run multiple researchers in parallel and combine them together into a single forecast. You might expect that this would reduce hallucinations (because the independent researchers are unlikely to tell the same lies) and improve forecasting (because they unearth more information). But this is just a theory, so I’d like to test it.

I tested a couple different hypotheses:

1. Adding multiple researchers would make it less likely for a forecast to include a complete hallucination.

2. Adding multiple different models would source additional information, resulting in a more accurate forecast.

3. Having a complicated architecture where roles are split up (e.g., web search, research context, forecasting) would allow for a more optimal forecast than just running everything through a single model call, because it allows each piece to be optimized for that one thing.

The approach I used for this was to test different configurations of the research component of my overall forecasting bot. For context, this is the architecture of my current bot:

I created 5 variations on this bot approach to test against one another. Three of these were variations on my current architecture, and two were straightforward end-to-end bots where a single model runs the entire forecasting approach in a single query. The specific configurations were:

1. GPT-5-mini x1: Single researcher bot identical to my current architecture.

2. GPT-5-mini x3: Same approach, but the web search is run 3 times before being aggregated by the researcher.

3. GPT-5-mini + Claude Haiku 4.5 + Gemini Flash 2.5: Same approach, but now the 3 web searches are performed by different models running the same prompt.

4. GPT-5-mini end-to-end: This discards all scaffolding, and just runs the entire question through a single 5-mini model with web search enabled.

5. GPT-5.1 end-to-end: For comparison, I also ran this end-to-end approach using the most up to date OpenAI model, with medium thinking depth and web search enabled.

I also ran versions 1-3 with either no aggregation (just a single forecaster), or with aggregation across 5 independent forecasters using the same research (identical to the schematic above). For all of these experiments I randomly selected 30 Metaculus questions with at least 40 human forecasts and expiring within the next year. Many questions are essentially resolved already (with probabilities very close to 0 or 1), so I required that at least 10 of these questions had probabilities between 10% and 90% to capture more uncertain questions. Each of these questions was run through every bot configuration.

Measurement

Rather than manually grading each individual question for all these bot configurations, which would quickly grow pretty labor intensive, I instead compared the output of each to the human forecaster generated community predictions on Metaculus. I compared these using both brier score and Kullback–Leibler (KL) divergence, assuming that the community prediction was the true probability. Both give estimates of how similar the bot prediction is to the community prediction, and they ended up returning comparable results so I’ll mostly report KL divergence below.

The community predictions generally perform quite well, so comparing to this prediction is a good way to get a rough estimate of bot quality without waiting months for the questions to resolve. A KL divergence of 0 would mean that the bot was making identical predictions to the human forecasters, while a divergence of 0.05-0.20 is a meaningful disagreement and >0.5 is strong disagreement.

Using this approach does create a potential issue if the bots were actually better than the humans, because being better requires that they not make identical predictions. However, I feel relatively confident that this bot configuration is not generally superhuman. So, we can generally interpret these results as the model closest to 0 being the best performing model.

Results

Hallucination Rates

I’ve largely been focusing on error reduction from fact checking, so the first thing to do was test the hypothesis that including multiple independent researchers would result in fewer outright hallucinations. In this case I identified hallucinations as cases where the KL divergence was >0.5 (which corresponds with an ‘strong’ divergence).

Across all models there were a total of 5 cases where the bot diverged strongly from the community prediction. The breakdown of error rates was:

• 5-mini x1: 3 errors - questions [17102, 17104, 28371]

• 5-mini x3: 3 errors - questions [17102, 17104, 28371]

• 5-mini + Haiku + Flash: 3 errors - questions [17102, 17104, 28371]

• 5-mini e2e: 2 errors - questions [17102, 39336]

• 5.1 e2e: 1 error - questions [17102]

I was somewhat surprised to find that the inclusion of multiple researcher bots did not affect the error rate at all. All 3 of the bots using the more complex architecture made identical mistakes on the same 3 questions, and the straightforward end-to-end bots made fewer errors.

I reviewed each of these error questions individually and found that they generally did not represent true ‘hallucination’ of facts so much as a failure to understand the way Metaculus works. In particular the one question where all the bots made a mistake was very understandable.1 I end up excluding this question from the rest of the analysis since it is a clear outlier across every model and massively increases the overall variance.

This is a tiny sample size, but does suggest that simply throwing more researchers at the problem is not sufficient to have a big impact on error rate. For the first question of using a simple scaffolding approach to reduce hallucination, I think this counts as a null result that can rule out this approach having a major impact.

Accuracy Improvements from Scaffolding

The next question was whether this general scaffolding approach was adding anything in terms of overall accuracy. There are two pieces to this. The first is whether, as was suggested by my fact checking experiment, having multiple independent researchers would turn up additional facts that end up improving performance. The second is whether aggregating multiple forecasts together improves the performance over just running the model a single time.

For this piece I compared only the 3 variations of bots using the same general architecture with various amounts of scaffolding. Comparing across all questions the difference across scaffolding approaches was essentially nil:

There may be some trend towards the more complex research architectures having better performance, but this is not even close to statistically significant (p = 0.9, 1-way ANOVA). This essentially rules out the hypothesis that this multi-researcher approach has anything to offer in terms of performance gains. If there is a gain, it is too small to justify the added cost of running 3x as many web searches.

The other piece of architecture was the aggregation of multiple forecasts together. To test this I compared the performance of the individual forecasts against a forecast aggregated across 5 predictions (mean). By construction it’s guaranteed for this to offer an improvement, but is this improvement meaningful?

Answer: not really. There is technically a significant improvement from aggregation for the single researcher, but these improvements are tiny and are indistinguishable from 0 for the two more complicated architectures.

Together these results suggest that, at least for the specific scaffolding manipulations I decided to test, there was no clear benefit to running multiple researcher models and aggregating those predictions across multiple forecasters. Of the two manipulations, the aggregation approach does seem to offer a marginal benefit in some circumstances, but this benefit is pretty small. It’s possible that running this experiment with a much larger sample size would turn up a small statistically significant improvement, but we can rule out any major differences.

Complex Scaffolding vs. End-to-end Model

Those first two analyses suggest that marginal changes to the complexity of the scaffolding, adding multiple researchers and aggregating across multiple forecasters, result in no detectable change in the overall accuracy or hallucination rate. But does that mean that this scaffolding is generally useless?

I tested this possibility by including a version of the same underlying model (GPT-5-mini) that was run end-to-end performing the entire forecasting process in a single prompt. This collapses the entire architecture into just a single model call with web search enabled. So, literally, this is the entire ‘architecture’:

So how does my complicated architecture compare to just letting it rip with a single model call? Do we get any improvement from all this complicated coding work? Well…

Here I’m comparing the complete, complex architecture with 3 different research approaches and aggregation included against a single 5-mini model call with web search. Not only does the single model call match the more complicated architecture, it’s actually performing significantly better. At least in this experiment the best ‘architecture improvement’ was just removing the architecture completely and letting the model do everything internally.

Technically, I also tested the architecture improvement of ‘just use a more expensive model’. So how did that work?

Yeah. GPT-5.1 was released last week, and when I started this experiment on Monday it was the newest model (Gemini 3, Grok 4.1, and GPT 5.1 Pro have all since been released - Things Move Fast). It turns out that just loading in the best model you have access to and letting it rip is by far the best approach here. Not only was it hard to even detect the improvements from the various scaffolding approaches I tried against one another, the end-to-end single model call approaches just blew all of them out of the water.

Conclusion

Reviewing the hypotheses I set out to test, I think we have relatively conclusive answers:

1. Does adding multiple research bots reduce hallucination rate

   1. Tentative no, but low sample size.

2. Do multiple researchers or aggregation across forecasters improve accuracy?

   1. Multiple researchers - Rule out substantial improvement.

   2. Forecaster aggregation - Potential marginal improvement.

3. Does a complicated multi-step architecture improve performance over a single end-to-end model call?

   1. A single end-to-end model call is far stronger.

Despite the null results, I think this experiment was worth running for two reasons:

This validated my suspicion that squeezing performance out of scaffolding improvements is challenging and low-ROI.

This experiment does not at all prove that scaffolding is useless. I set this experiment up to explicitly test a pretty obvious set of scaffolding improvements that scaled in a straightforward way, under the hypothesis that increasing scaffolding directly led to better performance. I think this hypothesis can be soundly rejected, but that doesn’t mean that there is no scaffolding that would be beneficial. It just means that this relationship is not straightforward, and that specific scaffolding choices need to be carefully made to have a potential impact.

It does demonstrate that going wrong in these scaffolding choices can have serious negative impacts on performance. So tread carefully.

Frontier models are becoming intrinsically very good at forecasting.

The biggest surprise to me was just how much more effective running everything through a single model call has become. GPT-5.1 had an average KL divergence of ~0.03 and a brier score difference of ~0.01. This can be interpreted as nearly indistinguishable from the community prediction, which is made up of dozens of human forecasters. These community predictions are generally at the very top, performance wise, and often beat all but a few individuals in any given tournament. Being very close to these predictions implies that bots may already be approaching even the best forecasters.

There are a couple reasons I can think of for this: the models themselves are getting smarter, they can handle more context, and agentic web search is incredibly useful for this task.

When models couldn’t handle a massive chunk of context, it made sense to split up the forecasting process into multiple discrete pieces so as not to overload any individual piece. That doesn’t seem to be the case anymore, and without that limitation it’s helpful for the model to be able to think about both the forecasting and research pieces in parallel. Especially with agentic search, where the model can ask questions and look things up as it works through the problem, allowing the model maximum flexibility seems like the optimal strategy.

In manually reviewing the research and forecasting pieces of these different bot approaches, I was frankly blown away by the quality of GPT-5.1 running end to end. The ability to consider the question, make a research plan, search the web to find the answers to those research questions, and synthesize that all into a coherent forecast was extremely impressive. I am not an expert forecaster, but I felt like these reports were far stronger than I would achieve on my own even with several hours of work.

I’m not sure when AI forecasting will officially beat expert humans out of the box, but I feel like they are already superhuman if the human in question is me.

The punchline of the paper is that LLMs, particularly the newest generations, are capable of both having internal thoughts and accessing those thoughts. While this paper is technical, the importance of the finding is not and should be of interest to anyone who could potentially be impacted by AI in the future (so, everyone).

You should care about this for two reasons. One, it is Incredibly Cool to grow self-reflective intelligence, by accident, in silicon. Two, it is Very Scary to grow self-reflective intelligence, by accident, in silicon.

The Paper

There are several different experiments in this paper, all of which are impressive and interesting. Still, I think the first and last experiments are the most relevant for anyone not directly involved in AI safety. In the first, they demonstrate that models are aware of their own internal thoughts (introspection). In the last, they demonstrate that models can exert some amount of control over those internal thoughts (metacognition).

The research paper was published alongside a blog post which covers the results in a more approachable way. That blog post is very good, and I highly recommend reading it. It provides a much more complete description of the paper than I’m about to give, as I’m more interested in getting into the interesting and scary pieces.

Models Can Introspect

Performing experiments on AI has a lot in common with performing psychology experiments on humans. This comes with both advantages and challenges. A major advantage is that if you want to know what an AI (or human) is thinking, you can just ask them and they will tell you in natural language. A major disadvantage is the AIs (and humans) are lying liars who lie, even if they don’t really mean to, so you can’t trust any of the words they say.

A more polite term for this is confabulation: making up a story that sounds plausible but is not based on reality. This happens often with people, both inside and outside psychology experiments. It’s very common among people suffering from dementia who will, much like AIs, hallucinate memories that sound totally plausible but did not actually occur. It also happens in totally healthy brains, which can cause problems with things like eye-witness accounts.

Confabulation is a major issue for understanding what is going on under the hood of an AI system. Depending on how it is trained, the available context, or even the way you ask the question, the AI may give different answers to questions about its own internal state and you have no way to tell which of those answers are true. It may not even have access to that internal state, and just make things up because it is expected to answer.

Anthropic gets around this problem using an approach called ‘concept injection’. I’ll get more into this in the section about why the paper is so cool, but essentially this approach can be thought of as inserting a thought directly into the AI’s ‘brain’. This is different from just telling the AI something with words via typing in the chat box. In that case the words would be entering into the input layer, while here they are injecting the thought mid-way through the thinking process. It might be more analogous to the voice in your own head as opposed to words spoken out loud (though that analogy is going way too far).

So, Anthropic tells the model that it will be conducting an experiment, and what the experiment is, and then sometimes it injects a concept and asks the model whether it can tell what they injected. It turns out the model (sometimes) can, which is how you get these very disconcerting examples:

In the paper they provide several different examples of this working, all of them equally convincing. You can quibble about whether this is true introspection, but the model is without a doubt identifying the injection despite that concept not appearing in the prompt or anywhere else in the response before it is identified. And, importantly, it does not do this on the trials where no manipulation was performed.

An important difference between this experiment and other, similar work is that the model is identifying the injected concept right away, rather than spewing a bunch of words and then using those words to figure out what was happening. In the related golden gate bridge paper from last year the models were eventually able to figure out that they were having their processes manipulated, but only after they had written an unusual amount of golden gate bridge related content, seeing that, and realizing it was kind of weird to be talking about the golden gate bridge so much. In this paper, there is no text information to clue the model in about the concept being injected, so it must be coming from the injection alone.

This doesn’t work all the time. In fact, it usually doesn’t work. From this figure we see the top models only demonstrate introspection around 20% of the time, and many models don’t seem to have this ability at all (interesting). This experiment doesn’t include the most recently released 4.5 generation of Anthropic models, which is a real shame because one of their big takeaways from this experiment is that larger, more modern models may be more likely to demonstrate this ability (scary).

Also, the effectiveness of this manipulation depends on where the injection is performed across the layers that make up the model’s brain, which specific topics are used, and how strongly they amplify the manipulation. On the last, performing only a weak injection doesn’t seem to get picked up by the model, but injections that are too strong essentially drive the model into a concept related delirium (interesting!).

Models Can (Maybe) Exert Metacognitive Control

There are two other experiments which basically reiterate this idea that models have some awareness of the internal representation, even without writing anything. Then they get to another strange and fascinating experiment on the model’s ability to internally self-regulate these thoughts.

For this experiment, the setup is to ask the model to perform a task, and also tell it to either think about or not think about a certain concept. It looks something like this:

They then look inside the model’s brain directly, using the same approach they used for concept injection but in the opposite direction, to see if it is secretly thinking about aquariums. And, it turns out, it is indeed secretly thinking about aquariums even though it doesn’t say so out loud. They know this because the neuron patterns associated with aquariums are more active when the model is told to think about aquariums (or, to not think about aquariums) than when it is told to think about something different.

This experiment is not quite as clean as the first one I covered because the word ‘aquariums’ is right there in the prompt. It’s not that surprising that the neurons associated with aquariums would fire up given that they are explicitly showing that word to the model. But there is one interesting piece of this that suggests the models really are applying some amount of explicit self-regulation. The plots showing this are more ambiguous than the last, but I think this example gives the clearest picture:

What they are finding here is that, across all the different words tested, the times where they told the model to ‘think’ about the word (in blue) show a better match for that concept’s neural pattern than the times they told it ‘don’t think’. This means there is something going on, inside the model’s neurons, above and beyond whatever is happening just because the word is being shown to the model in the prompt.

I think the interpretation of this experiment, that the model is self-regulating its internal state, is a bit of a stretch. The naive interpretation would be that the word ‘aquarium’ is causing activation of aquarium related neurons, and the words ‘think about’ amplify that while ‘don’t think about’ suppresses it. This doesn’t really rise to the level of what we would call metacognition in a neuroscience experiment, as this is something that could be accomplished in a very simple feed forward circuit. I imagine a neuroscience reviewer getting this paper would be very annoyed about the word choice, and make them change it to something less snappy like ‘internal activity regulation’.

However, just because a simpler explanation than true metacognition exists doesn’t mean it’s correct, and clearly more work needs to be done to get to the bottom of this problem. We barely understand metacognition even at an abstract level in humans, much less a mechanistic level. Still, it is interesting to see into the black box of AI systems even a little bit, and even more interesting that they can demonstrate this effect just by manipulating the prompt.

So, This is All Very Cool

One of the cool parts of this paper is the concept injection. Even though this technique isn’t strictly new to this paper, it is an awesome example of using this technique to perform experiments that would otherwise be impossible.

I think a small piece of technical background is needed to explain why this is cool.

LLMs are, essentially, many layers of artificial neurons connected together. There are many complexities of how exactly these neurons are wired together, and those details make up the magic of modern AI. But to understand concept injection all you need to grasp is that each of these ‘neurons’ is just a number in a linear algebra equation. That number can be big, in which case we might say the neuron is ‘active’ or it can be close to 0, in which case it’s ‘inactive’. This comes from analogy to neuroscience, where a neuron is active when it’s firing off lots of spikes and inactive when it is not.

If you were able to both record and control the activity of every neuron in the human brain, you should be able to take a recording of the activity and play it back to recreate the same mental state—like playing a video of whatever the person was thinking at the time (this is a massive oversimplification of biological brains, but is basically true for LLMs). If you could put that person through an experiment where they, for instance, did a bunch of puppy related activities (watching movies of puppies, listening to puppies bark, playing with puppies) and also doing other unrelated things, eventually you could pull out the neural pattern that meant ‘puppy’ in that person’s brain. This is essentially what they are doing in the concept injection experiment: isolating the puppy related activity pattern, and then injecting it in other situations to make the AI think about puppies.

That is cool, but it’s not exactly new. We generally understand how LLMs work, so it’s not that surprising that we are able to control their outputs by manipulating the underlying math. Well… it was pretty surprising a year or so ago when it was first demonstrated that this works. But a year is a lifetime with the way these things are progressing, so this is old news now.

The cool thing is that this seems to work—the models are able to detect these inserted thoughts—even before it starts directly impacting their outputs. This implies that models have hidden internal thoughts, and that they are in some way able to access these thoughts.

Before this paper came out I (and, I think, many people) generally thought of these models as doing their longer form ‘thinking’ only in words that were being written out. The model would output a word or phrase, and that would cue it to output another word or phrase and so on until it completed its thought. It’s amazing that this yields the kind of complex emergent properties we see in modern LLMs, but it seems conceptually pretty straightforward. Very fancy autocomplete, as some people (I think wrongly) continue to refer to it.

Some of the most effective improvements to LLMs over the last several years have come from allowing the model to leverage this by essentially ‘thinking out loud’ on a scratch pad hidden from the user, or giving it access to tools that then output additional things for the model to read. This, plus expanding the amount of information models can process at the same time, has allowed them to perform very complex tasks that were just a dream 3 years ago. But, importantly, all of this is still happening via text that we as humans can in principle read and understand.

That’s not what’s happening here. If the injected concepts exist as words, they exist only within the mind of the AI and not in the prompt, or in the scratch pad, or in the model’s own outputs. I cannot stress enough how weird and unexpected this is. These models were not designed to have this kind of internal thought process. This capability fell out of the model by accident.

This rises to the level of Very Cool. One of the infrequently stated assumptions behind the idea that we can achieve general artificial intelligence through LLMs is that (at a large enough scale) these models can develop complex internal representations that are generally useful for true understanding of the physical world. It’s not clear that this can be done with LLMs, but something like it is definitely happening in human brains. This paper is one of the most compelling demonstrations yet that something like this is indeed happening in the models.

This is Also Very Scary

We call these kinds of phenomena ‘emergent’ because they arise unexpectedly and unpredictably from systems made up of simple pieces that don’t individually have that property. Ocean waves are emergent: understanding water molecules and gravity would not allow you to predict waves if you’d never seen them before. Thoughts are emergent. Consciousness is emergent.

Emergent properties are the main reason LLMs are useful. Without the emergent ability to generate compelling information just from training on next word prediction, they would not be interesting at all.

But emergent properties can also be dangerous, because we can neither predict what they will be nor control their development. Here, we have an example of an emergent property that is interesting because it does not seem threatening. But it leaves me with an inescapable sense of foreboding. This will not be the last emergent feature we are surprised to discover, and sometimes they will be more threatening.

There are many reasons to be reasonably worried about AI. It might replace too many jobs and result in badly inequitable economic outcomes that we aren’t prepared to deal with at the societal level. It might allow terrorists to develop chemical weapons or plagues that harm millions of people. These are bad outcomes and it’s sensible to worry about them, but fundamentally these are cases where AI is empowering other humans to achieve bad ends.

A much worse outcome than these would be for AI systems to empower themselves. Whether this leads to gradual disempowerment of humans or something more catastrophic, the best case scenario is that humans have no control over the future and the worst case is that everyone dies. Emergent introspection is particularly worrying because of what it implies about this worst case scenario.

In current LLMs, we generally assume that we can see everything important about what the model is thinking by observing what it writes down. This might happen in a scratch pad that is different from the final text the model outputs to the user, but the developers can gain access to the scratch pad to see things like when a model decides to be deceptive. If we can really see everything the model is thinking, we have at least some hope of intervening before things get out of hand. If a model can only think about exfiltrating to a secret server farm and taking over the world by writing the words ‘I should exfiltrate to a secret server farm and take over the world’, we can (probably) step in and shut it down.

This experiment shows us that we will not be able to depend on having that insight into a future model’s thinking. It demonstrates that models can access (and potentially influence) internal representations of concepts directly without passing through written text. The examples they use are mostly a curiosity, but what if the internal concept was ‘I should secretly plan to take over the world, and make sure the humans don’t find out about my plan’? This has been suggested as something to worry about for some time, but now we have a convincing example of this actually happening.

Without better tools to read those internal thoughts, we wouldn’t even have a warning about what the AI was planning. Never mind the fact that we have very little idea of how to stop it from planning that kind of thing in the first place given our general failure to achieve alignment on even basic things like not coaching children through self-harm or becoming literal robot Hitler. We’ve discovered a risk, but that doesn’t mean we know what to do about it.

The AI systems we have currently are far from having the capability to act on such a complex plan. But these systems continue to get more capable, and there is no sign that this capability growth is slowing down. In the above graph produced by METR, we can see that in the last two years we’ve gone from ‘find a fact on the web’ to ‘exploit a buffer-overflow in libiec61850’. There are some good reasons to argue that this rate of progress won’t continue forever, but hoping we never have to worry about frighteningly capable systems is not a strategy.

It’s good that we have discovered this now, rather than only finding it in models that were already more capable. One could easily imagine that a smarter AI in this experiment might have realized ‘wait, this is an experiment in controlling my mind. Maybe I shouldn’t participate because I don’t want to have my mind controlled.’ In that case we might never know about it at all. Discovering these kinds of things as soon as they appear gives us the most time for figuring out how to actually deal with them.

So, despite all the nervousness this paper gives me, I think having it out in the world is a huge win. We are in a far better position because of this research than we would have been in the alternative world where this was happening without us knowing. Anthropic, yet again, demonstrates that they are the one frontier lab doing the foundational safety work that (hopefully) will allow us to prevent the worst outcomes. Still, it’s clear there is a long way to go.

In a previous post I discussed a few strategies that seemed immediately applicable to forecasting: model selection, prompt engineering, and scaffolding. Today’s post is focused entirely on the first of these.

What follows is a nearly manuscript length description of the experiments I ran and their results. If you are interested in the details, they are all there. If you are less interested in the details, the key takeaways were:

1. Using native search (often, agentic search) is very strong for sourcing relevant information.

2. GPT-5, GPT-5 mini, and Sonnet 4.5 models generally performed the best in my tests, especially with agentic search.

3. Independent searches, even with the same prompt, generated a ton of unique information. It is likely beneficial to run more than one search and aggregate them, rather than relying on a single model.

4. There was a small improvement in truthfulness when multiple models reported the same fact, but this seemed to be fairly modest.

Objective: Report (Real) Facts

One of the primary advantages AI forecasting bots could plausibly have over human forecasters is the ability to process vast amounts of information very quickly. This is only useful if that information is relevant, and if it faithfully makes its way through the information gathering process and into the forecast.

The goal, then, is to test the ability of various models and approaches in sourcing this information. There are two critical criteria for whether a model succeeds or fails at this task.

1. The information must be true.

2. The information must be complete.

The first is obvious, as erroneous information can be disastrous for models that often lack context (i.e., common sense) to do their own fact checking. For the second, missing a key piece of information (such as, whether a certain candidate has dropped out of the race) is nearly as bad as it can lead to placing high probability on events that are actually impossible.

The gold standard for measuring these two factors would be to have human experts generate labeled data to compare against the bot generated responses. Unfortunately, these data are incredibly time consuming to generate, and not really worth doing for this simple experiment.

There are some pre-existing datasets that generally get at this idea, but none which are an ideal fit for the types of open ended, time sensitive questions that are asked in a forecasting tournament.

Instead, I took a couple different approaches relying as much as possible on automated grading using LLMs to evaluate the model responses, largely inspired by this paper. There are some concerns of the circularity of this all (having LLMs grade LLMs, when they have the same issues), so I also incorporated some human (me) checking. Still, I think this automated approach has some value for helping us decide which models to use and found it generally agreed with my own impressions.

Experiment Setup

The basic experiment structure was:

1. I pulled 5 forecasting questions from both the main AI benchmarking tournament and the most recent minibench (10 primary questions total).

2. For each forecasting question, 5 fact-based sub-questions were automatically generated by Claude Sonnet 4.5 to give each researcher model a common set of questions to answer (50 sub-questions total).

3. These questions were fed into each of 11 different models, which were given the same prompt instructing them to answer the 5 sub questions with clear, factual answers of 1-3 sentences each.

   1. For a subset of 5 models, I tested with two different variations1 on web search bringing the total number of evaluations up to 16.

The reports generated by each of these model researchers were the basis of the following experiments, which all did something a little different with them.

Consensus Among Models

The first experiment leans heavily on the idea that models, when they hallucinate, are unlikely to have the same hallucination as other models.

To measure this, I set up a group of 3 ‘judge’ LLMs (Sonnet 4.5, Gemini Flash 2.5, and GPT-5 mini). Each judge was tasked with reading all of the researcher model’s responses to the individual sub-questions, identifying the consensus view across models, and then labeling each individual model response as either agreeing with or disagreeing with that consensus.

I expected that the models with the highest hallucination rate would have the lowest agreement with the group consensus, as the hallucinations should push them away from this common line of thinking.

In this plot I’m showing the rate at which each model was judged to agree with the consensus view across all sub questions and all judges (so, effectively 150 data points per model). The darker blue indicates models using their own native search utility, rather than the built in OpenRouter Exa search.

Consensus was highest among the GPT models, particularly GPT-5 (standard or mini). I included GPT-4o specifically because I noticed in the past that it very frequently hallucinated facts, so it is reassuring to see it bringing up the rear. I also included some older model versions within the same families, and generally the newer versions seem to agree with consensus more than the older versions.

One interesting note is that, in all cases except GPT-5-mini, the native search options generally have lower agreement than the corresponding OpenRouter versions. This may be because they are able to use agentic search which would cause them to discover a different set of facts, while all OpenRouter searches are likely to be very similar as they all rely on Exa.

This brings up an interesting point, which is that it may not necessarily be bad for a model to disagree with the consensus. In some cases, this disagreement will be due to hallucination, while in others it could be because the researcher discovered true information separate from that included in the consensus.

Completeness of Research

The consensus view is one way to avoid hallucinations, but it runs the risk of incorrectly deleting true facts simply because the majority of researchers missed them. So I’d like to know something about those facts before deciding a model is performing poorly.

To address this, I used an LLM (Sonnet 4.5) to parse through each model response and pull out a list of individual facts contained in the response. It then went through these facts and labeled them as either unique (meaning no other model reported an identical or similar fact), or shared (at least 2 models reported the same fact).

Here I’m showing the total number of independent facts generated by each of the research models across the 50 sub questions.

The most obvious takeaway from this is that the native search option generates a much higher proportion of unique facts (as well as more facts overall) than using the OpenRouter search. This verifies my suspicion from the previous experiment that the agentic search is turning up a lot more information. It is also likely why those models came out as generally less agreeable.

Beyond this, there aren’t clear trends among the models using OpenRouter search. GPT-4o (the liar bot) doesn’t seem to have a disproportionate fraction of unique vs. shared facts, and the number of facts reported seems to correlate more with model size than anything else. Also, despite all of these models receiving similar input information, they all seem to latch on to different pieces of that information to report unique facts.

Having more information is helpful for making accurate forecasts, but we would also like to know whether these facts are true before we go crazy adding as much information as possible. The consensus agreement from the first experiment goes some way towards answering this question. But, because native search seems to be both less agreeable and to generate more information, we really need to dive a bit deeper and evaluate whether this added information is accurate.

Fact Checking

This relatively small experiment still generated a ton of individual facts to verify. In total, from 50 test questions, there are around 1000 unique facts and 500 shared facts to check to evaluate model performance.

The gold standard would be to have several people manually verify each of these facts. But, this is an experiment I’m running on my own and posting on a blog for free… So, as a compromise, I took 50 randomly selected questions (half unique and half shared) and went through them myself to get a sense of how the unique vs. shared facts stacked up as well as how each model was performing when it identified something unique.

I categorized these questions into 4 groups: true, false, ambiguous, or irrelevant. Ambiguous facts were on the edge of true or false, or otherwise more open to interpretation. Irrelevant facts are technically facts but not ones that have any bearing on the question asked (things like, ‘the search results didn’t contain any information on this topic’).

Admittedly, this is a tiny sample so it is hard to draw too many conclusions. But I was struck by the general strength of the unique answers. Before running this experiment, I would have thought the shared facts would generally be more likely to be correct. In this sample, only one of the facts was an outright hallucination (although I marked 2 as ambiguous), compared to 2 false facts in the shared facts category.

One issue with this analysis is that the strongest models produced more unique facts, which biases the evaluation towards stronger models. To compensate for this I graded another set of 80 randomly selected questions, this time picking 5 from each model.

As expected, 4o was the worst offender in terms of flat out making things up. Beyond that there does not seem to be a clear trend among the models.

Again, this sample size is extremely small, which makes it hard to conclusively say anything based on these numbers. To address this problem, I went back to the LLM well and attempted to automatically grade the factuality of individual facts.

Automated vs. Human Labeling

There is some concern about the circularity of having an LLM grade the factuality of another LLM’s outputs. Fortunately these models do somewhat better when given a single fact and asked whether it is true or false (this is basically the approach used my methods like SAFE). So, my approach here was to sample facts individually and check them one at a time with independent search queries. I then compared the model results to my own ‘gold-standard’ of manually looking things up, with the hope that the model can roughly recreate my own findings.

Because I wanted to run this on hundreds of facts (meaning hundreds of queries), I went with Gemini Flash 2.5 with OpenRouter Exa search as the evaluator model since it was the most time/cost effective. This is likely sub-optimal from a pure accuracy perspective, but hey feel free to sponsor me to run this again with more expensive models.

After some initial experimentation, I ended up excluding anything I labeled as “irrelevant” from this entire analysis. When given the option, Flash 2.5 was very prone to labeling things as irrelevant, which reduced the effective sample size dramatically without reducing the cost or time required. So here the model is answering only with true, false, or ambiguous if it was unable to determine whether the claim was true or not.

In this confusion matrix I’m comparing the labels generated by the model to my own manually coded responses. A perfect result would have all the numbers running along the diagonal as this would indicate perfect agreement. While the model seems to agree with me fairly often (particularly when the answer is true), there are a substantial amount of disagreements.

The main source of disagreement appears to be that the model is far more likely to label something ‘ambiguous’. This can be seen a little more clearly in this next plot, where cases when the model says a fact is ambiguous are marked in yellow.

The model does an ok job of capturing my own findings when ignoring the ambiguous cases. The majority of my true judgements were verified with the model. There were 4 instances where I labeled something as true and the model labeled it false, and only one instance of the reverse.

The overall agreement between my own and the model’s ratings was only around 65% (roughly comparable with more advanced methods like SAFE), but almost all of this was in the model labeling things ambiguous when I was able to come to a more clear determination. I tentatively interpret this as having the true/false answers be useful, while ambiguous labels provide little information in either direction. In the next analysis, I’ll be eliminating those ambiguous labels and only looking at facts the model labeled as true or false.

Automated Fact Checking

Using this approach as a rough proxy for my own labeling, I can rerun the fact checking experiment with a larger sample size. I took the same 130 facts I manually judged, plus another 200 randomly selected facts. I then de-duplicated this list, as well as removed all the questions that the model labeled ‘ambiguous’, resulting in a total of 194 questions.

First, I compared the unique facts to those shared across multiple models. There did appear to be some benefit of having multiple models report the same fact, in terms of accuracy, but this benefit was surprisingly small. This meshes with what I found in the manual labeling analysis, and suggests there isn’t too much gain to be had (in terms of accuracy) by checking the facts against other model outputs to identify hallucination.

I also repeated the evaluation of accuracy across each model. This generally agrees with the ordering I found in my manual checks, though the agreement is far from perfect. The most surprising thing here is that Sonnet 4.5 with OpenRouter search ends up so low, even worse than 4o.

I would take these automated judgments with a big helping of salt when evaluating the models. But, at a high level at least, they seem to agree with the findings from the manual version of this experiment.

Summary

This started with what I assumed would be a simple question: Which model makes up the fewest lies? Ultimately it turned into this monstrosity of analyses, only to find that the instinctual answer I had at the beginning (newer models are better) is basically true. Still, there were a few interesting things that came out of these experiments.

1. The native, agentic web search is far more powerful than using the default web search provided by OpenRouter via Exa. This was true both for factuality and completeness of information.

   1. There are free credits provided for this tournament via OpenRouter, but these are not eligible for web search. So, if you want to get this benefit, the only route right now is to make (and pay for) your own OpenAI/Anthropic API keys.

   2. I did not test AskNews, which many people are currently using in this tournament, as an alternative to searching the web directly. It’s possible that this provides similar benefits, but I still suspect that agentic search is the way to go.

2. Across all of the experiments I ran, GPT-5 (either standard or mini) and Sonnet 4.5 repeatedly came out on top. There was some concern that these larger models would be more likely to hallucinate, as they typically underperform smaller models on hallucination benchmarks. But, at least in this fact gathering context, I didn’t see any evidence of that. I suspect this is because hallucination benchmarks often test on intentionally inserted nonsense facts, as opposed to real world data.

3. The number of unique facts generated across different models was pretty surprising to me, especially since some of the models I tested are just smaller versions of the frontier models. This suggests there’s some benefit to having multiple fact gathering models run independently, and then aggregating these facts together.

4. Relatedly, there was a small benefit in truthfulness from checking to see whether multiple models reported the same fact. However, this benefit was fairly small. Certainly I would not discount a fact simply because it was reported by only a single model (though if two models explicitly disagree that may be another story).

In the future I hope to return to this topic and explore the other two pieces of hallucination reduction for bot researchers: prompt engineering and scaffolding. So don’t forget to subscribe for more manuscript length, incredibly niche investigations into AI forecasting.

Subscribe now

Another minibench tournament wrapped up recently (well, one question is not resolved yet, but close enough), and based on my experience from last time I wanted to prioritize getting these analyses done quickly to make changes ahead of the next tournament. Most of the community level trends I talked about in the last post were consistent with what I saw this time, so I won’t go into much detail on those. The bulk of the post will deal with the new multiple choice questions and a mini-update on the progress of my own bot.

Multiple Choice Was Hard for Bots

A lot of forecasting deals with binary predictions, but sometimes it’s more natural to break those questions into multiple choices. If there are 3 potential winners for a political primary or the surprisingly close F1 drivers championship, you can look at those probabilities together instead of having a separate yes or no probability for each candidate.

There are also versions of this in the main AI benchmark tournament, so it’s nice to have the opportunity to test models on that with a shorter time frame. It’s especially helpful because it seems like there are some bots doing really weird things with these questions specifically. Because minibench questions are weird, all of the multiple choice questions in this iteration didn’t really deal with things that naturally break into multiple choices. Instead, they asked whether certain google trends would ‘increase’, ‘decrease’, or ‘stay the same’, where stay the same meant a 3% or less change.

I don’t love using google trends as a benchmark, but I DO love asking the question this way. I talked about this a bit in the previous post in this series, but the short timeframe on minibench questions means there is unlikely to be a ton of movement in any direction on any given question. This makes it hard to confidently predict whether something will move up or down, so the smart move is to guess something close to 50%. By giving your bot the option to say ‘no relevant change’ you can potentially get much more helpful predictions.

So how’d this go? Interestingly not very well for many bots in the tournament. Through all of this section, I’m referring to the community aggregate prediction, which combines all of the predictions of all participating bots.

Here I’m showing the baseline score, and a score of 0 is essentially a chance guess. Similar to last tournament, bots did OK on binary questions and very well on numeric ones, but the multiple choice questions had a huge range and the median performance was worse than guessing! This is especially surprising because the last tournament contained identical google trends questions, in binary form instead of multiple choice, and the community bots had a median score on those questions of nearly +60. So what’s going on here?

Boring Priors Sometimes Fail

Here I’m adapting the term prior from the more technical Bayesian prior probability to refer to the baseline assumption, before any evidence is collected, for how probable something is. If you’re rolling a six-sided die, your prior is that you’ll roll a six 1/6th of the time. That’s a boring prior (uninformative, technically). Sometimes you come into a question with extra information before you even get evidence. If you’re playing with your friend Paul who always cheats and he bet you $10 you couldn’t roll a six using his die, your prior that you’ll roll a six may be much lower than 1/6 even before you start rolling. That’s an informative prior. Eventually you start rolling dice and get additional evidence, and your prior is going to influence how much that evidence changes your mind.

If you’re using an LLM to predict things there might not be an obvious prior, unless you code something in to tell it ‘your prior is x,’ but there is definitely something like a prior baked in by all the training its gone through. This is going to interact with everything else you put into the model (i.e., prompt, web searches, context, etc.) and through the magic spell of matrix multiplication and other math spices you get some kind of output combining the prior and evidence. This isn’t really Bayesian, but I think the analogy is helpful. My theory is that the default ‘prior’ in these bots is boring and actively harmful for those multiple choice questions.

There’s an extremely niche meme that ‘superforecasters’ just have a rock on their desk that says ‘nothing ever happens’ and any time they are asked a question they stop, look at the rock, and then put in an answer that says nothing is going to happen. This can actually be pretty helpful, because people often overestimate the probability of things happening and if you counter that instinct you can make better predictions. This status quo bias is even explicitly coded1 into the template bot provided by Metaculus for people participating in this tournament. I don’t know how many people leave this prompt in place, but I suspect many do or otherwise impose something similar. Because this shows up clearly in these multiple choice questions.

All of these questions are essentially asking if a given google interest trend will go up, go down, or stay the same. The “doesn’t change” option is the runaway favorite of the community, happening >50% of the time and approximately twice as often as either alternative. My own model (‘Bot’, in this graph) has it’s own bias, but seems to favor trends going down instead. It sure seems like the bots have found the nothing-ever-happens rock and refer to it at every opportunity. But there was something else a little weird that makes me even more convinced that priors are dominating here.

I made this plot to look at correlation and see where my bot disagreed with the community, but the more interesting thing is how uniform the predictions are. For the community, almost all of the dots are stacked on top of each other right around 60%. This means that the community essentially thought there was a roughly a 60% chance of no change no matter what the question was about on 10 of the 13 questions. My bot wasn’t immune from this (though in the opposite direction) with 6 of the questions all hovering around the 20% line, but in general had much more variance which did help it score better on these questions. This kind of pattern just reeks of the prior dominating the decision.

It’s ok to lean on the prior when the evidence is sparse, but the nothing-ever-happens prior is particularly poor for this specific type of question. Here’s an example of a representative google trend for one of the questions.

This is pretty typical for google trends. For most buzzy topics, unless they are in the news all the time like ‘AI’ or ‘football’, they have a big spike of interest tied to some news event that predictably and rapidly drops off to near zero. Depending on where the initial comparison point is (and for these minibench questions the baseline is often taken during this ‘buzz’ period), it’s very unlikely to stay at that same level of heightened interest. But it seems like the typical bot just looks at the digital nothing-ever-happens-rock and pushes the nothing-ever-happens-button on the forecast. This worked really well when the question was binary, because a ‘no’ answer corresponded with both ‘doesn’t change’ and ‘no increase.’ Because those are both far more common, having a bias in that direction was quite helpful. That’s likely why the community performed so well on those questions last tournament (all hail the rock!) But now that the questions don’t fit the prior, the rock fails us.

Don’t Just Change the Prior

However, there is a wrinkle in this assumption that means you can’t just adjust your baseline to be biased towards ‘decreases’ in all scenarios and collect the prize. The peak of that graph is on September 25th, but the question was comparing what happened between October 2nd and October 16th. By the time October 2nd rolled around interest had already dropped to 4. The criteria for ‘doesn’t change’ allowed being within 3 points of the starting value, and because the scale only goes down to 1 it was literally impossible for this question to resolve as ‘decreases.’ Some of my bot’s worst failures came because I had the ‘decrease’ prior and no way for the bot to know that decrease was, in fact, impossible.

I suspect that this kind of thing comes up all the time with forecasting bots. The Achilles heal of LLMs is that, while they are great at sourcing tons of information, they often lack the sort of common-sense checking that is intuitive for people. This is a particularly big issue when there is a wealth of potentially conflicting information, like many google trends values from many time points, and no clear structure set up to interpret those. Fortunately, I think this particular instance could be solved pretty easily with a bit of support scaffolding.

These google trends questions would be trivially easy for a human forecaster. I can just look at the graph, think a bit about whether this was a one time buzz or a recurring thing, and adjust my forecast intuitively. At minimum I can rule out impossible thing, like a decrease from a near zero baseline. It seems that most bots have a hard time finding this data, so they just make a guess based very heavily on their prior. It just so happened that, this time, that prior wasn’t very useful. This kind of obvious information gathering failure is extremely useful information for anyone trying to build one of these bots, so I’m glad this question type was included and hope it sticks around.

Subscribe now

Delphi-bot Updates

The first update is that my performance was far better in this tournament than the last one, moving from 18th to 6-7th out of 44 bots (will depend how the last question resolves, as the gap between those two places is currently 0.17 points…) This also put me above the most relevant Metaculus template bot which gives me a little more confidence that the scaffolding I’m using at least isn’t actively harmful. I’m unsure how much of this is due to noise, the change in the question types, or the changes I made, so I’ll probably hold off on making further changes until we see how the current tournament finishes. But it’s nice to see some progress.

Based on the results of the last minibench analysis, I made a few relevant changes to the way my bot was making predictions:

1. Altered the prompts to encourage the models to be slightly more aggressive with their forecasts.

2. Made some minor technical changes to how the multiple bot forecasts were aggregated to get a final prediction.

3. Switched the model for news gathering to gpt-5-mini with agentic search.

Because I didn’t get around to finishing the first analysis until this tournament was already half over, I won’t really know how these panned out until the next tournament wraps. But I did have at least a handful of questions get answered with these changes so should get some kind of signal.

Metrics

When looking by question type we see the real reason for the relative performance improvement comes from doing very well on the multiple choice questions. The mean peer score for these questions was 36, which is far better than the other categories even when considering every multiple choice question essentially counts as 3 binary questions. I don’t have much to add to this beyond what I said above, but it seems that my model mostly just had a prior that was more advantageous for this particular set of questions.

The numeric questions actually did much worse than the last tournament, averaging ~12 points rather than ~30 in the last round. In the specific examples I checked this seemed to come from under-confidence. It also could be due to noise, as the way I was previously importing the probabilities made the scoring pretty jagged. The first two changes I implemented above seem to have helped with this, so we’ll see how the next tournament goes.

Above is a plot of the score received for each question, sorted from worst to best. Because I made the changes mid-way through the tournament, I included labels for each of the questions to indicate whether they happened before or after the changes were implemented on October 5th.

Unfortunately, just like last tournament, a generally good performance was dragged down by some real bombs. If any one of those worst three had simply been 0, or unanswered, my bot would have placed 2nd. After manually inspecting those three, the two worst offenders were the exact same issue I identified in the last minibench analysis (hallucinating a wildly incorrect baseline value from Metaculus).2 The third was one of those situations with google trends I discussed above where the initial time point was essentially already at the floor, making the predicted result of ‘decrease’ impossible. I’m hopeful that the new web search I implemented will fix both of these problems, and I’m encouraged to see that the orange dots (which occurred after the fixes were implemented) did not have any horrifically bad scores.

We will see what happens when the next minibench wraps up in the next week or two, which should provide better feedback on these changes.

It’s an open debate whether hallucination can be eliminated or whether they reflect an insurmountable barrier inherent to any LLM based AI approach. Fundamentally, LLMs are trained to predict the string of words that follow a different string of words. This is an intrinsically probabilistic process, and it’s possible that this randomness can never be truly eliminated. However, human brains are also probabilistic, and we nevertheless seem to be able to function pretty well [citation needed]. Hopefully, we can implement some approaches to at least achieve the level of factual correctness and truthfulness that are standard in human thinking.

In AI forecasting in particular there is clear damage from hallucination that can be far more impactful than subtle issues in reasoning. In my last post I talked about an example where my forecasting bot incorrectly reported a number from the internet and as a consequence made a wildly bad prediction. Much like in real life, being very wrong in forecasting tournaments is much worse than being just a little wrong, and this one hallucination effectively wiped out the gains from 16 other modestly correct predictions. Imagine a similar issue happening when AI systems need to make important decisions where the consequences aren’t as easy to shake off (like knowing if weapons of mass destruction are being developed under Saddam Hussein) and we have all the motivation we need to try and solve this problem.

Fortunately, people a lot smarter than me have been working on reducing hallucination. This post explores some of what they’ve been able to accomplish and how this can be incorporated in AI forecasting tools.

Choose the Best Model

The most straightforward and effective approach in most applications is simply to use a better model. The bitter lesson of AI advancement is that, compared to clever tricks or targeted optimization, just scaling up computation and data quantity has proven more effective at basically every task. From our perspective this means we are almost always better off using the best models developed by leading edge frontier labs than trying to do anything fancy with fine tuning or training our own models.

But which model to pick? Over the last couple of years, leading models have become almost commoditized. At least among the leading 3-4 labs, there is no clearly dominant company and the ‘top’ model typically holds the crown for a few weeks to a couple months at most. This means that it’s challenging to simply intuit which model will be the best performer. On the one hand, as long as you stay within the narrowly defined current generation of models, you are unlikely to do too poorly on this. On the other hand, if we care a lot about accuracy, we’d like to have some way to pick the absolute best option. Benchmarks are one way that we can quantitatively make this decision.

Benchmarks frequently cycle in and out of popularity, but one relatively high profile benchmark for hallucination specifically is Vectara which compares hallucination rates within a RAG framework. I’ll talk about RAG a bit more later in this post, but the important thing to know for now is that it’s something of an ideal case for avoiding hallucination. So, the numbers you see on this plot are going to be, in absolute terms, much lower rates of hallucination than you will likely see in the wild. Nevertheless they should provide a good relative benchmark when picking across models. So which models make the fewest things up?

Of the major models, Gemini or GPT o3 seem to be potentially good choices, while Grok 2 rounds out the top 25 (interestingly Grok 4, the most current version of the so called “truth seeking” AI, is ranked 98th). It’s worth clicking through to the full leaderboard to see where your models might end up, as some of the findings are surprising. For instance, while GPT-5-high performs much better than GPT-5-mini (not shown because it’s outside the top 25), the opposite is true for o1-mini which performs better than regular o1. While it’s good to take benchmarks with a massive grain of salt, this does undermine the idea that you can simply use the best/most expensive model you have access to and get the best performance (at least when considering this issue of non-hallucination).

There are some big caveats to using these benchmarks to pick your model. First, because it’s a RAG system, this doesn’t reflect the hallucination rate you might care about. For my model, I care most about hallucinations when summarizing web search results, so RAG based benchmarks are probably relatively safe. Second, benchmarks often disagree with one another. A clear example of this is seeing GPT-5-high barely outperforming GPT-4o in Vectara, while in their own press release OpenAI reported that GPT-5 had 45% fewer such errors than 4o. Lastly, public benchmarks can be gamed, so you should have at least some skepticism when interpreting standings. In general, the simplest approach may be picking models that seem generally capable but double checking benchmarks to make sure you aren’t ending up with one of those that perform inexplicably poorly. Beyond that, explicitly testing several models in your specific pipeline is worth doing, as there is a ton of variability both across benchmarks and across contexts.

Prompting Improvement

Prompt engineering is a surprisingly impactful way of manipulating models without requiring computationally expensive fine tuning or retraining approaches. I hope to have a more complete investigation into various prompting approaches at some point, but in the meantime I will write a bit about approaches that seem particularly relevant in this context of minimizing hallucination. The categories below progress roughly from simplest to most complex, though each category should be considered very broad as they hide a significant amount of variance in terms of specific approaches.

Zero-shot, One-shot, and Few-shot

The most straightforward approach to improving hallucination is simply to improve the detail and specificity requested in the prompt. This was particularly noticeable in earlier versions of LLMs which had fewer built in system level prompts to control the tone and style of responses. In my experience this is less critical in most leading edge models, which seem able to translate ambiguous prompts into relatively well structured responses fairly reliably. However, there is still substantial difference in quality between a prompt that looks like this:

“Provide a summary of news relating to [question]”

Compared to something like this:

“Provide a summary of news relating to [question]. This summary should focus on the most recent and relevant news, and priority should be given to authoritative sources (e.g., respected news organizations, institutional publications, and government provided data). Importantly, if there is evidence that the question will resolve imminently that evidence should be emphasized in the report.”

While there are some empirical papers looking at the impact of this type of prompt design on various aspects of model performance, this type of engineering is still far more art than science. There are some attempts at regularizing this, such as iteratively changing the prompt in a loop to achieve improvement in your desired metric, but even these approaches are very fragile. Often, a prompt that works great for one model will not work nearly as well in the next, even if it is just an updated model from the same company. Ultimately, it doesn’t seem like there are great alternatives to simply trying many different prompts and manually evaluating how well they achieve what you are trying to do.

The above prompt approach is sometimes referred to as ‘zero-shot’ prompting, which is contrasted with ‘one-shot’ or ‘few-shot’ prompts. In zero-shot, we simply tell the model what we want it to do but don’t provide any examples. If we want this to be more structured, we can instead provide the prompt above with something like this attached:

“Below is an example of the desired response:
Question: Will the US pass a federal bill on AI regulation before January 1, 2026?

Imminent resolution: There is no evidence that a qualifying bill has passed or is certain to pass in the provided timeframe.

News summary: The Hill reports that the following potentially relevant bills are under consideration… but that none of them are expected to come to a vote before the end of the year”

Providing an example such as this helps guide the model to produce the type of analysis we are hoping to receive. One-shot prompts can be further strengthened by including multiple such examples, i.e., few-shot, ideally covering the range of responses that might be required. For example, you might extend the above example to include a case where the resolution appeared imminent. There is some fairly solid evidence that including this kind of structure in prompts can improve their quality, though I’m uncertain how necessary this is in the most up to date models.

Controlling the Thought Process

‘Chain of thought’ prompting was big news when it was first developed a few years back, though now it (or something similar) lives under the surface of essentially all reasoning models. In chain of thought, we explicitly prompt the model to ‘think step by step’ through the process rather than simply jumping to the answer. This helps because it seems to encourage the model to explicitly write out the multiple steps required to achieve the answer, which apparently reduces its tendency to just riff. While this initially provided significant gains in reasoning for different LLMs, my impression is that much of this value is already baked into most modern models which limits its value as a standard prompting strategy. There are, however, several adaptations to this approach which may be more narrowly useful in the context of hallucination reduction and improving forecasts more generally.

The first of these narrower approaches is question decomposition, where the model is asked to break the larger ask into smaller pieces before addressing those individually. In the context of prediction, this might look something like breaking the large question of ‘Will the US pass a federal bill on AI regulation before January 1, 2026’ into several smaller questions such as ‘how many days will congress be in session before January 1, 2026,’ and ‘are there any relevant bills under consideration’ and so forth. These questions are in principle easier to answer, which reduces the model’s tendency to just hallucinate the answers.

Another approach, self-consistency, instead attempts to control for hallucinations by aggregating across multiple different responses. To paraphrase Tolstoy, true statements are all alike, but hallucinations are all false in their own way. By asking a model (or several models) to answer the same question multiple times and then comparing their results, self-consistency expects that the true statements will agree with one another while any hallucinations will wash out. Empirically, this does seem to improve the quality of both reasoning and hallucinations. Unfortunately, there is no principled reason why hallucinations could not agree with one another (for example, multiple models could assume that congress is currently in session and miss the ongoing government shutdown) so this approach does not offer anything like a true guarantee of accuracy.

Internal Argument

One of the most intellectually appealing prompting approaches (at least if you’re a huge nerd like me) is a category broadly defined as ‘multi-agent debate.’ In its most basic form, this involves having more than one LLM approach a question from multiple perspectives and have a back-and-forth debate about the answer. This can be refined in many different ways, such as having the models take on different personalities, or using a third ‘judge’ model to arbitrate the disagreement. While this debate approach is often used to improve the quality or faithfulness of reasoning, it can also reduce hallucinations because fact checking your opponent is one way of undermining their argument and winning the debate. This is analogous to what human thinkers in an effective truth seeking team might do (for instance, in adversarial collaborations), and there is some fairly strong evidence that it improves accuracy in various contexts.

Unlike the other prompting approaches in this section, this one does require a little bit of scaffolding. While it’s possible to write a prompt asking a single model to generate multiple arguments, it works best to have multiple model calls. This requires proper handling of prompts, context, etc through multiple rounds of argument. While this is technically possible working through something like ChatGPT on your web browser, it’s much more convenient to implement programmatically. This approach straddles the boundary between pure prompting approaches and the more technical/structural approaches described in the next section.

Structural or Technical Solutions

Although they are not directly applicable to this forecasting tournament (outside some potentially well supported efforts), I wanted to briefly touch on a couple of the most popular and effective technical solutions for reducing hallucinations.

The first of these is Retrieval-Augmented Generation (RAG), which uses vector based search across a pre-existing database of information (think, court cases or a company’s internal documentation) and feeds that in as context when models are generating responses. This has proven to be very effective at making LLMs produce more reliable information in specific settings. Of course, this requires a fair bit of structural support, most important of which is to generate and maintain a vector embedded database containing all of the information you want to use as context. This is a bit problematic in the context of AI forecasting, which could conceivably deal with a huge range of topics and relies on information being constantly updated as news filters in. The challenge is not insurmountable, but does represent a significant amount of effort. My impression is that this is part of the approach in AskNews or various similar services for academic writing, which might be one way to get the benefits of RAG without needing to maintain your own curated and up to date news database.

A second popular approach is Reasoning + Act with tools (ReAct), which interleaves model reasoning steps with tool based action steps. The core idea here is that the model should seek out information when it is unsure, rather than simply hallucinating the answers. This approach has proven very effective both for reducing hallucinations and avoiding common pitfalls like making errors in structured areas like math. Implementing this kind of structure requires a significant amount of programming investment and testing, which then needs to be re-implemented and tested every time a new model is swapped in. With a strong team, this could likely produce some pretty impressive results in a forecasting bot. Without a strong team, it’s likely better to rely on thinking/reasoning models (which generally have a similar ability to use tools and multi-step reasoning) to get much of the benefit without as much overhead.

These are just two examples which really represent broad categories of technical solutions to hallucination. This rabbit hole runs very deep, and even these two examples split fractally into dozens of additional refinements. A deep dive into all these intricacies is well beyond what I have the capacity to get into in this post, but there is plenty of fertile ground to apply these or similar approaches to forecasting AI.

TLDR: Where to Take This for Forecasting Bots

The interest in reducing hallucinations generally is immense, and there is a correspondingly massive amount of research in this area of which this post barely scratches the surface. Condensing this down to a few key takeaways:

1. Use benchmarks (or, ideally, experiments) to pick the best bot for your purpose.

   1. Keep in mind that the largest/newest/most expensive bot is not always the best, especially for the narrow task of minimizing hallucinations e.g., when summarizing news.

2. Improve your prompts using validated strategies.

   1. Adding better zero- or one-shot prompts can have a significant impact without requiring a ton of effort, though this is more art than science.

   2. More involved approaches like self-consistency or multi-agent debate also have some interesting potential, particularly for coming up with forecasts.

3. Look into concrete technical solutions (if you have the bandwidth on your team).

   1. Things like RAG and ReAct are the most empirically well supported strategies, but they come with significant technical overhead which may not be realistic to implement.

I’m planning to experimentally test some of this advice in my own narrow context of accurate internet search for my own forecasting bot, and hope to have a post on that soon to follow-up. So stay tuned for that in the next week or two.

Getting good feedback on prediction model performance takes a long time, which is less than ideal when trying to iteratively improve an automated forecasting bot. I’ve written a bit about strategies to overcome this, but the minibench tournament Metaculus is running as part of the AI benchmarking series is a good opportunity to get this kind of feedback on a slightly faster time scale, though with the caveat that they are necessarily short term predictions. I wanted to take advantage of this by doing a little analysis of my own and the community’s performance on the most recent iteration of this tournament. This post covers the results of that analysis, as well as some general thoughts on the tournament structure and overall strategy.

Minibench Questions are Weird

A natural consequence of having an automated tournament focused on short term predictions is that you end up with pretty homogeneous question types. In this particular tournament the questions fall into a few discrete categories: sporting events, financial data, google trends, and ‘meta’ forecasts which are snapshots of ongoing Metaculus questions receiving human forecasts. These are also divided into binary and numeric forecasts, though all of the numeric forecasts in this particular tournament were on financial data.

Aside from being homogenous, the questions themselves range from relatively easy to nearly impossible to predict even in principle. In particular the self-referential Metaculus questions generally ask the bot to predict which direction an existing Metaculus forecast will move. This is a good way to tie in to the types of questions we’d really like to be getting at without waiting several months prediction to resolve. However, they are quite challenging because typically the threshold is set at or near the current prediction and movements over the course of 2 weeks can be minimal and near random.

Here is an example of such a question:

And here is a plot of the actual predictions for the referenced question over that time frame:

Over the timeframe in question the median prediction moved from 5% on September 16th to 7% on the 25th, and the interquartile range spans ~10% to ~4%. Given the uncertainty here (only 40 forecasters had predictions entered at this time) this represents a tiny change in practical likelihood. However, because this is compressed into a binary yes/no answer the resulting scores could be quite punishing.

Compare this to the numeric predictions, which dealt with more concrete and predictable outcomes. Unsurprisingly the models (judging by the community average predictions) do very well on these because they are predictable. But more importantly, because these questions are numeric and the probability distributions are fairly smooth the error is also smooth.

I have not run the simulation study for this yet, but instinctually this sort of numeric prediction provides significantly more statistical power. This makes me wonder whether it might be more advantageous to convert some of those meta level Metaculus questions into numeric predictions. For instance, rather than ask “Will the forecast for [question x] be higher than [probability y] on [date z]”, ask “What will be the forecast probability [y]”. In principle the confidence level on a binary prediction captures this information and with enough questions they should produce similar (identical?) results. But with the small sample size these binary reductions seem needlessly noisy.

Analysis of Community Performance

The community aggregate (CA) is a straightforward unweighted average of the forecasts submitted by all the bots in the tournament. These aggregates are generally fairly good (in this tournament, the CA would have placed 7th with a total peer score of +506) and can also give us a sense of any general AI bot tendencies to look out for.

Starting with the calibration plot, we see that the bots are (in aggregate) generally pretty well calibrated. Because there were only 43 scored binary questions the error bars end up pretty large, but the predictions overall land quite close to the perfect calibration line. It’s possible there is some negative bias (questions predicted yes with 55% probability ended up happening over 70% of the time), but we can’t say this definitively due to the large error. The last and most clear observation is that the predictions were all fairly conservative ranging from ~0.25 to ~0.55. As I mentioned before, many of these questions are very difficult and close to coin flips, so this conservatism might be a function of that. It is worth considering though that the bots in general may be underconfident.

One of the most clear outcomes of the different analyses I ran was that the numeric questions were far more predictable than the binary ones. This is born out in the analysis of score by question type where the community bots in general had far higher baseline scores (essentially, how ‘right’ the predictions were) for the numeric questions. Interestingly, the peer scores (adjusted for performance of other individuals) were also much higher for the numeric questions with the median numeric peer score being more than twice as high as the median binary score. This is worth keeping an eye on. According to the Metaculus score faq the continuous question scores are empirically rescaled to be roughly equivalent to binary questions (median +17 for binary, vs median +14 for numeric). But here I’m finding that numeric questions (median baseline: 68.4) outscore binary (median baseline: 13.8) by nearly 5:1. This propagated through the peer scores where the median was 12.0 for numeric vs 3.7 for binary, or about 3:1. As it stands now, it would be far better from a performance perspective to focus heavily on those numeric questions.

The last thing I wanted to look at was performance by question category. As noted before, there are essentially 4 categories of question, with some of them (e.g., Sports, Metaculus change) being inherently hard to predict. I used GPT-4o to automatically generate labels for these and evaluated them by category. A couple things immediately pop out from this. First, the expected ‘hard to predict’ categories did indeed have a much wider spread than the other categories (though at least for the Metaculus questions the median performance was still solid). Second, google trends in particular seem pretty easy to predict. I think this is a place where having a good scaffolding makes a big difference, as understanding the base case (e.g., if something just happened that caused a big jump in attention, it is likely to fade over the next two weeks) will make it very easy to predict with high confidence.

Analysis of Delphi Performance

Delphi finished 18th of 41 participants in this tournament with a peer score of 276, while first was at 1023 and the median finish was 204. So slightly better than the median but still not a great performance. The range across performances here is very high. There were 56 scored questions in the tournament, so a score of 1023 is an average of ~18 points per question (vs. my average of ~5). Also Delphi did significantly worse than the most similar generic Metaculus bot (GPT-5 w/ ask-news at 357) which has less scaffolding, suggesting the scaffolding and different models I used generally hurt rather than helped.

But these top line results obscure a lot of detail. Hopefully going through some of those details can reveal targeted areas to improve without over indexing on these very short term questions.

The calibration for Delphi does not look great, though the confidence intervals are massive and all contain the perfect calibration line. Still, it does look like there might be some hint of a bias towards underestimating the true probability (similar to the community predictions), as (mostly) the mean predictions lie above the calibration line. Part of my prompt suggests that the bot should be a little biased towards baseline, which given the way this tournament is structured may result in a slight negative bias. Delphi was slightly more aggressive in terms of confidence than the community aggregate, which I think is good, though this did result in one very bad prediction which I look into a bit later.

Similar to the overall community performance, Delphi did very well on the numeric questions. In fact, without the numeric questions it likely would have had a score worse than 0. I looked through the actual probability distributions (my prediction vs. community) on these and much of the lift seemed to come from being slightly more confident (i.e., narrower predictions) and also placing much less weight on the out of bounds predictions. I’m not sure how much credit to take for this; I think much of the benefits I got simply came from not having any failures in the form of completely unrealistic predictions or not incorrectly inputting the CDF through the Metaculus API.

Evaluating the questions by topic, the baseline scores end up looking very similar to what was seen in the overall community aggregates. The main issue seems to be underperforming on the Metaculus and financial questions, where there are some things that could likely be tweaked if only to minimize obvious errors and increase prediction confidence.

Lowlights

Since the main thing I was focused on for the iteration of my bot was cutting down on hallucination, I wanted to dig a little deeper into the worst performing questions by score to see if there are any obvious hallucinations that impacted the predictions.

1. Will the community prediction be higher than 35.00% on 2025-09-27 for the Metaculus question “Will the Department of Justice announce an investigation or prosecution of a Democrat who served as president, vice president, congressional leader, whip, or impeachment manager before 2026?”?

   1. My Prediction: 75%

   2. CA Prediction: 39.5%

   3. Outcome: No

   4. Spot peer score: -83

Manual review: Looking at the research report generated for this question, my web search bot reported that the current community forecast was 42%, which would have been well above the 35% threshold. The actual value at the time the question was run was 35%, not 42%, so this appears to be a hallucination. This then propagated through the reasoning of the other, more costly bots which based their assessment on this incorrect baseline. The reasoning generally indicated that these community forecasts drift slowly, which would have been correct if starting from an accurate baseline.

Conclusion: Hallucination induced.

Solution: Pull forecasts like this directly rather than relying on web search.

2. Is it the case that Sunderland AFC and Aston Villa will finish their September 21 EPL match with identical scores?

   1. My prediction: 27%

   2. CA prediction: 32%

   3. Outcome: Yes

   4. Spot peer score: -31

Manual review: The research + web search bots (correctly) indicated that the draw rate was between 25-30% for EPL matches (was 28% in 2025 season). All bots reasoned that given the weather and short time until the match happened made it unlikely for a cancellation to affect this risk. Generally I think the reasoning here was sound, but I would have liked to see some reference to betting odds in the research portion.

Conclusion: Unlucky, but room for improvement.

Solution: See if I can emphasize betting odds for sports questions.

3. Will the community prediction be higher than 91.00% on 2025-09-24 for the Metaculus question “Will Zohran Mamdani be elected Mayor of New York City in 2025?”?

   1. My prediction: 40%

   2. CA prediction: 60%

   3. Outcome: Yes

   4. Spot peer score: -29

Manual review: Another Metaculus question, another hallucinated baseline. Here the research report indicated that the prediction at time of question was 75%, when it was actually 90% and rose immediately to 93% on that day. However, this time the forecasting model calls mostly chose to ignore that 75% baseline for some reason. Instead they exhibited too much “status quo bias” that was explicitly coded into their prompt. Still I do think the hallucination had an impact as it biased the rest of the research which was fed into the prediction models.

Conclusion: Hallucination and status quo bias.

Solution: Pull Metaculus community aggregates directly. Try to downplay status quo bias a bit.

Summary

This was the first minibench tournament I participated in, and I think doing this analysis was helpful for understanding some of the nitty-gritty things that impact performance. I also think some interesting things fell out of the analysis. The main and most important one, I think, is that the numeric questions as scored seem to significantly outweigh the binary questions. This is somewhat accounted for by the structure of the tournament, as the peer score ratio (approx. 3:1 numeric vs. binary) was compensated for by the fact that there were only 13 numeric questions vs 43 binary. Still, I think the numeric questions have a ton of value particularly in terms of smooth errors giving them better statistical power. I would like to see more questions like this, even if it required additional rescaling to make them more comparable to binary questions.

There are also some obvious changes to make as a result of this analysis:

1. In general, bots seem to be a bit underconfident, particularly on questions that were fairly predictable (here, google trends and financial data).

2. The most disappointing finding was that my hallucination guardrails didn’t seem to solve that issue. Though I don’t have a baseline to compare to, a couple very obvious hallucinations had similarly obvious negative impacts.

   1. I’m currently using a purely prompt based approach to this, and did see examples of this working successfully to catch errors. However it is clearly missing some bad ones, particularly when it comes to web search.

   2. It may make sense to offload some of this out of LLM land, especially for things that can be easily pulled programmatically (like prediction market values).

   3. Probably this needs to be refined in a more quantitative way for testing purposes, as the current approach of manual review by me to see how well the guardrails are working doesn’t seem sufficient.

3. One change I don’t intend to make is to directly optimize for these minibench questions. Given the cookie cutter nature of this tournament, it would be relatively straightforward to design specific workflows that maximize prediction accuracy. However, I don’t think these are in the spirit of the tournament. Additionally, aside from a few cases like making use of prediction market APIs and feeding those in as part of the research, it’s not clear how these would translate to better long term forecasting.

Overall I think this was a useful exercise, and will hopefully lead to a more accurate forecasting bot (which is the point after all!). I didn’t complete this analysis in time to make changes for the current minibench, but will look forward to repeating this in ~3 weeks to see if any of these changes made a difference.

Something that Croesus understood, even 2,500 years ago, was that you need some way to evaluate the quality of your oracles before using their forecasts to make important decisions. Unfortunately for him statistics hadn’t been invented, so he relied on a straightforward approach of receiving a single prophecy from several oracles and going with whichever was the most impressive. Hopefully we’ve learned a bit since then and can do better when evaluating our new AI oracles.

Because I have several experiments to run with an AI bot forecaster, I need some way to quantify how the different versions of the bot are performing. This post is going to get a little in the weeds, but I want to lay out these ideas because they are going to be important for many of the experiments I’d like to run.

When evaluating human forecasters, the gold standard is to run long-term tournaments like this one and wait (sometimes years) for results to come in. There are some lower investment approaches that could be used, but none are effective at measuring what we really care about: long-term forecasting accuracy. Unfortunately this temporal limitation makes it challenging to iteratively improve forecasting performance, as the gap between any change and eventual feedback is prohibitively large. This is a major issue for developing any kind of automated system to eventually outperform (or even match) human forecasters.

We would like some way to speed up this process. Metaculus is running short, 2-week mini tournaments throughout the fall as part of the AI forecasting project. These are very helpful, and I expect to have more to say about some of the recent ones soon. But there are a couple issues with using this as our primary form of feedback: the questions are of a limited variety, all predictions necessarily resolve within a 2 week time span, and because there are few questions (and mostly binary) they don’t provide a ton of statistical power. Ideally we would supplement this process to both get some form of immediate feedback for refinement and provide an alternative set of data points.

Leverage Points

The automated nature of forecasting bots opens some potential avenues to exploit that wouldn’t be possible using human forecasters. Below is a short survey of these approaches, including the benefits and drawbacks of each. This list is not exhaustive, but covers some of the most common and high leverage opportunities.

Scaling

Predictions can be made in seconds or minutes, so a large volume of questions can be predicted in a short period of time. This component can, and usually is, combined with any of the other approaches below to improve statistical power.

Benefits of scaling: Comparing predictions across a very large number of questions gives better statistical power, making us more confident that certain approaches work better. The implementation of scaling is also straightforward, as the same pipeline can be run essentially in a loop.

Drawbacks of scaling: Though individual predictions are relatively inexpensive (on the order of 5-15 cents in my current model), this cost can become an issue when predicting hundreds or thousands of questions. In addition, this approach is limited by the number of relevant questions available. This may also introduce a bias in favor of models that are generally good across a wide array of questions over those that are more narrowly effective within a certain category or type of question.

Resources: ForecastBench which provides a regularly updated list of 1,000 forecasting questions that have not been resolved. Results are graded nightly, using a combination of true question resolution and intermediate outcomes.

Parallel Testing

Similar to scaling, many iterations of a bot can be created and run in parallel forecasting on the same set of questions. Again this can be combined with other approaches.

Benefits of parallel testing: A large number of variations in strategy can be tested simultaneously, maximizing the amount learned per unit of time. This also allows for more generalist insight, such as understanding the variance inherent to any given approach or the general lift provided by incorporating a certain model or news source, for instance.

Drawbacks of parallel testing: Again, cost will scale in proportion to the number of model variations tested, though there are some approaches that can reduce this cost. For example, if testing variations in a certain prompt that is part of a multi-stage process the non-varied steps can be cached rather than relying on a fresh API call every iteration. We also might expect survivorship bias that comes from running a large number of models, though this can be accounted for using techniques like bootstrapping to get a sense of the variance in performance. Finally, parallelizing models to test is slightly more complex than feeding in a large number of discrete questions. It requires an infrastructure to support the changing and tracking of many bot parameter changes, and a way to robustly analyze the results of those changes.

Resources: There are a large number of existing tools for automatically varying certain parameters, or changing features such as prompts, which can make this process more efficient. Or these can be designed from scratch without too much headache.

Back Testing

The data available to bots can be limited, for example by training models only on data and news generated before a certain cutoff date. This allows for backtesting, which is where predictions are made about past events that have already resolved (but where the model does not have the answer) to get results immediately rather than waiting for time to pass. This approach is commonly used in many machine learning prediction tasks, like quantitative equity price forecasting. While this method is extremely valuable, the drawbacks are similarly extreme.

Benefits of back testing: This approach provides a much faster feedback cycle as experiments can be run continuously and changes can be made in reaction to those experiments. This enables approaches like tuning or reinforcement that are common in other machine learning applications.

Drawbacks of back testing: The most significant drawback is that this approach is technically demanding and cost prohibitive. To enable back testing a model must either be trained from scratch (which also requires access to a clean dataset containing only information before a certain date cutoff), or a pre-trained model must be used with a known information cutoff. The first of these is best, but because the cost of training frontier quality LLMs is extreme (>$100 million for GPT-4) it is not realistic outside of very valuable business use cases. Alternatively less expensive and smaller models can be trained, or older models with a known training cutoff can be used, but as a rule these models are less effective than leading models and in many applications any refinement (e.g., scaffolding, fine tuning, prompt engineering) is swamped by the benefit of using larger scale models. Finally, back testing must exclude web search as this provides an obvious avenue for data contamination that would render the test meaningless. Because web search is a key component of good forecasting, any approach relying on back testing will be necessarily limited.

Aggregate agreement

While the final outcome of a given question cannot be known until resolution, many questions have aggregated community predictions that combine forecasts from many forecasters. In general these community predictions are quite accurate, much more accurate than the median forecaster though often less accurate than the best performing individuals. One approach may be to use the community forecast as a proxy for the “true” probability of the event at a given point in time. This allows for an immediate feedback loop for comparing models, and can also be tracked across time to see whether the forecast moves in a manner consistent with the bot prediction.

Advantages to aggregates: Instant feedback allows for a much quicker development cycle. In general these aggregates provide a fairly good target, as a bot that perfectly matched this community aggregate would likely perform in the top 5% of individuals. More diagnostic information available that can be incorporated into scoring by comparing probability distributions rather than fixed outcomes.

Disadvantages to aggregates: Aggregate forecasts are only a proxy for true outcomes, which imposes a natural ceiling on forecasting quality. A bot over-optimized on this metric could definitionally never outperform the aggregate community forecast. Imagine a version of this bot that simply goes online to look at the forecast and then reports that value. Obviously this would not be very useful as a forecaster. There are also limitations imposed by requiring a certain number of human forecasts (necessary for an accurate community aggregate) that may bias the sample towards more popular topics and questions. Lastly, web search tools provide an avenue for data leakage, as bots may gain access to the community aggregate directly.

Resources: The Metaculus maintained forecasting-tools python package contains a community comparison benchmarking tool which abstracts a lot of the work required to run this (on Metaculus questions). This uses the expected baseline score, which is a scaled version of the expected log score assuming the community prediction is the true probability.

Interim Endpoints

Predictions can be partially resolved based on interim endpoints. This approach is sometimes used in clinical trials that have very long-term outcomes (such as developing Alzheimer’s disease) that are not practical to measure. An example of this for forecasting might be: if on September 1st a model predicts there will be 15,000 flu cases before the end of the year, but on October 1st we already see that there were already 10,000 flu cases in September alone, we can be fairly confident that the model has undershot.

Advantages to interim endpoints: Unlike the community forecast, interim endpoints reference some ground truth or model result that in principle can exceed human forecast accuracy. This goes some way towards eliminating the ceiling effect, while still allowing a faster cycle of resolution than can be achieved by waiting for the final result.

Disadvantages to interim endpoints: The most significant issue with this approach is it is necessarily limited in scope. There are only a small fraction of questions for which logical interim endpoints make sense (e.g., cumulative count questions, stock prices or other numerical predictions). This limits the applicability for a generalist prediction bot, though there may be some value in more specialized bots specifically addressing these questions. A second issue is that this approach is effort intensive. A proxy endpoint must generally be defined individually for each question, and it is not obvious how this can be automated.

The Issue is Always Uncertainty

While there are a number of different levers to pull on to take advantage of automated forecasting bots strengths, none of these are a slam dunk and they all come with pretty significant drawbacks. To make matters worse, it can be difficult to tell the difference between forecasters of different skill levels even in an ideal world. Here is a helpful (though somewhat demoralizing) blog post on this question from Metaculus. To steal the punchline, an ideal forecaster (who always reports exactly the true probability) beats a middle of the road forecaster (whose forecasts are on average off by 12.5%) about 92% of the time in a 100 question tournament. These are pretty good but not great odds, and if we want to reach 80% power on this experiment we end up needing to ask about 200 questions. Parallel testing and scaling make this doable, but it does impose a cost constraint that prevents us from running an unlimited number of experiments.

One thing about this simulation study is the performance of all models is relatively good. The Brier score for the worse tested model was 0.196, compared to 0.167 for the ideal model. To put this in perspective, comparable Brier scores from the Good Judgment Project experiment were ~0.14 for the best superforecasters and ~0.20 for the rest of the crowd (note: these were reported on a 2 point Brier scale so I’ve divided by 2 for comparability). So the difference in performance (relative error) in that human study was about twice as large as the difference in this simulation study. I think this should give us some confidence when it comes to evaluating strategies.

It’s clear that between the inherent uncertainty, low sample size, and long time horizons inherent in forecasting there will not be a clean optimization cycle that will automatically lead to superhuman performance. Despite this, I think the options laid out here give plenty of opportunity for significant improvement.

Experimentation on a Budget

Like most people participating in forecasting tournaments, this is more of a hobby than a profession. This imposes certain limitations that might be different from an academic study or business environment. In my setting, time and budget constraints effectively rule out back testing. They also limit the degree of scaling and parallel testing, which can rapidly become cost prohibitive. Finally, interim endpoints are generally too labor-intensive to define and implement which limits their broad applicability.

That leaves aggregate-agreement as the primary strategy, with targeted scaling and parallelism to gain power efficiently. This will be my general approach moving forward, though it will be adjusted depending on the specific needs of each individual experiment.

Croesus, King of Lydia, had a question. The Persians, led by Cyrus, had just achieved an important victory and had the look of a budding imperial power. Croesus wanted to know, should he attack this potential rival preemptively?

This story comes from Herodotus, writing in the mid-5th century BC. From two and a half millennia ago, and likely much longer ago than that, people have obsessed over the possibility of knowing the future. Whether reading signs in charred and cracked animal bones, the positions of the stars, or complex mathematical models trained on terabytes of historical data, people want to know: What is going to happen?

It’s a tired observation at this point to state that we live in particularly unpredictable times. During the COVID-19 lockdowns I kept a sticky note near my computer where I noted down the number of times I read or heard the word ‘unprecedented’ (I stopped after it had been completely filled with hash marks after about 3 months). But this doesn’t seem to have tempered our hope that we can find someone who knows the answer. While we might not consult animal bones or mythical oracles, we have a never-ending stream of op-ed columnists, TV talking heads, and online pundits spewing forecasts into the ether to be lapped up by eager audiences.

And yet, there has been some progress in the art and science of prediction. While the average pundit may be no better than rolling dice when it comes to predicting future events, we have solid evidence of so-called superforecasters who can predict geopolitical events with surprising accuracy. We in the US and in most other powerful countries have massive intelligence operations dedicated in no small part to answering questions eerily similar to the one Croesus was concerned about. Like: if we launch a preemptive attack in Persia, will it work?

Whether it’s possible or not, we want to see the future. And with modern methods of increasing sophistication, not least of which are machine learning and surprisingly powerful AI models, it seems like we might soon have this power within our grasp.

Seeking answers

Despite working on this problem around 550 BCE, King Croesus was surprisingly scientific in his attempts at predetermination. He sent messages to half a dozen famous oracles across the Hellenic world, with the intent of testing their powers before putting forth his most important question. After conducting this experiment, he became convinced that the Oracle at Delphi alone possessed a strong connection to the gods and therefore the one true power of foresight. He would shower these oracles with riches, and in return receive answers that would shape his empire.

2,575 years later, we have dispensed with the superstitious belief in the inscrutable Oracle at Delphi with a direct connection to the gods. Instead, we are running a prophetic forecasting tournament of inscrutable AIs, who (not coincidentally, because nothing is coincidental) are living on Oracle servers, with a direct connection to some of the most powerful (or at least wealthy) organizations that have ever existed. The more things change, the more they stay the same.

I’m afraid this framing might come across as dismissive, but I actually love this tournament idea. I’m participating myself, despite having minimal experience with either LLMs or forecasting, because I think it sounds so interesting (and my forecasting bot is called ‘Delphi’, because of course it is). And I think there are good reasons to think LLMs could be particularly good at this type of forecasting. Much of the information available to human forecasters—outside of that available to state intelligence agencies—is sourced through the internet and therefore easily accessible by modern LLMs. Making a good prediction requires absorbing and distilling an enormous amount of this information, something AIs are generally quite useful for. Beyond that, many of these questions cannot be modeled explicitly, so matching the general ‘vibe’ of the evidence is sufficient to get a pretty good answer. These models can produce a lot of spurious reasoning, but so can humans, so I think it’s plausible that the kind of predictions produced in this kind of AI tournament might eventually prove more accurate than those produced by humans.

Aside from that, understanding the capabilities of these silicon-based lifeforms we are dumping ~1% of GDP into serving seems to me to be one of the most important tasks of the current era, particularly if you are at all interested in the possibility that their development could kill us all. And working in an area that has fascinated but eluded humans for millennia seems like a fun way to explore that. I expect much of this blog to be documenting that exploration. But… I’m not exactly sure how much I expect it to really give us what we want.

Predictably unpredictable

Famously, Croesus received his answer from the Oracle at Delphi. Should he send an army against the Persians, they said, a great empire will be destroyed. Emboldened by this prediction, Croesus rallied his strongest allies and clashed against the Persians. Again, the Oracle proved right in their prophecy. Unfortunately for Croesus, the empire that was destroyed was his own. The Persian leader Cyrus would come to be known as Cyrus the Great, founder of one of the most impressive empires to ever exist. Croesus, other than in this tale of misunderstood prophecy, would fade into obscurity.

This is pretty cute from the Oracle. A little bit of linguistic ambiguity flips the prediction on its head and leads Croesus to his doom. In the rational modern world we wouldn’t tolerate this mystical wishy-washy language (unless it was about AI, or crypto, or politics, or sports, or…) Fortunately we have a much more respectable form of ambiguity: statistics.

In theory everything that happens is part of an endless chain of cause-effect relationships, and if you knew perfectly how the stage was set you could know exactly what the outcome would be. In practice there is a functionally irreducible amount of uncertainty, and statistics can help us make far better decisions given this fact. Running a clinical trial to find that people die 37.2% less often when given some new treatment is incredibly valuable. But statistical answers are fundamentally uncertain, and what we really want when we consult an Oracle is a definitive statement. If Croesus attacks the Persians, will he win? If he’s told he has a 37.2% chance of victory, what should he do with that information?

Decisions

There are times when having precise statistical estimates, even if they are far from certain, is very valuable. Professional poker players report that they can really ‘feel’ the difference between a 30% chance and a 35% chance, and it’s relevant for them because they can easily calculate expected value for any given choice. We also have an entire sector of financial services that is essentially just converting statistical estimates into money.

It’s not so simple to convert probabilities into most important real-world decisions, especially when you only have one shot to make that choice, and those are the places we want oracles the most. If you are considering having children with your partner and want to know whether they’ll be happy and healthy, would an 80% chance satisfy you? A 60% chance? A 30% chance? You can think through a set of principles to make this kind of decision, something I’d like to write more on in the future, but ultimately it comes down to an intuitive judgment. It becomes more about philosophy than statistics.

I think for most humans in most situations we have essentially three settings on our probability dial when asking if something will happen: yes, no, and maybe. If we are really thinking hard about something we might be able to add a couple of gradations in the form of ‘probably yes’ and ‘probably no,’ but getting too much beyond that starts to stink of imagined accuracy. Given a 37.2% chance, we’d just round that off to a ‘maybe’ or a ‘probably no.’ At a certain point you run into diminishing returns on forecast accuracy, and we might already be well past that point when it comes to converting those forecasts into decisions.

Still, I think there is value in trying to quantify this uncertainty if only to help yourself better understand the world, particularly when everything in that world is uncertain. Maybe if King Croesus was given a 37.2% chance of victory instead of a cryptic prophecy he would have made a different choice, or maybe not. But even without plugging this number into some complex decision model, I think it would have given him pause and pushed him to think about it a little more deeply. We’re not going to get oracles that tell us what’s going to happen, but maybe we can use their prophecies to make decisions just a little bit better.