Benjamin, I can grant you all three trends and still see no reason, on the strength of those trends, to expect broadly superhuman AI in 18 months.
First, revenue. You estimate what companies might pay if AI could do most office work, then treat that level of spending as evidence that AI can do most office work. The inference only works in one direction. A hundred million people might each pay for help with one part of their job and spend the rest of the day doing things the software still cannot do. Every buyer could be perfectly rational. That is quite a lot of cognitive science to ask of a sales figure.
Second, task horizons. A system that solves a problem that would take an expert three weeks has achieved something impressive. But who chose the problem, decided which assumptions were reasonable, and specified what would count as a solution? As a teacher, I could make a maths problem considerably harder while leaving the most important decision to myself simply by telling the class which method to use. Three weeks of work inside a given frame tells me remarkably little about who can recognise when the frame itself is wrong.
Third, recursive self-improvement. Researchers using AI to build better AI faster is a genuine feedback loop. The question is how much of the research judgment still comes from the researchers. If they identify the weakness, choose the experiment and decide whether the result counts as progress, then measuring how much faster the team works does not tell us when the model will be able to take over those decisions. Yet your forecast depends on precisely that distinction.
That leaves three problems your argument never addresses. The semantic problem: how does the system establish that an intermediate result means what it is supposed to mean? Two sign errors can give the right final answer. Reuse either intermediate value elsewhere and the error is back. The frame problem: what prompts the system to question the assumptions of a task when every success criterion it has been given rewards staying inside those assumptions? The safety problem: once an agent can modify parts of the system that evaluates it, how do we independently establish that a better score reflects a genuine improvement and that the checks themselves still work?
I would therefore leave the burden of proof where it started. I owe you no explanation for why progress will stop. My objection works just as well if revenue keeps rising, task horizons keep lengthening and AI labs keep getting faster. What you owe the reader is a reason to expect those trends to carry us across the gaps above. Eighteen months is a strikingly precise number for an argument that leaves the crossing unexplained.
"More importantly, the world is already rushing to scale robotics, and with AGI’s help, full deployment may only take another 3-5 years, and could be much faster. Explosive change just a couple of years later is not particularly comforting."
No this is not true. Happy to go into it more deeply as I digged into this for weeks. How quickly after ASI could you build an industrial state the size of let's say China? It's probably at least 14 years, but most of the probability mass has to be put on 25+ years
Your WW2 scenario lacks a lot of detail, things have changed between now and then. The Epoch models are alsojust unaware of how supply chains actually function, where they are, as well as are able to scale or not scale at what timeframes even with infinite labor in the world. Again, happy to go into it.
Reading this is a little dizzying, but I’m glad I did. When you’re not in the technical AI or AI safety community on a daily basis, you sometimes wonder if you’re crazy to see where these trends lead, because average awareness of the ever-changing research landscape is low. Zoom out from the models and RSI, and the money is moving too: markets reacting hard to Nvidia’s key-person change, in a scandal that’s also about chips reaching China; CoreWeave losing much of its value; Anthropic’s IPO shrinking. And the politics: “AI populism” at the midterms, anger at data centers, moratoriums on permits, and voters united on it almost everywhere, which may bring investigations and new rules. I wrote a few days ago about how these three pillars of AI uncertainty affect each other, because I think looking at each in isolation distorts the real picture. As you say, algorithmic RSI may not need more chips. Spreading it across the economy still needs money and permission. I’m curious how the next weeks, months and years will play out.
If you just extend the trends of the last four years of the 1960's, you reach broadly livable-level “moon base” in 2-4 years, and colonies in Alpha Centurii soon after.
Just because something has been growing exponentially does not mean nor imply it will continue doing so, almost everything faces diminishing returns. In 1902 there had never been a functioning areophane, in 1969 humans set foot on the moon. Today humans have not travelled to any new astral bodies. And getting a human to Mars is something we know is possible, creating a sentient AI through incremental improvements in LLMs is not.
I discuss this in the post. Sure at some point things will hit an s-curve, but right now we still seem to be on the steepening part of the curve, rather than the diminishing part. Moreover, the underlying drivers of AI progress (compute and R&D labour) seem set to continue to increase at trend at least the next couple of years. AI is also pretty unlike space travel because it generates revenue that enables further investment into it.
I take your point on space travel, but I’m still not convinced. The argument you are making is contingent on your implicit premise that recursive self improvement is a) possible at all, b) possible given our current level of technology, c) that AI researchers have already identified a model for implementing recursive self improvement which works, and d) that compute and R&D labour are the inputs towards RSI. If any one of those premises are untrue your argument fails.
The fact that measures point towards a steepening of the curve does not imply that recursive self improvement or artificial general intelligence is technically possible, that the current approach is the correct one to get there, nor that these measures actually measure what people claim they measure. For instance, revenue is a terrible measurement for technological progress as ancient technologies like baking bread, making metal locks, and selling lumber generate large revenues. It is also really important to recognise that Anthropic and OpenAI are not pursuing their research for the public good alone; they pay high salaries, stock options, and generate lots of media opportunities for high level staff. They have incentives to make the numbers look as good as possible, especially in the week before their IPO.
You only need RSI to get the 1-2yr timelines. The existing rate of progress seems sufficient to get to very broadly capable AGI in 2-4 years (and inputs should increase enough to maintain the existing rate of progress at least 2 years more, and then perhaps another 4 years at half the speed, getting us there in 2032).
I agree RSI might quickly peter out, and so not cause a huge speed-up. That said it seems to be already happening so some degree.
I agree we should be skeptical about what Anthropic says, but overall I wish I'd listened to them more in the past. I trust METR and they have seen a lot of the internal data and came away convinced.
The points you make about revenue mean that it will *under* measure the value of a technology (because a lot of the value ends up as consumer surplus). Arguably this supports my argument further (since it means that current AI revenues are understating how useful AI already is).
Revenue is a good lie detector and a bad job description. A firm paying for AI drafts that a person then checks line by line lands in the same number as one where the AI does the job alone. So $10 trillion in sales could mean digital workers, or a very large pile of first drafts. The trend can't tell those two apart yet.
A company pays the same invoice whether AI is doing the work unsupervised or a human is checking and correcting output that isn't yet reliable enough to ship on its own. The $10 trillion threshold needs the first kind of dollar: substitution, AI doing the work a human would have done. I'd expect a meaningful share of current enterprise AI revenue to be the second kind, augmentation, and the revenue line can't tell them apart.
The two could diverge: revenue keeps tripling while the autonomous-completion share of it grows much slower, if reliability turns out to be harder than raw capability. Is there a way to see substitution versus augmentation inside the revenue numbers, or is that invisible until someone tries to strip the humans out and the invoice drops?
Clear and sobering, in a useful way. If the trend lines hold, the question I keep coming back to is what shape that future takes. Several competing frontier models feels much healthier to me than one winner-take-all system, for the same reason we don't want one publisher or one newspaper. Do the trends say anything about whether capability stays spread out or concentrates?
Benjamin, I can grant you all three trends and still see no reason, on the strength of those trends, to expect broadly superhuman AI in 18 months.
First, revenue. You estimate what companies might pay if AI could do most office work, then treat that level of spending as evidence that AI can do most office work. The inference only works in one direction. A hundred million people might each pay for help with one part of their job and spend the rest of the day doing things the software still cannot do. Every buyer could be perfectly rational. That is quite a lot of cognitive science to ask of a sales figure.
Second, task horizons. A system that solves a problem that would take an expert three weeks has achieved something impressive. But who chose the problem, decided which assumptions were reasonable, and specified what would count as a solution? As a teacher, I could make a maths problem considerably harder while leaving the most important decision to myself simply by telling the class which method to use. Three weeks of work inside a given frame tells me remarkably little about who can recognise when the frame itself is wrong.
Third, recursive self-improvement. Researchers using AI to build better AI faster is a genuine feedback loop. The question is how much of the research judgment still comes from the researchers. If they identify the weakness, choose the experiment and decide whether the result counts as progress, then measuring how much faster the team works does not tell us when the model will be able to take over those decisions. Yet your forecast depends on precisely that distinction.
That leaves three problems your argument never addresses. The semantic problem: how does the system establish that an intermediate result means what it is supposed to mean? Two sign errors can give the right final answer. Reuse either intermediate value elsewhere and the error is back. The frame problem: what prompts the system to question the assumptions of a task when every success criterion it has been given rewards staying inside those assumptions? The safety problem: once an agent can modify parts of the system that evaluates it, how do we independently establish that a better score reflects a genuine improvement and that the checks themselves still work?
I would therefore leave the burden of proof where it started. I owe you no explanation for why progress will stop. My objection works just as well if revenue keeps rising, task horizons keep lengthening and AI labs keep getting faster. What you owe the reader is a reason to expect those trends to carry us across the gaps above. Eighteen months is a strikingly precise number for an argument that leaves the crossing unexplained.
"More importantly, the world is already rushing to scale robotics, and with AGI’s help, full deployment may only take another 3-5 years, and could be much faster. Explosive change just a couple of years later is not particularly comforting."
No this is not true. Happy to go into it more deeply as I digged into this for weeks. How quickly after ASI could you build an industrial state the size of let's say China? It's probably at least 14 years, but most of the probability mass has to be put on 25+ years
Your WW2 scenario lacks a lot of detail, things have changed between now and then. The Epoch models are alsojust unaware of how supply chains actually function, where they are, as well as are able to scale or not scale at what timeframes even with infinite labor in the world. Again, happy to go into it.
Reading this is a little dizzying, but I’m glad I did. When you’re not in the technical AI or AI safety community on a daily basis, you sometimes wonder if you’re crazy to see where these trends lead, because average awareness of the ever-changing research landscape is low. Zoom out from the models and RSI, and the money is moving too: markets reacting hard to Nvidia’s key-person change, in a scandal that’s also about chips reaching China; CoreWeave losing much of its value; Anthropic’s IPO shrinking. And the politics: “AI populism” at the midterms, anger at data centers, moratoriums on permits, and voters united on it almost everywhere, which may bring investigations and new rules. I wrote a few days ago about how these three pillars of AI uncertainty affect each other, because I think looking at each in isolation distorts the real picture. As you say, algorithmic RSI may not need more chips. Spreading it across the economy still needs money and permission. I’m curious how the next weeks, months and years will play out.
If you just extend the trends of the last four years of the 1960's, you reach broadly livable-level “moon base” in 2-4 years, and colonies in Alpha Centurii soon after.
Just because something has been growing exponentially does not mean nor imply it will continue doing so, almost everything faces diminishing returns. In 1902 there had never been a functioning areophane, in 1969 humans set foot on the moon. Today humans have not travelled to any new astral bodies. And getting a human to Mars is something we know is possible, creating a sentient AI through incremental improvements in LLMs is not.
I discuss this in the post. Sure at some point things will hit an s-curve, but right now we still seem to be on the steepening part of the curve, rather than the diminishing part. Moreover, the underlying drivers of AI progress (compute and R&D labour) seem set to continue to increase at trend at least the next couple of years. AI is also pretty unlike space travel because it generates revenue that enables further investment into it.
I take your point on space travel, but I’m still not convinced. The argument you are making is contingent on your implicit premise that recursive self improvement is a) possible at all, b) possible given our current level of technology, c) that AI researchers have already identified a model for implementing recursive self improvement which works, and d) that compute and R&D labour are the inputs towards RSI. If any one of those premises are untrue your argument fails.
The fact that measures point towards a steepening of the curve does not imply that recursive self improvement or artificial general intelligence is technically possible, that the current approach is the correct one to get there, nor that these measures actually measure what people claim they measure. For instance, revenue is a terrible measurement for technological progress as ancient technologies like baking bread, making metal locks, and selling lumber generate large revenues. It is also really important to recognise that Anthropic and OpenAI are not pursuing their research for the public good alone; they pay high salaries, stock options, and generate lots of media opportunities for high level staff. They have incentives to make the numbers look as good as possible, especially in the week before their IPO.
You only need RSI to get the 1-2yr timelines. The existing rate of progress seems sufficient to get to very broadly capable AGI in 2-4 years (and inputs should increase enough to maintain the existing rate of progress at least 2 years more, and then perhaps another 4 years at half the speed, getting us there in 2032).
I agree RSI might quickly peter out, and so not cause a huge speed-up. That said it seems to be already happening so some degree.
I agree we should be skeptical about what Anthropic says, but overall I wish I'd listened to them more in the past. I trust METR and they have seen a lot of the internal data and came away convinced.
The points you make about revenue mean that it will *under* measure the value of a technology (because a lot of the value ends up as consumer surplus). Arguably this supports my argument further (since it means that current AI revenues are understating how useful AI already is).
Revenue is a good lie detector and a bad job description. A firm paying for AI drafts that a person then checks line by line lands in the same number as one where the AI does the job alone. So $10 trillion in sales could mean digital workers, or a very large pile of first drafts. The trend can't tell those two apart yet.
A company pays the same invoice whether AI is doing the work unsupervised or a human is checking and correcting output that isn't yet reliable enough to ship on its own. The $10 trillion threshold needs the first kind of dollar: substitution, AI doing the work a human would have done. I'd expect a meaningful share of current enterprise AI revenue to be the second kind, augmentation, and the revenue line can't tell them apart.
The two could diverge: revenue keeps tripling while the autonomous-completion share of it grows much slower, if reliability turns out to be harder than raw capability. Is there a way to see substitution versus augmentation inside the revenue numbers, or is that invisible until someone tries to strip the humans out and the invoice drops?
Clear and sobering, in a useful way. If the trend lines hold, the question I keep coming back to is what shape that future takes. Several competing frontier models feels much healthier to me than one winner-take-all system, for the same reason we don't want one publisher or one newspaper. Do the trends say anything about whether capability stays spread out or concentrates?
It seems harder and harder to ignore where this is heading. I’ve put together some thoughts about the shifting balance of power here https://minnesotaconfluence.substack.com/p/who-or-what-will-be-on-your-side?r=bwkxp&utm_medium=ios
if the burden of proof is now on showing progress stops, wisdom needs a trend line too. so far nobody's drawing it