Discussion about this post

User's avatar
Steffen Rentschler's avatar

Benjamin, I can grant you all three trends and still see no reason, on the strength of those trends, to expect broadly superhuman AI in 18 months.

First, revenue. You estimate what companies might pay if AI could do most office work, then treat that level of spending as evidence that AI can do most office work. The inference only works in one direction. A hundred million people might each pay for help with one part of their job and spend the rest of the day doing things the software still cannot do. Every buyer could be perfectly rational. That is quite a lot of cognitive science to ask of a sales figure.

Second, task horizons. A system that solves a problem that would take an expert three weeks has achieved something impressive. But who chose the problem, decided which assumptions were reasonable, and specified what would count as a solution? As a teacher, I could make a maths problem considerably harder while leaving the most important decision to myself simply by telling the class which method to use. Three weeks of work inside a given frame tells me remarkably little about who can recognise when the frame itself is wrong.

Third, recursive self-improvement. Researchers using AI to build better AI faster is a genuine feedback loop. The question is how much of the research judgment still comes from the researchers. If they identify the weakness, choose the experiment and decide whether the result counts as progress, then measuring how much faster the team works does not tell us when the model will be able to take over those decisions. Yet your forecast depends on precisely that distinction.

That leaves three problems your argument never addresses. The semantic problem: how does the system establish that an intermediate result means what it is supposed to mean? Two sign errors can give the right final answer. Reuse either intermediate value elsewhere and the error is back. The frame problem: what prompts the system to question the assumptions of a task when every success criterion it has been given rewards staying inside those assumptions? The safety problem: once an agent can modify parts of the system that evaluates it, how do we independently establish that a better score reflects a genuine improvement and that the checks themselves still work?

I would therefore leave the burden of proof where it started. I owe you no explanation for why progress will stop. My objection works just as well if revenue keeps rising, task horizons keep lengthening and AI labs keep getting faster. What you owe the reader is a reason to expect those trends to carry us across the gaps above. Eighteen months is a strikingly precise number for an argument that leaves the crossing unexplained.

Boogaloo's avatar

"More importantly, the world is already rushing to scale robotics, and with AGI’s help, full deployment may only take another 3-5 years, and could be much faster. Explosive change just a couple of years later is not particularly comforting."

No this is not true. Happy to go into it more deeply as I digged into this for weeks. How quickly after ASI could you build an industrial state the size of let's say China? It's probably at least 14 years, but most of the probability mass has to be put on 25+ years

Your WW2 scenario lacks a lot of detail, things have changed between now and then. The Epoch models are alsojust unaware of how supply chains actually function, where they are, as well as are able to scale or not scale at what timeframes even with infinite labor in the world. Again, happy to go into it.

10 more comments...

Ready for more?