#250 – Toby Ord on where AGI timelines go wrong

Rob Wiblin: Today I’m speaking with Toby Ord, senior researcher at Oxford University’s AI Governance Initiative, and the author of The Precipice: Existential Risk and the Future of Humanity.

Welcome back to the show, Toby. I think it’s your fifth appearance.

Toby Ord: It’s great to be back.

Rob Wiblin: It feels like today everyone is talking about recursive self-improvement (RSI), or at least like everyone in my circles is talking about AI recursive self-improvement.

Do you think that RSI will lead to an intelligence explosion, like a massive takeoff in AI capabilities?

Toby Ord: No.

Rob Wiblin: Go on.

Toby Ord: [laughs] At least I think that’s unlikely. However, the chance that it might happen I think is credible. And this possibility really is one of the biggest issues in AI at the moment.

Rob Wiblin: So why do you think it probably won’t work or won’t pack much of a punch?

Toby Ord: There’s a couple of reasons. One is that I think that AI research involves a lot more than just programming, and a lot more than just running the kinds of simple experiments that have been automated so far. So I think that there’s a pretty reasonable chance that some major breakthroughs are still needed and that AI assistance isn’t going to be able to provide that.

Rob Wiblin: What if AIs do get better, such that they can more comprehensively replace the work that staff at the companies are doing?

Toby Ord: Well, there’s a lot of different types of work at a company — this includes, say, cleaning and catering, also programming and other aspects of software engineering, and then running some simple experiments.

But it also includes, I think, genuine strategic decision making and also AI research. That is not the case of programming — where you know what you want and then you type it in — but a case where you don’t know what you want, you don’t know what changes to these basic architectures will lead to solving some of these long-standing issues and really letting these systems take off.

And there’s not much evidence at the moment that AI systems can do that kind of thing. They might be able to really speed up the programming. But even if they made programming instantaneous, I think that may not actually change the timelines all that much. So they really would need to be able to help with these harder aspects of research.

Rob Wiblin: Is that the crux of the issue? If they could replace human researchers, like the full suite of things that the staff are doing, do you think then we would get an intelligence explosion?

Toby Ord: I think it’s quite plausible. There’s still some other issues. When you think of an intelligence explosion, you tend to think that this rate of progress over time is kind of bending upwards in some way, but most people think that then it will eventually kind of plateau off, and so it will start to bend back down as you reach some kind of limit, or maybe the increasing complexity of the system starts to overwhelm your ability to improve it.

So most people think it will bend back down to flatten off. The real question is, before that, is there a phase where it’s bending upwards? Or is it just the phase that’s flattening off? I don’t think that it’s clear which one of those it is — even if it could replace all of the human skills.

Rob Wiblin: What are the most distinctive aspects of recursive self-improvement that make it different than humans doing the work?

Toby Ord: There’s an interestingly long history of talking about recursive self-improvement — quite a few decades of talking about it, followed by now we’re at the situation where a lot of people are suggesting that they’re doing it.

This started famously with I. J. Good, who talked about this possibility of an ultraintelligent machine. What I think a lot of people forget about his story of recursive self-improvement — although he didn’t use that term, he used the term intelligence explosion — he was thinking of an AI system where, once you have an ultraintelligent machine, one that’s more outside the human range, it’s beyond anything that a human can do, and in fact it’s beyond what the entire research community can do; once you’ve got that point, he said that this is the last invention you need ever make because it would be better at everything by definition.

One of those things is building more intelligent AI systems. He thought that this would lead to this ultraintelligent machine building ultraintelligent machine number two, which then builds number three and so on — and that this would have an explosion, leaving humanity far behind, although perhaps it would also peter out at some very high level. That was the first idea.

Then there’s a big change to it by Eliezer Yudkowsky around about the year 2000, where what he considered was, maybe you don’t need it to be ultraintelligent to start this off.

In I. J. Good’s version, the AI community have had to get all the way up to ultraintelligent machines without any recursive self-improvement. But in Yudkowsky’s version, he thought, well, maybe if the one thing the system was good at was improving its own code, then you could have a system that’s below the human level and easier to build, but then it keeps improving itself and improving itself. And now that it’s so good at improving its own code, it works out how to improve other aspects of itself and then can ultimately blossom into this full set of capabilities at some very advanced level.

But his version still involved this idea that it was a bit like ‘AI improver’ is, say, your programme written in Lisp or some other kind of symbolic system. And then you run ‘AI improver’ on the source code of ‘AI improver,’ and you put that in a loop, and then you hit return — and then this thing whirs away and improves itself like a thousand times, say over the next day. And then you’ve got this superintelligent system with no real opportunity for humans to be in the picture anymore.

They’re the previous versions. Then the modern conception that the labs are talking about now, it’s more like you’ve got companies with thousands of human researchers and programmers, and also they’re building AI. And so far the AI’s abilities depend upon what these researchers do. But we could have a little bit of input from the AI itself where it kind of feeds back in this kind of feedback loop.

For example, at DeepMind they developed an AI system that found a slightly faster matrix multiply algorithm. Then the new chips for the next generation of AI could be, I don’t know, like one part in a thousand faster or something. It wasn’t that big a deal, but it was interesting. That was a first stage where the AI improved its successor. And there have been a few other things like this. The most notable these days is the AI coding systems that could take on a whole lot of the coding responsibilities for creating the next AI.

But we’ve got a system there where there’s humans and AI together, and the AI is not way below the human level, it’s not way above, it’s somewhere on the lower outskirts of the human level at a lot of things. And it begins by inputting a small amount into the process. But that amount could increase and increase until it’s mainly driving the process.

Rob Wiblin: Yeah, I think the companies imagine it now as a gradual process of replacing their staff, where gradually the AI is going to be able to do more and more of all of the things that the humans were doing. But I guess at the moment the AIs and the humans are incredibly closely integrated, they’re working together basically as colleagues, because there’s lots of things that one can do that the other can’t, and vice versa. What implications does that have that maybe shift people’s expectations relative to…

I think so much of this conversation has been informed in a way that people don’t always appreciate by the Eliezer Yudkowsky vision that has been so dominant in shaping the stories that people tell about how it’s going to play out. But if it’s not playing out that way, why is it different?

Toby Ord: I think the biggest change is one with respect to safety, which is that there’s much more of an opportunity for humans to remain in the loop.

I remember noticing back, I don’t know, 15 years ago with the Yudkowsky story, that he had this compelling vision about why AI could be really dangerous. And at some point I noticed that so much of the danger was coming from this recursive self-improvement concept. And I remember thinking, maybe there should be a moratorium on this about 15 years ago. But it was outside the Overton window to talk about not doing this thing that no one could do. So it’s a little bit hard to get these things off the ground when you’re too early.

But yeah, it was driving a lot of the risk because it was this thing where you had to get everything absolutely right. You’d had to understand human values and be motivated by them at the point where you start this process. And he was imagining a system that can’t even speak English at the point where you start the process. So how do you write something—

Rob Wiblin: Even begin to communicate, yeah.

Toby Ord: Maybe you write this English text that, once it’s halfway up, it then can read it and then it tries to act on that and so on — but really dangerous without being able to have any way to kind of interact with the system once it’s going, no ability to learn from feedback for us.

Whereas this new version is substantially more gradual, but also it has a lot more opportunities to understand what’s happening. We even have interpretability tools to try to understand what’s going on inside the systems while they’re climbing up this approach. So I think that it does look somewhat safer than these old versions.

Rob Wiblin: Something that I find quite frustrating is it feels like, I guess I and my colleagues have been thinking about this for years now, the recursive self-improvement possibility. It feels like we’ve gotten almost no greater clarity on how fast it’s going to go and at what level it will plateau.

Did you also have that view that we don’t know whether it will work, we don’t know how long it will last, and we don’t really know on what level it’s going to cut out or peter out if it does work?

Toby Ord: Yeah, that’s right. I’m actually working on a paper at the moment trying to understand these possibilities of really explosive growth, which I do think is credible and I’m concerned about it.

But a lot of that is trying to understand the dynamics. Is this the kind of explosive growth that’s exponential? Or is it the kind of thing that is faster than an exponential, such that it would have this kind of vertical asymptote — where there’s a particular time at which the level of this intelligence goes to infinity? That kind of dynamic, as opposed to the more typical compounding exponential growth from populations and things.

Some of the debate is over that. But even when you’re debating that kind of question, almost everyone in this area thinks that the intelligence is going to cap out at some particular maximum level anyway.

So to some extent the exact shape of it may end up being moot, if it caps out not that much above where it currently is, for example. Or there’s these questions of: sure, it’s an exponential, but is the doubling time on the exponential like a week, or is it a decade? That’s a big deal. And often the theoretical analysis doesn’t shed much light on that.

One of my colleagues, Tom Davidson, has great work on this, great theoretical work where he’s looking at all these questions about the theory of it, and will it have this vertical asymptote which gets called a ‘mathematical singularity’ — not to be confused with the more sci-fi concept of the singularity, although maybe they could be related.

But then his bottom line is saying something like he thinks that it’s most likely that there’ll be something like five years of AI-research progress that happens within one calendar year and then it starts to plateau, something like that. And it’s like, does all of this stuff go to infinity, or do this thing in the limit as time goes on forever, what happens to it? Then it comes down to some little rule of thumb, like it could go five times faster for a bit or something.

Rob Wiblin: It would be quite a hectic year, I suppose, if we had like five years of past AI progress compressed into one. But I suppose it doesn’t feel like it’s impossible that we would be able to manage that.

Toby Ord: No, and it also depends on where that leaves us. Suppose we had the previous five years compressed into one year. That would have been very disruptive, all from pre-ChatGPT through to where we are now. But it would leave us at a point with a somewhat manageable technology.

So it’s not clear exactly where this thing does plateau. It might plateau at a point that is not vastly superhuman and that is controllable. Or it might be that the five years takes us through that period where things get out of control. I think that would be a big deal. Obviously, the five years itself is just guesswork.

Rob Wiblin: You said that you thought in the picture that people have had historically, much of the risk came from recursive self-improvement. Why is that distinctively risky? And do those types of risks carry forward to the kind of recursive self-improvement that we might see in the next few years?

Toby Ord: If I cast my mind back to how this is being thought about before large language models, part of it was that there was this kind of question that got called ‘the value loading question‘ — where if you take the richness of human values and all the things we care about, and the way that if you miss out important aspects of the human condition, that it could lead to this impoverished future that is dystopian in some way.

Well, Stuart Russell was one of the few people who predicted what happened, which was that the AI system has read all of the books we’ve ever written, which are millions of pages of text. On almost all pages, some human is judging some other human about something, whether it’s Lizzy Bennet judging Mr Darcy, or what have you. So you can learn vast amounts about this. And that was outside of the previous conception because it was going to happen so quickly that it wasn’t clear there was any moment where you could get it to understand what we care about.

And also that’s one aspect, but maybe the broader aspect is that kind of feedback, some ability to actually say, if you’ve got some experiment, “the experiment’s going off the rails — quickly, pull the emergency stop button” or something like that, or to learn other aspects about how it’s progressing and to then be attentive to them.

Maybe if you’re planting a tree in your garden that you’ve never planted before, if you can go out every day and check on it and see that its leaves are wilting a little bit and adjust and so on, it’s a lot easier to make that tree survive than if you had to build a machine that will appropriately measure out water for it every day or something, and just turn that machine on and hope the whole thing works.

Rob Wiblin: So that’s the risk that stood out to people 15 years ago, I guess. What’s the picture now?

Toby Ord: Yeah, so now there’s a few different kinds of risks that come from this. I think the fundamental thing is that everything’s going a lot faster. That’s where a lot of the risk would come from.

Ultimately, speed itself is not quite the risk. Although if it did bring forward some apocalypse by a few years, then all our lives would be shortened by a few years, which would actually be a pretty big deal. But it’s probably even bigger than that because probably what would happen is it would change the probability that things go off the rails and that things would get a bad outcome.

And the way it could do that through sheer speed is that it could change the speed of the AI activities that are increasing risk, such as improving the capabilities of these systems faster. It could make more of an impact on that than it makes on some of the processes that are reducing the risk, such as technical research on AI alignment or other aspects like AI governance work. I don’t think it’s going to speed those things up as much as it would speed up the capabilities work of improving its own capabilities.

So if these things get out of whack, then what you’d expect is that for any particular level of capable AI systems, that our ability to control them is lagging behind. So we’d pass through all of these thresholds less prepared than we otherwise would be.

Rob Wiblin: So is recursive self-improvement itself risky, above and beyond the fact that it speeds things up? I guess I feel quite unsure about this. And it seems like people disagree at the companies and outside of them.

Imagine that it didn’t speed things up. I guess the obvious way that it’s more dangerous is that the AIs are doing the work themselves, and so it’s more difficult to monitor someone else doing something than to be doing it yourself. Is there more to it than that?

Toby Ord: Let’s call that the second one, the second source of risk. If you imagine producing one AI system, let’s call it GPT-7 and then that produces GPT-8, which produces GPT-9 and 10, and so on. If any of those in that line were to be misaligned, and to be able to scheme about what they want to achieve, then they would have an incentive to misalign the systems that they were building and to make sure that whatever its peculiar objectives were that differ from human objectives, that that was maintained in the future systems. That’s the second big source of risk, which is also a really big deal independently from the first.

And it could have different remedies for it. If you look at each of these different ways that recursive self-improvement could be risky, there could be different forms of governance or different kinds of rules applied at the labs in order to manage them.

So then a third one would be that, when humans create a new technology — say guns: first we create muskets, and then we make rifles, and then eventually we’ve got handguns and machine guns and things like this. And we’ve got this opportunity though to learn from the intermediate stages. So we don’t unleash the most powerful form of the technology on society all at once. Or with cars, we had much slower cars before they could go at the current speeds, so that gave us opportunities to learn. Also there were only a few of them around at first, instead of everyone having a car.

Having those kinds of intermediate levels is extremely useful for society. We’re actually really quite bad at predicting what’s going to happen if some new technology were to arrive and to regulate it in advance. But if we get to witness some of the ill effects on a smaller scale, then we can learn from that.

In this case, if things are going — let’s say they’re going five times faster — maybe instead of it being the model that would have been released in 2027, in 2027 we get the model that would have been released in 2032 or something. So we have some really big jump up there. If we get that, we don’t get the opportunity to learn from these intermediate things. And it could be that there’s one that really would have created a lot of havoc, but at a level that was ultimately manageable by society — but we really learned our lesson. And in this case we might not get to learn that lesson.

Rob Wiblin: Yeah, is there a fourth one?

Toby Ord: I think a fourth area is that, if you imagine the rate of progress just with human-only research in AI — let’s say that’s kind of ticking along — and then imagine that there’s a leading lab where their research is improving the capabilities over time. And then there’s a trailing lab that’s a year behind them. The difference in the capabilities of the leading lab and the trailing lab, it’s noticeable, but it’s only so big.

Whereas if you have this recursive self-improvement and the capabilities really bend upwards steeply at some point, then the first lab to go through that process, it could be that — if we go with Tom Davidson’s rule of thumb of five years of progress in one year — it’ll be then five years ahead on the old scheme, where the lab that was a year behind is. So that is more likely to lead to a winner-take-all scenario, which encourages racing, and also to lead to a final outcome where there’s much more concentration of power towards whoever gets there first.

I think that’s a fourth reason. There’s probably more. I’m hoping that people will take up this idea of actually trying to think about this, and trying to work out what are the different sources of danger, and then what kind of remedies would be required to deal with them, how much danger are they producing, and so on.

Rob Wiblin: I think people who are very sceptical that there will be any slowdown, the argument that they most lean on is just the competition between the US and China and the distrust between the countries will make it impractical to delay the arrival of things. It’s going to, if anything, inspire the government to want to push forward to superintelligence as quickly as possible in order to avoid being vulnerable to a country that they see as partially an adversary.

Do you think an agreement between the US and China is more likely than those folks believe?

Toby Ord: Yeah, I think, at least in the long run, I think it is quite credible. In the short run, could there be a deal between China and the US on recursive self-improvement, and how would that work? I haven’t thought about that one too much, but that could be a bit harder.

It does depend on the nature of the deal. But if the idea was something more like a moratorium on superintelligence — that we’re willing to go a certain distance into the world of AI and AGI, but that systems that completely dwarf the intellectual powers of humans are off limits — I think that such a deal would be very much in the interests of both countries from the perspective of AI risk, that once they realise that there’s a very credible chance that by allowing these things that they’ll be killed and that their children will be killed and their whole culture erased from the Earth, perhaps — if that is what the evidence suggests, then if people ultimately wake up to the evidence, I think it’s just in their own interests generally, if they can, to negotiate some way to not go there.

Rob Wiblin: So the game theory is a lot easier if both governments basically are extremely scared of superintelligence, and they think the risk posed by the AI itself, or the risk posed by AI advances, is large relative to the risk posed by the adversary country that they’re worried about falling behind against.

But I’m not necessarily sure that the countries will end up being as scared of AI as that, to make such an agreement easy.

Toby Ord: I think that it’s pretty likely that they will be. At the moment, it’s only people who are looking out quite far ahead who can see some of these possibilities clearly, and are willing to trust some of this extrapolation about what’s going on. That’s only a subset of people.

But I think of it like with COVID, where I remember — partly by reading some of your writing on it at the time — really tracking this early on and being very much ahead of the curve, and really looking at these graphs and working out how many cases there really were and so on. And overall, I would say this just brought me like two weeks ahead of everyone else.

So there were things that I was deciding — for example, to cancel my book tour, going to America for my book — but two weeks later it was just totally obvious. And like anyone would realise that you have to cancel it. In fact, planes might not have even been flying at that point.

And so I think there’s something here, where for us to see now that — in 2030 or something — these systems could be extremely dangerous requires something of a leap of logic or something. But to see that as we approach it will become increasingly obvious.

And so if we approach it gradually enough — such that before they’re at the level where they could take over the world if they wanted to, they’re at a level where they could do other amazing things at a similar level to taking over the world, and then before that they’re just a little bit below that, and so on — then it should be actually pretty obvious to people. You don’t have to look into the future, you just have to look into the present, really.

So I think that we neglect that. We think a rare combination of things to really take ideas seriously is needed. But I think that won’t be needed. And in fact, just like with COVID, just the natural self-preservation instinct of “I don’t want to die, I don’t want my children to die” will kick in, and will be enough actually. So it partly depends on the gradualness.

I don’t know whether that will quite be enough, but it will definitely make it easier and easier over time to see where this is heading, if it really is heading there.

Rob Wiblin: The other pushback I get on this notion is firstly, if you talk to people who have done negotiations with China or been involved in that kind of diplomacy, on the US side they’re unbelievably jaded. They feel like there’s a lot of bad faith on the Chinese side, that it’s very hard to get them to sincerely agree to anything and follow through on it. It wouldn’t shock me if on the Chinese side there’s also some scepticism about the US’s willingness to follow through on treaties.

And another concern is that the national security community, when you describe a new, incredibly dangerous thing, military people, their first impulse is not “we have to ban this”; it’s “we have to have this before anyone else.”

And we see that with the cyber capabilities of Mythos and Fable, that there’s a certain hunger to grab those powers ahead of anyone else. Do you think those are severe barriers?

Toby Ord: Yeah, there’s definitely a barrier. But reading, say, Situational awareness by Leopold Aschenbrenner a couple of years ago when that came out, it painted a very hawkish picture of America maintaining its advantage on AI and using that in order to be able to disarm and subjugate China — which might have seemed very attractive to hawkish people in the administration.

But if you think of it from China’s perspective — suppose that there is a technology that will mean that you can unilaterally just disarm this other superpower, and possibly have regime change and so on, and finally be able to get all of your policy objectives at this international level with them — then China would be in a position where its nuclear arsenal, the value of it goes to zero at the time when that system’s turned on.

If so, there’s a strong incentive to threaten nuclear retaliation for them to even cross lines approaching that level of power over them. So I think that this alternative strategy to having some kind of moratorium — the strategy of getting there first and then lording it over them and using that to disarm them — I think doesn’t really work because of this issue.

It’s actually very similar to the situation when Ronald Reagan was doing this Star Wars programme (a missile defence shield against Soviet missiles), where again, it sounds defensive, but once you have that shield, all of a sudden the value of Soviet nuclear missiles goes to zero. So they’re strongly incentivised to strike before you do that.

And so all of a sudden you’re creating this kind of first-strike incentive. I think that this is not a good strategy actually because it’s failing to take into account what’s the best response to this strategy, which is ultimately threatening nuclear war.

I don’t think that you can actually just seize this technology, at least not if your opponent’s paying attention. You could maybe only do it if they just don’t realise that the option’s available to them or something. But that’s usually when RAND or other senior people give strategic advice, they assume the best response from the opponents. They try to take this game-theoretic approach of: “Assuming that they’re not an idiot, what should you do?”

Whereas I feel that in this case it’s a little bit like moving your queen somewhere where there is a subtle way that the queen could get taken, but you just hope they don’t see it. And that’s not how this advice normally actually works. I think that there’s been a failure to consider the best response. So I think that counter to the idea of “we should get it first” doesn’t really work as well as people think.

Rob Wiblin: I guess especially not when you’re shouting about it in the press in an extremely easy way to pick up.

Toby Ord: Exactly. And it’s almost like: “We’ll move the queen here. It can be taken, but we’re hoping they won’t see it.”

Rob Wiblin: “We’ll write about it in The New York Times, but we’ll hope they won’t read it.”

Toby Ord: Exactly. I mean, I’ve heard that there are five different Chinese translations of that essay, for example. So they’ve definitely heard it.

I think that approach of “we just get there first and then have all the power on the world stage,” I think is overrated. There’s also this question of is it possible to verify it? And so on. How do we trust them?

And the jaded negotiators, it is a problem that they’re feeling like that. But I think that they’re jaded on discussing a whole lot of issues that are a lot less important than this. I would say this issue, to those negotiators, is more than 100 times as important as any issue they’ve ever negotiated before. If the negotiator was told they’ll get killed if they don’t succeed in this negotiation, they’ll be like, “Oh my God!” They won’t be feeling jaded at that point, right? They’ll be feeling, if anything, a little bit too excitable or something.

So because the stakes become really high and it’s very much in their interest to actually reach some kind of agreement, I think that it can push through that barrier. But there are these questions about how would you make sure that they’re holding up their end of the bargain? I think that the key aspect that you would need is basically to get it to be in China and the US’s interests to have such an agreement, if it could be verified. And I think it is. And I think that they will start to see that as we get closer.

Then we also need an ability to make the agreement actually bind them, and then they could police their own spheres of influence. So you might not have to get everyone else on board at first. So the key step, I guess, is the verification.

Rob Wiblin: Yeah. How would you do the verification, do you think?

Toby Ord: I think there’s a bunch of ways to do it, and it depends on exactly what the technology is at the time and so on.

People have often been thinking about some very technologically advanced verification mechanisms, such as having the next version of the NVIDIA chips be able to know exactly where they are in the world with GPS. And if someone blocks the GPS, then maybe the chip shuts down. It will only start working again if it can share where it is, and so on. I mean, that would be great if we get those things. Then, for example, there would be credible ways to prove that your chips aren’t actually working on advancing AI further.

But even I think people forget that the universe of possible ways of doing verification is very large. And I think the Cold War is really illustrative. For example, one of the key things was that the US and the USSR wanted to reduce the number of nuclear-capable bombers that could threaten the other person’s home soil.

Eventually some bright person, thinking outside the box, worked out that these bombers were on the runway, and they could saw them in half with a huge circular saw. And then, with tractors, pull the halves apart from each other so that spy planes from the opposing side could witness this and credibly see the destructive capacity be destroyed. As far as I understand, those planes are still on the runways, in halves, because someone was like, “Hang on, this is a way you could do it.”

That’s the kind of thing you come up with after you’ve been thinking about this for quite a long time. Whereas, at the moment, there’s a lot of people who’ve thought about it for less than one day, like less than eight hours full-time thinking whether verification is possible, who say, “It’s probably impossible, and so I guess we will all die.” I feel that is a bit of a ridiculous approach.

It’s reasonable to think it might be impossible to do the verification. Maybe we’ll think about it for years and we won’t find any ways. But I think that there are a lot of credible ways, and that’s an example that you couldn’t have initially done. It required the Overton window to move a bit before that one was possible — for the US to say, this will literally involve destroying capacity we’ve already built — and that would have been annoying at first, but once they realised that it would also destroy the same amount of capacity on the other side, you start to think even though we’ve already built it, it’s probably worth doing.

I think an example that would be very analogous to that is: at the moment, there’s a lot of chips that aren’t being tracked, and where we don’t know exactly how many have been smuggled into China and things like that, but you could have, say, like-for-like destruction of that capacity.

As a simple example, you could say the US agrees to bring a million GPUs, or a million H100 equivalents’ worth of GPUs, to Geneva, and China will bring a million there as well. And what they’ll do is they’ll both test each other’s GPUs to make sure they all work, and then they’ll both destroy them. That would be the simple version, but probably you could do better than that and just both put them into escrow. For example, that compute is now being used for some kind of humanitarian purpose by an international organisation, that they both have people on the ground.

Rob Wiblin: And anyone can inspect what’s going on.

Toby Ord: Exactly. But there are ways like that, where even if you don’t know exactly how many chips they have and so on, you could at the very least do one-for-one kind of swap offs like this.

And that’s the kind of thing you come up with one person just thinking for a while about it. I’m sure there’s much smarter approaches you could do as well. I do think that we just haven’t spent much time at all thinking about ways to verify, once the Overton window is broad enough.

Rob Wiblin: Once people are really motivated.

Toby Ord: Yeah. So that idea: I think now we’re at the stage where at least maybe some of the people listening to this will think, “Oh yeah, I guess you could do that.”

And maybe some other people will be thinking, it’s insane to destroy or give away these GPUs to this neutral party. But I think it’s the kind of thing that you can now think of as like, “Yeah, I guess you could do that,” whereas five or 10 years ago you kind of couldn’t even really think about that.

And as we get closer, I think just more and more things will start to seem like, “Hang on, if the alternative is that my children get killed, yeah, I guess I’m prepared to do this thing. It didn’t sound great at first, but now that you point out seriously bad alternatives, is it better than that?” And that the answer will be that, yeah, it’s way better than that.

Rob Wiblin: Yeah. I feel like the distinctive Toby Ord perspective over the last couple of years has been: “What if society does realise or come to think this is a very serious issue, it’s a very big threat, and what if they’re not idiots about it? And what if they do the sensible thing?”

It’s surprising that that has been quite a contrarian take, or quite a contrarian expectation, and not many people are imagining how that would play out. I guess it also requires that things be a bit gradual and not really explosive.

Toby Ord: This is another thing that goes back to that Yudkowsky kind of idea of an intelligence explosion that’s completely automated and happens really quickly. So because of that, he was imagining — when he first started ringing the alarm bells about AI risk — that it would take you from a system that was only of academic interest to a system that could take over the world if it wanted to, without humanity waking up to that outside of some narrow part of the research community.

Whereas what we’ve had is something much more steady, where the intermediate phases are visible. I guess it’s related to this idea that humanity is actually OK at learning from feedback, once we try things and we find out that they’re bad for us, we can adjust our course. So because things have ended up being more continuous than that, and perhaps they will remain so, you can then anticipate that these opportunities will arise.

Whereas recursive self-improvement, one of the problems is it may mean that we go from systems that are quite impressive but not amazing to systems that are all-powerful without that being visible. If so, then we’re back in the world—

Rob Wiblin: Back in the difficult world.

Toby Ord: Yeah, but I do think that a lot of people aren’t thinking quite seriously enough about how easy it will be to notice these things later on.

Rob Wiblin: If we’re going to have a moratorium on something — and we could imagine like multiple different levels of moratorium; it could start with just a couple of companies agreeing not to do something that they regard as a dangerous research practice, and then perhaps one government can get involved and then you could have an international treaty — what should it be about, what should it be on, and how would you potentially design it?

Toby Ord: Yeah, look, bottom line is I don’t know. I think that there are heaps of options that are all quite interesting. When I think about recursive self-improvement, here are a couple of different things that you could do.

One of them is you could say there’s some level of AI capabilities that we’re not going to just rocket through — that it’s high enough above our current levels that we would stop if we reached it. That’s one approach.

You could also have a speed limit. You could say there’s a certain amount of capability increase per year. Maybe it’s the speed that’s the problem, so you could set a limit that we’re not going to go faster than a certain amount.

There’s also ways that are somewhat arbitrary but also verifiable. For example, companies could say they’re going to do some AI coding, like every company in the world is doing AI coding. It’d be somewhat weird if the AI companies that are making the AI that does the coding are the only companies that are not using it. But there could be a limit on how much compute can be used for that. Say that they will not use more than some very large number — that’s somewhat arbitrary — of tokens on AI for AI R&D automation. So they’re allowed to use a whole lot of compute when training their models in the normal ways and so on, but this approach of “exactly how many tokens are being used per researcher” or something, at the lab.

And you could put a limit on that and people would ask: “Why is it here, and why isn’t it a little bit higher or a little bit lower?” But the point would be that so long as we all agree on some particular limit, then it could actually be fairly easy to check — at least within Western labs — whether they’re all following it or not.

Rob Wiblin: What do you think are the pros and cons of trying to set a speed limit vs trying to just stop progress completely, which is something that at least some people have suggested?

Toby Ord: I guess if the idea is to stop RSI at all, like if it’s some kind of moratorium about recursive self-improvement itself, then I think it’s very difficult to have a thing never get started, because I feel like they’ve already started. So you could say: “OK, you’ve started and the mistake was to let you start. And now we’re going to remove all of that — and no more AIs making better matrix multiplication algorithms, and no more AIs being used for coding of other AIs and so on.” You could try to roll it back, although it’s challenging to do so.

In general, there’s a question about which lines are legislatable and so on. And so that’s one reason why I’ve generally been a bit of a fan of compute thresholds on things, because I think that it’s easier to write them into law and it’s easier to inspect them and so on.

Whereas if you say you’re not allowed to do anything that counts as recursive self-improvement, which is any way that you use AI in order to help with the AI R&D process, there’s probably going to be a lot of corner cases and so on that are quite hard to police. So that’s a less good line to draw. Doesn’t mean it’s impossible, though.

It could also be that you say, yeah, it’s going to be hard to work out. What’s going to happen is that, if we think there’s a violation, it’s going to go to this set of five judges, and if a majority of them rule that it’s a violation, then it’s going to count as a violation. You could just say we’re just going to do that — like how courts actually decide law in the real world.

Rob Wiblin: So one source of benefit from a moratorium would just be slowing down progress so we have more time to learn through trial and error.

I guess a different reason to have a moratorium or some limitations on recursive self-improvement is so that you are more likely to retain a human in the loop who’s monitoring what’s going on and might pick up misalignment or attempted, I guess, sabotage of future models by an AI that was designing its successor.

Do you have any thoughts on how you can encourage that to keep occurring?

Toby Ord: Yeah, I think that there’ll be a lot of incentive actually to keep a human in the loop. There’s a couple of useful terms that came up, and do get legislated in the case of lethal autonomous weapons, which are like having a “human in the loop” and then also having “meaningful human control.” And I think that they should try to do both those things in this case. I’m not sure if they’re the best places to try to set the rule, but I think that there’s a lot of incentive for the companies to keep humans in the loop when it comes to these things.

Now, could the humans on the safety team tell if the AI — GPT-8 — was misaligned and was making a misaligned GPT-9? I don’t know. It’s possible that they couldn’t catch it even if it was.

Rob Wiblin: At least not directly, probably. But they might be able to use other AI tools to inspect the outputs and to have them debate one another about whether they’re faithful. There are various things that people have suggested.

Toby Ord: Yeah, there’s a lot of things. I know when Buck was on this podcast, he listed a huge array of different possible control tools, so I wouldn’t rule out that there are ways of achieving that.

But it might be that, of the four different ways I outlined, it makes things riskier to do recursive self-improvement. It might be that having a human in the loop as one of the possible remedies doesn’t help with this particular one of these ways that things get riskier.

Rob Wiblin: Back in 2023, there was a very famous letter suggesting that we should have a six-month pause on training new, more powerful models than GPT-4. And I guess in the meantime the request was that the companies get their house in order and figure out internal practices that would allow them to more safely train and release models going forward.

I decided not to sign the letter at the time because I wasn’t sure whether it was a good idea. I think with the benefit of hindsight — I suppose hindsight is 20/20 — but I think it would have had almost no benefits, or at least the pausing on the training I don’t think would have been very useful because it wouldn’t really have stopped progress: compute would have continued advancing, they’d still have been able to do many types of experiments, and I think we would roughly have the same capabilities now as we would have given that there was no pause.

So given that, in 2023, I think it would have been a mistake, there’s a question: why is it better now? Why do it in 2026? Why not wait until 2027 or 2028? I think there’s various things that you’re trading off.

One reason that it was premature in 2023 was that the models just weren’t capable enough to actually be a direct threat in any particular way. So there was no immediate risk that was being reduced. And furthermore, during the pause, the models weren’t yet capable enough to assist you with alignment research, with governance research, with really anything. They in fact weren’t actually that useful as it turns out. So in both respects, not very much would have been accomplished.

And so in my mind, what you’re doing is trying to optimally time the point at which you have some moratorium, so that you’re actually reducing some direct risk that the AIs might pose. In the meantime, they can help you speed up a whole bunch of other work that you want to do, that you want to speed up relative to AI capabilities. But you don’t want to miss the boat completely. You don’t want to delay this so long that things go horribly wrong before you’ve had a chance to slow them down at all.

Which is a difficult question. But I feel like the case that the timing is right is stronger in 2027 than it was in 2023, and it might be stronger in 2028 than it will be in 2027. What would you make of this optimal timing issue?

Toby Ord: Yeah, I mean, it might be stronger in 2028, or maybe it’s too late.

There’s a lot of people who are framing it like this, where the idea is there’s something like a six-month pause, there’s a certain amount of pause in capacity — let’s say at six months. And then the question is this optimal timing.

I don’t really like this framing of it. I did sign the pause letter back then, and that’s partly because it had this phrase in it. I can’t remember it verbatim, but it said, as the Asilomar principles, which many leading people in AI have signed — and in fact, I was at the conference and I signed them — one of the principles was that creating artificial intelligence that surpasses humanity in general abilities would be one of the most monumental things to ever happen in human history, and should be planned for and managed with commensurate care and effort.

And I signed that, and it’s obviously true. You can’t say, no, it should be an incommensurately small amount of effort compared to how grave an issue it would be or something. And then I thought, what is the current level of effort? And the current level of effort in 2023 was not very high. The labs had alignment teams, but the teams were something like 10 people. And that is not a commensurate amount of effort when thinking about something that would be one of the most transformative effects in the entirety of 300,000 years of human history.

And so I thought, actually, yeah, that’s right. I signed this principle. The principle’s clearly right. And the principle doesn’t allow us to continue at the moment, like when we’re getting into this zone.

The letter did heavily mention six months, but it also said at least six months, and it kind of implied because it said six months to give them time to do X, Y, and Z. And it did say “at least” a couple of times. So I interpreted it as the pause isn’t ending in less than six months. But if you don’t do these things, if it doesn’t look like humanity is spending commensurate effort on this thing, then it’s going to continue until humanity does. But it was a bit ambiguous.

But that’s the kind of thing that I’d be more in favour of. I think a pause, we tend to think of it as a time-limited concept in discussions these days. So I think that a “moratorium” is perhaps a better word: it implies, A, there’s a moral gravity of the issue. A pause kind of implies we want the thing to happen, like we want to end up at that destination, we just don’t want to end up at it quite yet. Whereas I don’t think it is clear that we want to end up at the destination of superintelligence, at least not for the foreseeable future.

There’s some serious issues we’d have to sort out before we perhaps condemn all future generations that could ever live to living under perhaps the yoke of these superintelligent entities that we’re unleashing into the world. And for those conversations to spread far beyond Silicon Valley, but also beyond the West — to involve people all over the world and to hear what people have to say about it — you could never get complete consensus or complete permission to proceed with that, but wanting it to be something that at least half of the people on Earth think is a good idea, as opposed to something that substantially fewer than half the people on Earth think is a good idea at the time–

Rob Wiblin: Or have even thought about it.

Toby Ord: Or have even thought about it. It just seems kind of crazy from any kind of legitimacy perspective of ushering in this thing against the will of almost everyone.

Therefore, I think some kind of a moratorium that’s more of the format of, “We won’t go beyond some point, either until some kind of standard is met or for the foreseeable future — but people in the future are welcome to lift the moratorium if they think we’re past that point.”

Where you could just say, “Look, I’ve decided we’re not going skydiving.” But that doesn’t rule out that maybe your child then convinces you by showing you a whole lot of evidence that skydiving is actually OK. And you’re like, “OK, skydiving is back on the menu.” You’re not saying, “Forever: though the sky may fall, we will never go skydiving” or something. You can say we’re making a decision not to do it, but you could revisit that decision.

I think something more like that, rather than just, “By default when this amount of time elapses, we’re right back at it” would be the smarter approach.

Rob Wiblin: Yeah. If you talk about moratoriums with people at the companies, I think the thing that they immediately want to know is: “What exactly do you want us to do while we’re doing this pause or while we’re slowing things down, what exactly is on the to-do list so that we can then release this moratorium?”

In a sense that’s good, because they’re the people who might be having to do a bunch of this stuff, so it’s natural for them to ask, “Exactly what are you requesting?” But do you think you would want to design it with that structure, that we need to basically tick all these boxes and then we’ll feel comfortable and then we’ll go ahead? Or do we want to say we actually don’t know when we’re going to feel comfortable?

Toby Ord: Yeah, perhaps more the latter, although gesturing at some of these boxes I think would be reasonable. And there have been some attempts at this. I think Yoshua Bengio has good stuff on this, suggesting not that it would have to be absolutely impossible that the thing would go wrong, but it would be kind of normal evidentiary standards for imposing large amounts of risks on a whole lot of strangers or things like this — that it would meet the normal standards that we apply to that.

I think that there’s ways of gesturing at it, but it could well be that we don’t currently know what those set of things would be.

Here’s another one of these principles from the Asilomar principles, which I think have generally been forgotten: it said that because advanced AI systems could pose some risk of human extinction, the expected consequences of this — so like the probability multiplied by the stakes — could be colossal and things should be planned for proportionately with those consequences.

Well, if you think that there’s say a 1-in-100 risk of that — so even just 1%, so you’re 99% sure it doesn’t happen, so almost as confident as you could possibly be that it’s going to be safe — that still is 80 million lives in expectation, which is bigger than World War II, and that’s just thinking about the one generation, not even taking into account anything about future generations. So the expected consequences are bigger than World War II.

So are we putting in commensurate effort with something at the scale of a World War II? We’re getting towards the perspective of putting in commensurate effort to World War II in creating the risk: all these people going out, building these data centres as quickly as they can and investing giant amounts of money. But in terms of actually avoiding the risk, and trying to actually pay appropriate attention to all of the human lives that could be immiserated or killed in this thing, not remotely commensurate to World War II, in terms of the effort. There aren’t millions of people who are working on trying to avoid this thing. It’s bigger than it was 10 years ago, but it’s very small.

And so I signed that principle. The principle’s obviously true. You can’t say, “No, we should have an amount of effort that’s incommensurately small compared to these stakes and the people who would be threatened by our technology.” But clearly we’re not meeting it.

You know, these principles were signed by Demis Hassabis and Dario Amodei and Sam Altman and Elon Musk. They all signed them. They’re also obviously true. You can’t say, “Back then we were so naive, we didn’t realise that you shouldn’t have incommensurately small efforts compared to the stakes” or something like that.

Not that you can bind people or something with this, but I think that you can use that to lead a discussion. There was a recent call for a moratorium on superintelligence, which didn’t really clearly define what threshold that would be, but the point of it — and I signed that — the point was to say, here is a kind of vague area of AI systems that far surpass humanity across the board at all cognitive tasks. And to say, we are not remotely ready for that. Should we begin the process of working out exactly what lines to legislate on and so on, because we definitely don’t want to go there?

And the idea was that people would say we definitely don’t want to go there. Exactly how far down the path we want to go is up for debate, but let’s start that debate. Let’s form the agreement that we don’t want to go to the end of the road, and then we can work out how far along the road it would be reasonable to travel.

So I think that you need to have those conversations. If people say, “But why not one step further along the road?” or something, and they try to catch you in some little paradox or something. I feel that that is missing the point. There are people who will just walk right off the cliff.

Rob Wiblin: I do definitely worry that if we impose some sort of moratorium too early, that there will be a significant backlash to this and people will view it as a failed effort and people getting far too anxious, getting anxious far ahead of time. And eventually the moratorium would be lifted, and this would then make it more difficult to do it at the appropriate time, when the actual direct risk and the ability to speed up the research that you want is much more promising. Do you worry about that?

Toby Ord: I think it’s a concern. It could well be true. For example, one way to say what you’re saying is that the longer you run the moratorium, the harder it is to keep it going or something. And so there’s some kind of counter-reaction to it which then makes you lift it, and perhaps lift it with it blowing some steam off from the thing you were trying to suppress.

But I think another possibility is that the longer you have the moratorium, the easier it is to keep it. And I think that’s probably right.

Rob Wiblin: Because it’s normalised?

Toby Ord: Yeah, it’s normalised. And also people have stopped doing the other thing. The people who had overinvested on these extremely bullish projections about what they’d be able to do if there were no brakes, they’ve all written off their losses, and other people aren’t champing at the bit to follow them and make a whole lot of investments that turn out to not be cashable.

So I think it’s not clear. Some people say it’s this limited resource that we should be very careful when we use it and so on. And in a world where it’s also very hard to know when that optimal timing would be, it’s not like we just know the optimal time — and also once we say “This is it, this is the optimal time,” how long will it be before it actually happens? It could be quite a while. So we might need to say, “Let’s go now” so that in a year’s time they actually do it.

Yeah, so I think there’s other people who say it’s like a muscle that the more you practice, or the more you’ve had this moratorium, the better you’ll get at it. I think they’re both possible, but we don’t know. So it could backfire and we should take that seriously as a possibility. But by the same token, there’s no certainty that it would backfire.

Rob Wiblin: Another way that I worry it would backfire is while you’re under the moratorium, unless we have a bunch of other rules as well, the supply of compute continues increasing enormously. Like we’re still going to be printing lots of chips and making lots of fabs and all of that, doing further research on chip design and so on. And so at the point you later release the moratorium, you would expect a whole bunch of catch-up growth as you train bigger models or now use the prohibited techniques because many of the underlying driving factors that were causing the progress have continued underground in the meantime.

And furthermore, on top of the fact that it might go faster at the point that you release it, you would expect that if you had a moratorium on particular practices, that you might get basically a whole bunch of different actors, a whole lot of different companies converging up to the level above which you can’t go. And so it’ll be like a more multipolar, a more competitive situation at that point — which could give groups even more reason to try to rush ahead. What do you think?

Toby Ord: It could be. I don’t think that’s impossible. Once you’re worried about that, there’s ways of adjusting the thing that could help deal with it.

For example, if ASML stopped producing extreme UV lithography machines, it’s not clear that there would be a growth in the amount of chips that the fabs can be producing per year and so on. So there’s possibly just one single company in the world, who being told that they can’t do the thing that they’re doing, would actually stop that issue. People often throw up their hands, like how can you possibly deal with this? And it’s like they don’t realise quite how concentrated some of these skills are.

Rob Wiblin: It’s basically just a handful of companies. I guess it does feel like a heavy lift, at least at the moment, I suppose, to imagine this.

Toby Ord: Yeah, but again, it’s the question, even if the people at these companies start to think that they will just die if they keep producing the things they’re producing? So we’ll see how far we go down that road. But it might actually not be that heavy a lift. Or again, the people who regulate them might have to just think, you know what–

Rob Wiblin: “We’re just going to bite the bullet.”

Toby Ord: Yeah, exactly.

Rob Wiblin: I think ahead of getting any moratorium on superintelligence or RSI, I think a type of moratorium that is plausibly on the table, I think in the next year, is a moratorium on a particular dangerous research practice that I think many of the companies are not enthusiastic about, which is unmonitorable chain of thought. So eliminating the ability to inspect what the models are thinking as they’re reasoning about stuff.

There are various research avenues you could go down that would make it impossible to read what they’re saying, or at least much more difficult to read it. And I guess there’s ways of making each forward pass so long that they could actually engage in quite a substantial amount of reasoning before you saw anything that they output.

I guess many different senior people in AI at different companies have said that they think this is not a great path to go down. And as far as I know, there’s no company that has gone very far down that path. So there would be no one who’d be super disadvantaged if they all could come to the table and say, “We’re going to set that one aside for now, because this is probably the worst risk/reward of any of the options that we have available.” So it’ll be an interesting step that we could take towards beginning to limit what research practices we permit.

Toby Ord: Yeah, I think that’s a good example. I remember at the time when one of the papers had just come out talking about how important this was, and my response was that people should take this really seriously.

If someone creates some new method — and publishes it and so on, and it gets used — that makes chain of thought illegible and gets some kind of substantial advantage out of that in terms of capabilities, then it’s possible that that’s the worst piece of research that’s ever been done by any human — worse than inventing chemical weapons, or various other things that have happened. They should really take this seriously, kind of like: “Are you the villain of the history books?” If we even have history books.

This is a big deal, and people should be very careful about it. And that feeling in their bones should be more common. And I think that this is an interesting case actually, because at the labs it has started to solidify over time. I think the longer you go without someone doing it as well, the more it seems like this is a norm, right? I think you get this kind of norm solidification.

A bit similar with the nuclear taboo — where for a couple of years after the end of World War II, it was just: we’re not in a major war, that’s why there haven’t been more nuclear weapons dropped. Then when the Korean War finished without more nuclear weapons, it was like: actually this seems to be a thing that we don’t want to break.

So, yeah, I think that a norm is kind of growing around this, and it would be good if it was solidified. There’s a lot of ways of doing that. There have been various papers with coauthors from different labs and things that help to try to solidify it. You can move beyond that to try to create industry best practices or standards. There could be standards and best practices around not training on chain of thought that also help to delineate some of the related issues, such as perhaps systems that can do more in a single forward pass, also making things less monitorable.

There’s a number of different approaches. I think getting more clarity on what they all are and then building these agreements, I think that it would be great if people did more work on that — of actually bringing people around a table, really getting the people to look each other in the eye and realise just how important it is and build up that trust that, yes, we need to push on this and we need to get to some point where we agree not to do it.

Maybe it possibly could have an escape valve on it, in terms of we will not do any research on this, but if someone else is out there with systems that have unmonitored chain of thought, then we’ll do it. “We won’t be the first to do this” or something. That could be a way that they’re more happy to sign up on, or something, assuming that they can actually tell if someone else has broken it. I don’t know the details about how hard it is to verify this one. It could actually be quite challenging.

Rob Wiblin: You might require whistleblower rules, I guess.

Toby Ord: Yeah, that could be the way to do it. But I think that would be great. I do like the example. It’s a smallish one, but it’s very focused, and it really does have bad risk/reward tradeoff and so on.

Rob Wiblin: Yeah, I feel like I’d really love us to urgently codify this, because I worry that you could have some kind of rogue researcher who I guess isn’t worried about any of this and just pushes ahead. And once you know the technique is there, once it’s so easy to implement, then I guess one actor might do it and then they’re like, “Well, I’m taking a relative disadvantage if we now agree to this.”

Toby Ord: That is indeed what they’ll all be saying. It’s also an interesting example because I think that things have kind of turned on… Not from everyone, but it’s like a lot of people don’t want to train on chain of thought.

They’re not thinking, “If only we could do this and get away with it, or something, that would be great. But we won’t because other people will…” They kind of find it distasteful or something. By this amazing fluke, compared to all of the thinking about this 10, 20 years ago, the AI systems, by and large, think in English and we can kind of observe their thoughts. If you’d said that to people 10, 20 years ago, they would have just been like, “It’s insane. There’s no way that that is true.”

Rob Wiblin: It would have seemed miraculous.

Toby Ord: What’s your best verifiable, or your best interpretability—

Rob Wiblin: You just read their minds!

Toby Ord: Like, “Oh, you just look at what they’re thinking.” It’s like, really? And yet we got that, it wasn’t even a deliberate alignment technique, I think.

I think that was something that’s just an absolute gift, like manna from heaven for the people who care about the safety of this technology. And giving that up or throwing it away for — I suppose it’s like a factor of two compute saving or something. That’s probably the kind of thing people would throw it away for. But you get factors of two compute savings every couple of months, through some kind of efficiency savings that new cache rules bring or something like that.

But maybe we will be so dumb as a collective group to throw it away for something so small, but we should definitely try not to in building these things. I think that once you get these norms changing, and once you get that people don’t even want to do it, that’s maybe the key to a lot of these discussions: is that lots of people are imagining people are champing at the bit in order to do this behaviour, and they’ve got all of these incentives, they make a whole lot of money, or they avoid going out of business when their rivals are getting too far ahead of them, or something like that.

But if you can make it so they don’t want to do the behaviour, if you can win the moral argument — that it’s bad, you’re a bad person if you do this behaviour; possibly one of the worst people who’s ever lived actually, if you create this technology — that changes things.

If you ask why isn’t anyone cloning any humans, or why isn’t there a whole lot of genetic engineering going on creating genetically superior humans? If you went back and looked at what everyone was saying from 1880–1920 or something, every intellectual group was talking about eugenics and it was just generally thought that, once the capabilities were there to understand how to actually change the makeup of people in future generations, that everyone would be doing it.

And I think they would be pretty bewildered to find out that we actually gained the abilities to do that and then just no one’s doing it. And they’d say, “Is that because you’ve got all these verification techniques and you’re in the labs in all these different countries just making sure that no one’s doing it?” And it’s like, “No, we did this thing–”

Rob Wiblin: Nobody wants to.

Toby Ord: We stopped wanting to do it. It’s like, well, how did you do that? And the answer is moral change and norms change.

People really undervalue this. I think that this is a different approach to trying to use narrow incentives, to try to resist the massive corporate incentives to keep building more and more powerful systems. That’s a really hard battle to win. But actually changing the norm so they no longer want to do something can definitely work.

Rob Wiblin: Yeah, you’re so idealistic, Toby. I feel like this kind of idealism is rare in AI discourse at the moment. Or the idea that well, we could just, for moral reasons, not do stuff. I don’t know. But there’s this analytical frame that you get into where it’s like, “Well everyone is going to just follow their incentives to do that.”

Toby Ord: Yeah. I think it’s rare in the world in general to think this way, but I think it’s powerful. And I feel like if you were an economist and you’re trying to understand, say, women’s suffrage in England back when those debates were happening, you would have thought there’s this currently kind of privileged class of people — men, or if you go back further, landowning men — and why would they share their franchise with other people? Currently they’re holding almost all the power. Why would they ever do this? And I think that something along the lines of “it’s the right thing to do” is actually a key aspect of what made those things happen.

Rob Wiblin: They’ve almost gone too far.

Toby Ord: I think it kind of has. I think it is part of the moral picture, but–

Rob Wiblin: It’s a bit overweighted, perhaps. And I feel that someone who is just predicting everyone’s behaviours as like narrowly self-interested, kind of psychopathic people in these economic models, I think they would have made bad predictions. What they do is then eventually, maybe in their utility function, one of the things they care about is like their wife, or one of the things they care about is justice, or maybe justice gives them a warm glow which is one of the forms of happiness. They can have a Mars bar or they could have some justice. They have these ways of trying to incorporate it in, in order to make sense of something after the fact, but they rarely think of that in prospect when they’re trying to predict whether something will happen.

Toby Ord: But it turns out that, “Why didn’t I do some thing? Because I thought it was deeply wrong” is often the answer.

Also that we can change what we think is wrong and we can learn that something’s wrong. My favourite example is with environmentalism: that up until say the 1950s or so, it just wasn’t part of what people thought ethics included. Then there was this very big change over the 1960s and 1970s, to the point where I think in the moral education at schools in Britain, the main moral education is kind of don’t litter, carbon dioxide, global warming, a whole lot of these things.

It’s maybe a bit overweighted, but it’s hard to remember how underweighted it was. But if you read certain things, like I was reading children’s books, I think Richard Scarry from the early 1960s. Boy!

Rob Wiblin: I don’t know this one.

Toby Ord: It’s like Busytown or Happytown or whatever, where there’s all those little animals going about their day. Then on one of the pages, they’re like, “Let’s build a road to this factory.” And there’s this nice little wilderness and they bulldoze it all down and they build hotdog stands along the road, and it’s just–

Rob Wiblin: And they celebrate this?

Toby Ord: Yeah. From today’s perspective, that would appear as a parody of this evil person who’s doing that. But that was just like, “Hey, we’ve now got our road and we can have a hotdog on the way.” And then the factory is belching out all of these fumes and things.

Rob Wiblin: It’s unimaginable in a children’s book today.

Toby Ord: Exactly. And it’s just a generation ago or maybe two now or something. But you know, moral change can happen in a big way, possibly even going too far or something. Yeah, people really neglect this.

Rob Wiblin: A lot of prominent people in the AI world think that AGI is coming very soon, like in the coming years. I know that you think things could take significantly longer. Why could it take until 2040 or even longer than that to get to AGI?

Toby Ord: There’s a bunch of reasons. A lot of the bullish projections at the moment are driven by the possibility of recursive self-improvement, and it’s possible that doesn’t pan out. That’s one possibility.

Another is the possibility that we decide not to do it, so that there is some kind of global moratorium or something else like that, where even if we could, we don’t. And in that case, I think that it’s just a very credible way that you get at least a 10% chance that it doesn’t happen by 2040, to do with choices made not to do it and some kind of treaty that then has some verification or something that’s actually blocking it.

But even if neither of those… well, I guess let’s suppose that there isn’t such a moratorium. It could just be that it’s a lot further off than we think. In the world of AI, a lot of the progress is tracked through benchmarks. You start off at just solving a few percent of the problems in the benchmark set. Let’s say it’s image recognition. And it started off with this MNIST [Modified National Institute of Standards and Technology], which was identifying individual digits, handwritten digits, like 4. And then progress kind of went from getting a few percent of them right to getting almost 100% of them right.

And then they started off with ImageNet, where it was more complicated colour photographs of little objects and things, like a cup. And it had to work that out. It went from just getting a few percent of them right, and it only went up through 50%, and then got to saturation.

Then you have more challenging image recognition tasks. For each benchmark you can often predict: it looks like it’s curving upwards now and it looks like maybe in a couple of years we’ll saturate out this benchmark and it’ll be getting almost 100% right. But it’s really unclear how many more benchmarks there’ll be before you get to where you want to go.

I used to think it’s just another benchmark away or something. I felt like there won’t be that many. And now I think, could there be 10? How many times do you have to go through this process? It’s very hard to actually know.

That’s one aspect: that people who think that we’re very close to getting something, where does this knowledge come from? That if we just go through this process again, say with mathematics or something, and this AIME [American Invitational Mathematics Examination] — that benchmark was amazing for the AI community, because it took them basically through the reasoning-models era, from late 2024 through to the end of 2025, they went from a couple of percent on this thing to saturating it. So they climbed this benchmark. But then at the end of that there’s still so much more to go. So that’s one aspect.

Another one is, if you think about something like the METR time horizons: this question of how long a task — in terms of how many hours it would take a human to do it — can the AI succeed half the time if it deals with it? We’ve seen this exponential progress in this number. Then people often think, once it hits a full workday, it can do eight hours of human tasks, an eight-hour human task, then it’ll be able to just repeat that or something each day. But it doesn’t quite work like that. And then they thought, maybe 40 hours? It’s like you can do a whole workweek or something. But it’s just not clear why any particular number corresponds to reaching human level or something on this metric.

So it’s something where we’ll have this graph and there’ll be people who say they believe in the god of straight lines on graphs or something: that if you see some kind of trend, that you should be betting that you can keep extrapolating the trend. Let’s just grant all of that and let’s suppose that you can completely extrapolate this trend. Well, how high does it have to get before we reach this kind of AGI level or transformative AI or superintelligence or something? Well, no one knows. Even if you grant the heroic assumption that you can extrapolate the trend as far as you want, you’ve got the straight line, eventually if it hit this level, you could read off the time at which it gets there — but we don’t know what level it would be.

And that leads to this big uncertainty. Even when we’ve got the data, none of the data comes with a thing that then you can just read off the year from that data. That if you just want to follow the data and follow the projections and do as little second-guessing as possible and just see where the data leads, it just doesn’t lead you to an actual year when this will happen. So that’s one of the reasons to really have uncertainty about it.

Rob Wiblin: I feel like that line of argument certainly gets me out to 2030. It probably gets me out to 2035. And then I’m like, 2040, 2045… I just have a degree of disbelief that things could take that long, given the level that the AIs are operating at now. And by 2040, even on conservative projections, we would have 10,000 times as much compute, maybe much, much more. We’d have done so many more experiments, there’ll be many more people working in the industry.

Do you share? Is there any intuitive disbelief or astonishment you have at the idea that it would take until then?

Toby Ord: Not really. One aspect is that the rate at which, say, models were getting 10 times bigger or 10 times as many parameters or 10 times as much compute being needed to train them and so on, that rate — like how many years until the next 10x — has started to slow down. And there’s reason to think that there’ll be a bit of a kink in the curve at 2030 or so, when a lot of the growth has not been how fast can we scale up the compute being manufactured, but how fast can companies open their wallets?

Rob Wiblin: And suck compute away from everyone else.

Toby Ord: Yeah. Part of it is: how quickly can we convince venture capitalists to invest a whole lot of extra money, or convince people we’re borrowing from, or convince the CFO of our company to make what would seemingly be outlandish investment decisions?

Then part of it has been how quickly can TSMC [Taiwan Semiconductor Manufacturing Company] convert the share of their manufacturing that’s GPUs instead of being chips for iPhones and things like that. How quickly can they increase that share towards 100%? But they can’t just make 10 TSMCs very quickly. The time needed to have 10 copies of their entire manufacturing footprint or something is really quite large.

And so a lot of this kind of early growth does taper off. We’ll be ripping through the orders of magnitude in scaling more slowly over time. We also run into this issue that, even if you have more compute for pretraining, the amount of human data is limited and we have already gone through all of the scientific papers that have ever been published, and all the books that have ever been written up until 2025. And then if you say, “What about the books from 2026, how much is that going to move the needle on the sum total of human knowledge or something?” Maybe not very much. So data limitations and so on.

There’s a bunch of reasons to think that, conditional upon scaling not giving you some extremely powerful level of AI by 2030 or by 2035 or something, maybe that’s a suggestion that actually you need something more than scaling at that point. And I think that there may well be additional capabilities that are needed, that may require actual research breakthroughs rather than just tweaking the algorithms a little bit.

Rob Wiblin: Our new host on the show, Tom Reed, recently wrote a piece arguing that he suspects that it’s not possible to have a recursive self-improvement loop that takes you to superintelligence just occurring in a data centre because you won’t have enough training data to become highly skilled at many different kinds of tasks. Because in order to do that, you actually have to try doing the task. You have to be deployed in the real world, in factories and in offices in order to get the training data and the feedback to become highly skilled at those things. And he thought this was going to be like a super important constraint. Do you buy it?

Toby Ord: Yeah. I read this piece, and it really frustrated me at first, and then something clicked and then I really liked it. So what I was frustrated by is that it felt to me like there was a conflation between intelligence and capability. It’s true that there might be a whole bunch of capabilities you can’t gain without a whole lot of access to real-world data and attempts to try them and learn from experience and so on. Especially the things that don’t have verifiable rewards, or at least they don’t have verifiable rewards in an artificial environment. You need to go to the real world to find out. And I thought, I don’t see why you couldn’t become extremely intelligent, but you would still lack those skills.

But that’s just really a rephrasing or something of his point, or a slight rearrangement of it. But it gives a version of it that I think is really interesting, which is that maybe you could reach a kind of superintelligent system in a data centre, where it’s training and learning a bunch of things. It gets really skilled at things like mathematics, and maybe by self-improving it gets really good at a whole lot of other things.

Suppose you did. Suppose there was no obstacle to how smart it was. And so the thing that came out of this data centre at the end of this, you could think of it like an amazingly bright student who’s just finished their undergrad degree; they’re going to go out into the workforce, and the world is their oyster. They could go into politics, they could go into business, into tech, into finance, into journalism. And let’s suppose they’re so capable, they’re the most capable ‘person’ applying for the job in any of these areas, but they would be applying for an entry-level job. So the most capable person with the most potential for journalism who hasn’t yet—

Rob Wiblin: Ever written anything.

Toby Ord: Exactly, exactly. They haven’t yet reported on a hot-button issue and then got a whole lot of flak or whatever and had to work out how to navigate it, and they haven’t worked out how to trade off the risk to the reputation of the newspaper they’re writing for vs the integrity to the facts and so on.

I think what I would say is, even if you could train something with recursive self-improvement to become extraordinarily intelligent, there may just be a lot of skills and capabilities that it won’t yet possess. And you wouldn’t fault its intelligence for that. You wouldn’t say it’s less intelligent because it doesn’t possess these skills. Suppose, for example, Tim Cook is stepping down from the CEO of Apple and they’ve already got someone lined up, but suppose that they didn’t and they thought this AI is superintelligent, we could have it be the next CEO of Apple. Well, I’m not sure about that. In the same way as if there’s a really bright student who’s just finished college.

Rob Wiblin: The most fast-learning 21-year-old, I guess.

Toby Ord: Yeah. You’d still say no. Actually there’s a bunch of experience about the culture of Apple, for example, that you would need to know in order to do this job.

And also a lot of relationships and trust that some people have built up. It helps you realise that it’s connected to the diffusion question of how much of a lag is there from having these amazing AI capabilities demonstrated to them, to actually being out everywhere in the world? Where it could be that to be a great CEO of a Fortune 500 company, that it does actually take a decade or more of just calendar time in terms of approaching new opportunities, seeing the business cycle happen, seeing people who are betting the wrong way get wiped out by the market, and it’s not just cases where you can learn from what’s happened in the past. There’ll be a whole lot of new conditions that have never existed before, including with AI totally changing everything.

So it made me realise that, instead of saying you can’t become superintelligent in a data centre, I’d say even if you become superintelligent in a data centre, that doesn’t necessarily make you super capable and able to do all of these jobs. What it would probably leave you with is being a great entry-level employee in anything, and advancing along their career trajectory faster than a normal human would. So you’re better than a relatively fresh-faced 21-year-old of any stripe or something, but not that you’re better than people with 30 years of hard-won experience or context on those jobs.

That could mean that that doesn’t say much about how transformed will the world be in 50 years’ time, but it does say that actually maybe if you had this amazingly intelligent AI in a particular year, maybe a couple of years later, the world isn’t that transformed because it can’t be doing most of the jobs yet.

So that did make me think about an additional delay to add into this kind of calculation of: at which point do you get recursive self-improvement, if it’s possible? Then how long does it take to reach a of highly intelligent system? OK, but then there’s another delay between how long does it take to have the highly intelligent system, let’s say, and it getting into a whole lot of companies.

Rob Wiblin: And having lots of concrete skills.

Toby Ord: Then once it’s in those companies, how long does it take from entering that company to understanding enough about these types of roles in order to be able to really deliver at a senior level of performance? It may add, I don’t know, somewhere between six months and 20 years, depending on the particular job of those areas. I thought that was very interesting.

Rob Wiblin: Yeah, I thought it was a very interesting point. I guess Tom sounded pretty confident that this was going to lead to big delays, and I think that might be right. But I guess I also think it might be wrong for a couple of different reasons. You could have really big improvements in the speed of learning basically by that point, or the sample efficiency is the technical term for how many examples or how much experience do you need to gain to figure out how to do stuff.

I guess human sample efficiency is actually unfathomable, extraordinary in some ways. I don’t often get to pull the parent card, but having a toddler, it is unbelievable that they can see like an example of a pig and then they’ll be able to tell all of these other things that look very different are also pigs. And this picture, this stylised picture of a pig is also a pig. I don’t actually understand how on earth it’s going on. I guess AI researchers don’t know either.

Toby Ord: It is incredible. I went through the same thing with my child.

Rob Wiblin: The out-of-sample generalisation.

Toby Ord: And you know, these drawings will be in the standard way that we depict things, we don’t teach the children how we depict things. But it is these line drawings, and things look nothing like line drawings. The line drawing is this extremely stylised thing, where the edge of a mug or something, that’s the only bit that you draw and you don’t draw anything in between. But there are actually no lines there if you look at the world. Yet they’ll see a drawing of a teapot and then they’ll be able to recognise one in the real world immediately. It is very remarkable.

I think that the upper bound on what is possible to learn with Bayesian systems and various other things is even better than the humans on this. But for deep learning, it is really an Achilles heel. Before it showed that it was just so successful once you pay the costs of doing all this extra training, a lot of people were dismissive of it because of this lack of sample efficiency, and that hasn’t gotten much better.

Rob Wiblin: Yeah, so that’s a point against what I was saying. They have awful sample efficiency but you could get like a million-fold improvement. We know it could be a million-fold better because humans are a million-fold better and we’re presumably not the very best that’s possible.

That’s one way that this sort of thing could not pan out is if, through recursive self-improvement or some other way, you have a different architecture for learning.

Toby Ord: It could be that they can learn from each other. It could be that there’s a lot of complicated things about being a partner in a law firm or something like that. But each year there’s like 10,000 deployed models that are learning and pooling their information. You could have a situation like that where that’s also enabling it to learn faster. Although if the markets are correlated and so on—

Rob Wiblin: Still only have experience of that year.

Toby Ord: Yeah, that will be helpful for rare things like a particular type of client who comes in and is really obnoxious or something. And how you deal with that rare case, it’ll be helpful for that like pooling it across many different firms.

But it won’t be helpful for the rare, like once-in-a-decade or once-in-30-years, events that happen and upset the whole thing. So it is by no means certain, but it’s quite possible that there would be substantial delays, at least for them doing these non-entry-level tasks.

Rob Wiblin: Yes. So that’s one way, is that the AIs could learn very quickly because many, many instances of them are deployed and they all collect data and they pool it together.

Another one is we’re already beginning to put video cameras basically on people’s heads and record all of their keystrokes. It’s possible by this time we’ll have actually a huge new data set of basically what lawyers do and how they think and what they look at and so on that they could learn from.

Toby Ord: Yeah, that could well be right. It is interesting that Meta were kind of a trailblazer on this one and have just recently gone back on it. The employees absolutely hated it.

Rob Wiblin: Oh, really? OK.

Toby Ord: It was also clear that they were just literally automating. It was a humiliating way of being automated out of your job, with like these cameras just watching it. The level of demeaningness to the person and so on is a real kick in the teeth for these people. They’re already pretty demoralised over there with falling behind on AI and so on. So they hated it.

Then there was, lo and behold, a big leak of information because it was logging every single thing that they did. I don’t know the details of it. But then when that happened, they were just like absolute revolt on this issue and we’re getting rid of it.

If we’re in a world where there’s increasing protests on the street about unemployment and so on, being driven by this — then the answer is–

Rob Wiblin: “We’re sticking cameras on all of your heads.”

Toby Ord: All the remaining lawyers who haven’t yet been automated in the firm and now have their cameras on them and so on, because the owner is planning to replace them next year. You know, people may not wear it, is what I’m saying. It’s possible. I don’t know which way that would go because there could be large financial incentives for the few firms who do it. It could be possible. Maybe you pay them larger than their entire salary in order to wear this camera because of the amount you’re making.

Rob Wiblin: It’s like a redundancy payment.

Toby Ord: Yeah. But if the public sentiment is just put sand in the machines–

Rob Wiblin: I could imagine it being banned.

Toby Ord: Yeah, it could be banned. It’s a good point. It might be that companies that try to do it are just shamed. I don’t know. But by the same token, I think there’ll be some of it.

Rob Wiblin: It’s another ‘it might or it might not.’

Toby Ord: Yeah, it’s hard to predict.

Rob Wiblin: And just to finish out my various objections, I think the last one was: some things I imagine you can learn through simulation, and I guess they already kind of do this. I guess that RLVR [reinforcement learning with verifiable rewards] is a sort of simulation. I don’t know.

But yeah, you can imagine if you have a really good internal world model, then you might not have to go and actually do these things in the real world because you can just imagine it. I guess it’s like I think humans learn partly through dreaming, and I imagine there’s some of this is like we imagine situations and react to them and that helps us to process the information that we have. And I guess perhaps AIs could do that as well.

Toby Ord: That’s right. Although there’s also another interesting case where, when people are trying to estimate how much they’re being sped up by AI, they tend to think of their hours in the office, so on-the-clock hours. But if you get an AI to do some task for you and you save four hours and it does it instead, you don’t learn from having done the task. So you’re not necessarily in as good a position as you would have been if you’d done it.

And then also there’s a bunch of thinking, certainly in academia, that people classify as like ‘shower thoughts’ or something — where you’re still only having one shower a day, even at Anthropic, where they’re being sped up by 4x — and there’s just certain kinds of processing of what’s going on. And perhaps more realistic than showers, although that’s perhaps where some of the thoughts come out because you can’t be reading a book.

Rob Wiblin: This is the one time you don’t have a screen in front of you. For now.

Toby Ord: Or similarly, going on a walk or something is another kind of famous example. But sleeping, you know, you still only sleep for eight hours. They’re not getting like three nights’ worth of processing of everything that you’ve been tracking and having it click into place in your mind or something. But a lot of — if you read biographies of scientists who’ve had major breakthroughs — it seems to be some of these aspects of they finally had a change in scene, they went on holiday or something, or they woke up and the thing was just clear to them, that maybe it had been processed.

We know that the brain does an awful lot of stuff during these dreaming processes and so on. And presumably it’s not nonsense and it is actually part of what makes this work.

Rob Wiblin: If you deny people dreaming, I think their ability to remember what happened the previous day is catastrophically ruined. So it’s a huge factor in learning, I think.

Toby Ord: Yeah. It’s worth noting that we’re not speeding those things up, and so it might be that some of the numbers that we’re getting are just really not the relevant ones.

Rob Wiblin: I’ve got this piece that we published recently where I try to make sense of all of the different updates that we’ve gotten about timelines to AGI through 2026. And one of the ones I had on the list was that OpenAI, one of their models — I think they haven’t named which one — made some progress on disproving a commonly believed conjecture about the unit distance problem, which I guess is like a possible sign that it’s becoming a creative researcher. It’s able to have original thoughts.

But I ended up concluding that I’m really unsure whether to be impressed by this or not. It’s cool, but it’s very hard to interpret. Do you have a take on whether this is impressive or not?

Toby Ord: Yeah, I think it’s impressive and I think that the mathematical community were clearly impressed. That said, it may have already been baked into your assumptions. A lot of people are assuming radically impressive progress every year, so seeing something that is really impressive may just be only enough to keep it on trend.

But this unit distance conjecture is, I think, a genuinely natural and interesting question. Ultimately, the idea is if you have a 2D plane and you can put some points on it, if you could put N points anywhere on this plane, how can you arrange them to make as many as possible exactly, let’s say, one centimetre away from each other? So putting them in a circle that’s evenly spaced would be a way to do it.

And that would let you get, for N points, you could get N unit distances, but it turns out you can get a little bit more if you put them in a square grid. And then the AI, it had been conjectured by Erdős, who’s like a very famous mathematician, that the square grid was basically as good as you could do, but then it showed that you could do something a little bit better. And so it didn’t prove a theorem, but it disproved his famous conjecture that a lot of people thought was true.

And this is unlike a lot of other results that AI systems had produced before this. This was one that genuinely a bunch of mathematicians had spent some time trying to solve. It wasn’t just like something that wasn’t solved because almost no one had spent any time on it.

So I think, pretty cool disproof of a conjecture. Unfortunately, the conjecture was quite elegant. And instead now we’re in this world where it’s like, oh, it’s more complex than that in some kind of slightly ugly way. But it did it, if you look under the hood, by combining this area of combinatorial geometry with another area of mathematics that was previously thought to be unconnected.

So in some level, it’s a bit of a shallow result because if you happen to know about both those things, like if you found out that the person who had done it was a human and they’d just been brought up in this other area where they became intimately familiar with this kind of weird mathematical structure instead of a square grid, this other kind of structure that you use, and then they were told about the unit distance conjecture and they thought, “Why don’t I use the thing I did in my PhD thesis?” And you’d be like, “I guess lucky that you had that combination.”

What was exciting about it for mathematicians was that there wasn’t a previously known connection between these two areas. It found a kind of connection which will let mathematicians import a whole lot of ideas from one area into the other area. They’re normally excited about results like that, when humans do them, and so they’re also excited here.

So I think pretty cool stuff. But there’s a bit of a feeling in the maths community that — and people vary on this — but there’s a bit of a strand of feeling that, “Whoa, there might not be any jobs for mathematicians, this really could be coming for us.”

And maybe it is, but I think it’s illuminating to think about the limits on this thing. This isn’t even a theorem; it just disproved a particular thing. And one of the reasons it hadn’t been disproved before is everyone thought it was true, so not many people had tried to disprove it. But let’s suppose that we kept going in this direction, and much further, and we had a system maybe in a year or two or something.

Let’s suppose we have a system that — if you give it a formal statement of mathematics, or make some mathematical claim — that it could prove or disprove it in a microsecond. OK, so we’re not going to get that. In fact, Gödel’s incompleteness theorem says you can’t actually quite get that. But let’s suppose you got it somehow, impossibly. Would maths be over if you can just take any claim and check whether it’s a theorem instantly? It turns out it’s very much not over. I think mathematicians are underselling the other aspects of mathematics.

Rob Wiblin: Yeah, what are those?

Toby Ord: One of them is like, why were we interested in the unit distance conjecture? It’s a pretty simple kind of claim.

But suppose you could prove one of these things per microsecond, if you started doing that now — I did some calculations on this — the stars would have burnt out by the time you even came up to the unit distance conjecture to even try to prove it. There’s just too many mathematical statements. There’s all kinds of statements like 1+1=2, 1+2=3, and so on. There’s infinitely many true mathematical statements. So you have to kind of greatly prune this space of statements to the interesting statements.

One way to rephrase that is like, what questions should we be asking? What are the interesting mathematical questions?

And it’s not clear that it can do that, that it can work out which questions to ask. At that point you’d still need humans to say, “This one is one of the ones I want to actually have the system spend its time on, not this exponentially growing thicket of uninteresting questions.”

So that’s the first step. Then beyond that there’s even richer things, there’s this question of: the AI systems can take some formal statement in some theory of mathematics — in this case discrete geometry — and then try to prove it, but they can’t create new theories of mathematics. So if you go back in time, say before Claude Shannon invented information theory: now, today, we could ask questions about the information that can be carried over a noisy channel and optimal coding and stuff like that. But we didn’t even know how to ask those questions back then. We didn’t know what kind of formal statements to have that would correspond to these informal and inchoate ideas that we hadn’t fully pinned down.

So when mathematicians invent areas like that, there’s a kind of extreme creativity. If you go back to the 19th century, there’d been thousands of years of geometry, and then in the 19th century mathematicians worked out that you could ask questions about things beyond three dimensions, like a four-dimensional cube: how many corners would a four-dimensional cube have? But they didn’t ask these questions prior to then. They also worked out, in the 19th century, you could have curved spaces, and they didn’t realise that we actually live in one, but they just thought it was an interesting mathematical question. They also kind of worked out about fractional dimensions, like things between two and three dimensions.

And there was this burst of creativity that you could ask all of these types of things. Or if you think about the origin of calculus, that we could ask about not just how high up is some curve, but we could ask about the slope of the curve and how that changes over time and so on. And that once they came up with these theories, all of a sudden there was this whole infinite range of new interesting questions you could ask. And the AI systems haven’t shown that they can do that at all.

So I think it’s instructive to see that, even if mathematicians spend a lot of time proving things, there’s these other layers that haven’t really begun to be automated.

Rob Wiblin: So they can’t do it now. But I would expect that stuff is coming soon. Do you agree?

Toby Ord: It might be, yeah. I’m not claiming that it won’t be able to do it, just that there’s this kind of thing that we think we can see this trend. We’re laser focused on this issue about proving things or something. Then we think that a mathematician, that’s what they do, and that it’s going to automate that away. But it’s easy to lose track of the fact that there are these higher-level questions, which I’ve always thought are the more important questions in mathematics. I’m less impressed if someone proves a difficult theorem than if they invent a new branch of mathematics where entirely new questions come into view and new concepts. That’s always been what I thought was the more impressive thing.

If we look at the history of automation of mathematics, for most of the time — until the 20th century actually — mathematicians spent something like half their time doing numerical calculations. Then the calculator automated all of that away, so it automated half of a mathematician’s job. And we don’t think that was a great shame or something. We also didn’t think we were on the verge of a singularity or something when we did that.

And then from about 1980 to now, symbolic manipulation, like solving algebraic equations and integrals and things like that, has also got basically entirely automated before the AI era — and again, that was then what mathematicians spent a lot of time doing. Now they don’t have to do it at all. They don’t even really talk about that very much.

But I think that, again, they were by and large freed from a relatively pedestrian part of their job — and that maybe if proving things gets automated, they’ll also be freed to be asking the questions and inventing these new theories. And these are areas where creativity is needed.

I think that those lessons aren’t just relevant for the mathematicians listening to this, but are potentially relevant in a whole lot of jobs. Famously in the case of radiology, the reading the scans bit we tend to, from a distance, think that’s what a radiologist does, but it turns out that less than half the time of a radiologist is spent interpreting scans. And it’s like, “Oh.”

I think there’s similar things with a lot of jobs, where we look at some component of it and see this curve of it being automated away. But we forget that there are richer and higher-level things, where it’s not that the humans are necessarily being pushed into the gaps of like… For example, some mathematicians talk about maybe humans will still be needed to explain to other humans what the new AI mathematicians have achieved. I feel like that is a case of being pushed… I think actually science communication is really good–

Rob Wiblin: You wouldn’t feel central to the story, I guess.

Toby Ord: Yeah, they would feel less happy about that. Whereas if they’re instead inventing entire new branches of mathematics, then the AIs are helping them explore them, I think they would actually feel pretty good about that.

We don’t know how long it will take before it can do those things. It could be that it’s just another year after or something like that. But I’m more pointing out that we don’t know, and that it’s a bit like with these benchmarks: we’re seeing proving start to get off the ground, where it’s starting to be able to prove nontrivial theorems. Maybe that will saturate and it’ll be able to prove all kinds of challenging theorems that would take humans years to do.

And yet we’d find out there’s another benchmark above it, which is asking the right questions, and then that starts to saturate. And there’s another one, which is inventing new fields. I think we’ve seen that a lot in the history of AI, that we start off thinking something like chess is… “AI complete” was this kind of term, which means it’s as hard as anything, such that if it can do that, it’ll be able to do everything because it’s like the last rung to fall or something.

Then people go and work on it and then they find out, oh no, actually they still can’t understand English at the time when they can master chess. Then we get to the case of Go or something, and it turns out they still couldn’t speak English fluently at that point. Then we think maybe speaking English is the thing. And then it’s like, oh no, we’ve got systems that can quite fluently deploy language, but they’re unable to do some other things.

So we genuinely don’t know how far that process goes. And there are very few attempts to systematically say, “Here’s the set of things and here’s how there isn’t much left.” Certainly I’ve found, as someone who’s generally been — compared to the average, actually pretty bullish about AI timelines — that I’ve noticed–

Rob Wiblin: How many times there are more things to go.

Toby Ord: Exactly.

Rob Wiblin: You wrote this piece not too long ago arguing that we shouldn’t think of ourselves as having short timelines or long timelines or medium timelines, but rather having broad timelines: we should embrace the reality that we don’t know when AGI is going to come.

Yeah, make the case for that. I guess you’ve already somewhat made the case for that, but is there much to add?

Toby Ord: Yeah. There’s kind of two things.

One of them is that there’s a lot of expert disagreement on this, like lasting expert disagreement. The experts just aren’t intractable. If you look overall, Helen Toner has this great piece showing that actually their timelines have shortened over the last five years quite significantly, I think by about a factor of 10 or something — at least from some forecasting platforms, have gone from 50 years to five years.

So it can change, but there’s still a very large amount of disagreement between different people. And those people come often from different fields, incorporating a lot of different forms of expertise. So obviously people who are AI specialists have very clear and relevant expertise in this, but also people who are, say, psychologists have a lot of relevant expertise. Economists have a lot of relevant expertise, if the question is will this be able to replace humans in labour and their role in the economy or something — they’ve seen a lot of waves of automation and are kind of experts perhaps at automation.

There’s a lot of different forms of expertise being brought to bear. The individuals who form their opinion are not aware of a lot of the hard-won insights that other people are bringing to the table. They’re not in a position to actually say, “I do not need to listen to your view on this.” There’s this question of what’s the chance the other person knows more than you? Half the time, if you take two experts, the other one had a better idea of the picture than you did, better or equal, but we’re assuming they disagree. And yet in order to have a narrow range of timelines, you really have to be excluding a lot of people and saying, “Nope, you’re wrong and I’m not interested really in hedging a bit by moving towards what you’re saying.” That’s one point: lasting expert disagreement.

Rob Wiblin: And yeah, actually I can’t remember: what’s the other key point?

Toby Ord: The other thing is that if you look at forecasters on this, the best forecasters I think don’t just give a point estimate for when it’s going to happen, but they explain their own uncertainty. The ideal way to do this is by having a probability density function, like a bell curve or something, although it doesn’t have to be symmetrical, that shows exactly how much probability in which year. And the lay version of this is a confidence interval — to say “somewhere between this year and this other year.” But the probability distribution is even nicer if you can get it. A lot of people have actually sketched these out. For example, the AI Futures Project — who gave us the AI 2027 scenario — they’ve got some very nice and interesting modelling of recursive-self-improvement-driven AI progress.

And the two leads on that, Daniel Kokotajlo and Eli Lifland, have got their own probability distributions you can look at. And Kokotajlo is generally thought of as—

Rob Wiblin: About as bullish as you can get.

Toby Ord: Yeah, exactly. Real short-timelines person. But if you look at his distribution — and he’s recently changed it to actually be a bit more bullish in the last few months — but his distribution, he thinks that there’s a 10% chance it will happen within nine months. That’s pretty bullish. Then his median estimate, so the 50/50 point, his over or under number, I think it’s four years at the moment. Then he thinks that there’s a 90% chance it will happen within 14 years, which is taking us up to 2040. But he would say there’s a 10% chance it goes beyond 2040.

So I like to think about these estimates in terms of what I call the 80% confidence interval. We’re often familiar with a 95% confidence interval in science, like ranging from the 2.5 centile up to the 97.5. Here I’m just thinking actually from the 10% to the 90%. So the kind of confidence interval where you think there’s a 10% chance it’s earlier than anything inside this window and there’s a 10% chance it’s later than anything. So there’s only an 80% chance that it actually happens in the window that you specify. We’re not asking for giant amounts of confidence. It’s very possible that it will fall outside this. So Daniel’s one is from nine months to 14 years. That’s pretty wide. It’s maybe at the narrower end of what I’d call broad timelines, but I think that that counts as an example.

Ultimately that’s about a factor of 20. He’s kind of saying there’s a factor of 20 uncertainty in when this thing will happen, between nine months and 14 years. And he’s saying that there’s a 20% chance it doesn’t even lie in that giant window that he’s created. So that’s pretty broad already.

If you look at other people’s, they’re also really broad. There was a summary that Epoch did of I think 10 different forecast methods, and all of their 80% confidence intervals were more than 50 years long. And I think almost all of them had more than a factor of 10 between the early number and the late number.

Then also if you look at this big Katja Grace et al. paper, where they surveyed a whole lot of experts from people who were presenting at the major machine-learning conferences that year, and they surveyed more than 2,000 people, they had another vast range where the 10% number, I think, in that case was just four years, and the 90% number was beyond the span that was being looked at — so more than 100 years. Again that’s at least a 25-times multiplier.

My own 80% interval is something like, I can’t remember, I did work this out recently, it says something like two years to more than 100 years. I think three to 100 was my estimate. So 30-times multiplier.

So people are trying to say, with these attempts to actually express their uncertainty, that there’s these wild levels of uncertainty in these things.

Rob Wiblin: In the piece you lean reasonably heavily on the fact that I guess it seems like even the most bullish-timelines people do have reasonably wide confidence intervals. I’m a little bit nervous about over-relying on that, because I guess in the case of the AI Futures people, I think they’ve spent a lot more time modelling the rapid timelines than they’ve modelled the longer timelines. So I worry that under the hood what you would find is that they’ve just said, “…and there’s a chance that it takes a lot longer and we don’t know” — it might not be as considered as the rapid recursive self-improvement scenarios that they write about.

Another reason is, maybe they’ll say it could take much longer than 2040 or 2050. I’m not sure whether they’re saying it won’t be technically feasible until then because it just requires an insane amount of compute, or we don’t have the data — or they’re saying there could be a war between the US and China that destroys civilisation. Maybe that’s why it takes 100 years. It’s very different implications, I think, between these two.

Toby Ord: I don’t think they’re saying much about the latter. In general, I worry that a lot of people who are forecasting this are largely saying, in business as usual or something, when would we have the capacity to create these advanced AI systems if we wanted to? Or something. Such that if their moratorium happens, and then we reach 2040 and it hasn’t happened — as in we haven’t reached this level of transformative AI — I worry that a lot of people would say, “Yeah, well, I’m not wrong because this thing got in the way.”

Rob Wiblin: “Because we could have.”

Toby Ord: “We could have done it.” Yeah. That’s not the forecast I think that they should be making, or at least if they are, they need to be super clear about it. Because I think the forecast that matters for most people is when will we have these systems? Not when would we, if we’d behaved optimally in a certain sense and ignored how dangerous it was, when would we have them? But rather something more like when will we have them? Although it does depend on the use.

Rob Wiblin: Yeah, maybe I’m focused on the technical feasibility question a bit more because I’m thinking this creates a deadline by which we have to have figured out a bunch of stuff, like being able to coordinate to decide whether we’re going to go ahead with it or not, or figuring out some way to do it safely.

Toby Ord: Well, it depends on the question. Suppose the question is alignment, and what you’re thinking is that should I invest in just kind of tweaking the current alignment paradigm, or working within this paradigm, or should I invest in creating new ideas: like exploring and finding fundamentally different ways of doing alignment that could have better guarantees? Like Yoshua Bengio is doing with his project.

If you’re thinking about that, it is relevant if, suppose, that there isn’t this kind of transformative level of AI by 2035 because we decided not to do it. That does mean that you have more time to actually do things.

Rob Wiblin: Figure things out and become relevant.

Toby Ord: Yeah. If instead you said, “But because we could have done it, I’m going to then ignore the possibility of these higher guarantee type methods, this exploring the space of ideas, and just focus on tweaking the current things.” I think that would mislead you. I think that for many purposes, actually what you want to care about is when will these systems exist? Rather than when could we have had them exist?

There’s also a bit of a thing that annoys me, where they’ll never know that they’re wrong if they do that version because they’re saying we could have if we hadn’t had that pause or, if the companies had just leant more into recursive self-improvement and they hadn’t chickened out on it, then we would have had it or something.

We’ll never know if that’s right or not. A lot of the idea that the forecasting community prides itself on is actually falsifiability by some date. All of the versions that say, “If a whole bunch of things that are not even fully well-specified happen, if we’re in one of the normal cases, it will happen by this point.” And then they’ll say, “I guess we weren’t in one of the normal cases, so my forecast is moot.”