Ep163 "How Quickly Does AI Become Smarter Than Humanity?" with Ben Goertzel

0:00Speaker 1:What is artificial general intelligence and how close are we to getting there? And how do we define what comes after that, which is often called artificial superintelligence? And why might the gap between those two be dangerous for us humans when machines become smarter than us. What roles are left for humanity? Should we think of AI as software or as something we're raising like a child? Will tomorrow's AI destroy us or ignore us? Or protect us? Today we talk with computer scientists and AI researcher Ben Gertzel, who over two decades ago introduced the term artificial general intelligence.

0:46Speaker 1:Welcome to Inner Cosmos with me, David Eagelman. I'm a neuroscientist and author at Stanford, and in these episodes we sail deeply into our three pound universe to understand some of the most surprising aspects of our lives. One of the many remarkable things about the brain is that it spends a lot of its time confronting situations that it's never encountered before. So every day you're walking into conversations you've never had, You solve problems that you've never seen.

1:30Speaker 1:You recognize something like a face or a building from an unfamiliar angle. Sometimes you might invent a joke that no one's ever said before, or you step onto an icy sidewalk and you instantly discover a new way of balancing. So life is full of novelty, and you're never explicitly taught how to deal with every situation you encounter. Your brain wasn't programmed when you were a kid with a lookup table of every possible circum stance. Instead, as I discuss often here, brains build internal models of the world, and those models are used to improvise when reality throws something unexpected your way. There are a lot of competing definitions and ideas about what intelligence is, but this ability to creatively improvise and extrapolate to new approaches this is likely to be one of the defining features of intelligence.

2:31Speaker 2:Now.

2:31Speaker 1:For decades, what we called artificial intelligence didn't really look like that at all. What we saw instead is that computers excelled at really narrow tasks. They could beat us at chess, or recommend movies, or recognize faces or translate languages. Each success was super impressive in its own domain, but every system was essentially confined to the job that it had been built for. When something extraordinary happened. Large language models arrived, and suddenly we found ourselves conversing with machines that could write essays and compose poetry, and debug software and generate images and explain quant mechanics and carry on conversations about essentially any subject you could think of. Now, speaking as a scientist who's been in this field for decades, the progress has been breathtaking. We have all been witnessing this.

3:29Speaker 1:Every few months we see the capabilities leap forward again, and every week we see some announcement that would have sounded like science fiction a few years ago. But even with all this genuine excitement, there is a key question for us. Are these AI systems actually intelligent or have we simply become really good at building machines that absorb vast amounts of information to predict what comes next. This is a really important distinction when we're asking about intelligence.

4:03Speaker 1:So let's do a consciousness raiser on this. Imagine that we have a mutual friend named Fred, and Fred memorizes every textbook ever written, and he memorizes every novel and every scientific paper, and every movie script and every recipe and every legal case and every conversation posted online.

4:24Speaker 2:He reads it all, and he remembers all of it.

4:28Speaker 3:Now.

4:29Speaker 1:Fred would certainly appear absolutely brilliant in conversation. You would want him at any dinner party. He can answer questions on almost any topic. But is that everything that we mean by intelligence or might something still be missing? This question, of course, reaches beyond computer science because it forces us to ask what our own brains are doing. When you get struck by some insight for a new way of doing something, where does that come from in the brain.

4:59Speaker 1:When a scientist invents a new theory, where a musician creates a new style, or when a child discovers a solution that no adult had suggested to her, what kind of computation is taking place inside the skull? What this all points to is the idea that what we think of as intelligence isn't only about how many answers you've already socked away, but instead it's about what happens when none of your previous answers work. Today's guest to spend decades on this topic. Ben Gertzel is a computer scientist and an AI researcher and an entrepreneur, and one of the people responsible for popularizing the term artificial general intelligence or AGI.

5:47Speaker 3:Long before the recent.

5:48Speaker 1:Explosion of AI, he was asking what it would actually mean for a machine to possess general intelligence instead of isolated skills. He's one of the field's most provocative things. And in our conversation, we're going to explore what separates AI from AGI. We'll ask whether language models are enough for whether something fundamentally different has to get invented. We're going to examine what happens after human level machine intelligence arrives, and why the transition to super intelligence may happen much faster than many people expect, and what all of this could mean for society and education and work and our sense of purpose.

6:31Speaker 2:A lot of the things that.

6:32Speaker 1:We're going to be talking about feel futuristic, but boy, they are becoming present tense quickly. Here's my conversation with Ben Gertzel. Okay, so Ben, everybody's talking about AI. You and your colleagues in two thousand and five introduced the term AGI. Tell us what AGI is?

6:55Speaker 3:Sure? So AGI as we intended it when we title that two thousand and five edited volume Muntificial General Intelligence, I mean AGI referred to AIS that can generalize beyond their experience to a quite significant degree. In more technical terms, you would say they can learn out of distribution, like there's been some probability distribution of knowledge in their life history, and then the world throws something that's unpredictable by that probably distribution that AGI can pivot and deal with that with some measure of what you might call imagination. And the important point to this relative to the current commercial AI scene is it's not just about the AI being able to do a lot of stuff, because of course, being able to generalize is one way that you can become able to do a lot of stuff, but as we've seen recently, being trained on the entire Internet is another way that you can become able to do a lot of stuff. And if your ability to generalize is only modest.

8:04Speaker 1:What do you see is the difference between AGI and modern AI.

8:08Speaker 3:It sounds to me like it's.

8:09Speaker 1:The creativity piece, it's the out of distribution piece.

8:12Speaker 3:Yeah, I mean, there was a Microsoft paper a few years ago called Sparks of AGI which was arguing that lms show little indications of the ability to generalize out of distribution, and I think that's true. I mean, I think llms are like non trivial progress toward AGI, yet they seem to be sort of plateauing in their ability to generalize out of distribution while increasing in capability in many other ways. Right, And so I think that transformer neural nets, which is the particular AI architecture underlying llms like Claude and open AI and whatnot, like, I think these are probably not capable of like leaps of creative imaginations such as as the human brain is. But what's subtle is they can generalize a fair bit beyond their training distribution. They're not totally inert in that way.

9:14Speaker 1:Now, one thing that I have found interesting is that human creativity is in some sense a remixing of everything that we've pulled in, meaning you know, you bend, break, blend all the information that's pail. And so there's an open question, I think about whether we are actually ever operating out of distribution.

9:34Speaker 3:So an example I've used before is in the use of AI to model music. And because I'm a weekend musician, so I mean it's natural domain for me, and I've worked with AI models for generating music Udio Sonos, I've trained some of my own models. And what you find there is you can generate like a twelve bar blues song that's quite good right now. I mean, if you look at Led Zeppelin, Jimmy Page Azz Since I've been Loving You, that was a blues song that was actually original, right, which Jimmy Page always said he was proud of that song because it's so hard to do something new in the blues, right, because so many things have been done before, and there's a certain structure you have to follow. Now, I haven't seen any ai model come up with a blue song of that level of originality, but they can spin out twelve bar blues songs very competently. But the thing you could think about is if you took a deep neural net style like transformer neural net style model of music and trained it on all music up to the year nineteen hundred, so it's our only music up to nineteen hundred, you know, classical music, West African drumming, Balinese, Gamalan, whatever,

10:48Speaker 3:but nothing after nineteen hundred. I would submit that neural model will never invent neoclassical metal, it will never invent hip hop, It will never even invent you know, Duke Ellington's style jazz, right, It will recombine things on a shallower level. So if you if you ask it to put West African drumming and Western classical music together, like you'll get some funky Beethoven to a polyrhythm or something. And so in a sense you can say, yes, jazz was a combination of Western classical music theory with West African rhythms, but it was a combination at a different representational level than transformer neuralnets or are representing things. So yes, we do combine and munch things together, but we combine and mung them together based on abstracted representations of what they are and based on what they mean to us as embodied creatures. Like we don't. We don't just combine together the surface patterns, right, And I mean and that that is, I guess, kind of a subtle difference, but it's it's a quite real difference, right. And you can see that if you teach kids too, like our kids just like combining stuff that they class or are they

12:02Speaker 3:engaging with the material in a real way, Which doesn't mean what you're doing is like radical novelty out of the blue with no resemblance to anything you ever learned. It means you're leveraging what you learned based on a deep understanding of what you learned, which lms are not really doing. But they're still doing amazing stuff and in their own way.

12:24Speaker 1:And so the way that you see the field now is LMS by themselves are not going to get us to a level of AGI.

12:31Speaker 2:Now we know that lllms.

12:33Speaker 1:Of course we're getting combined with all sorts of other tech, right But even with that, right now, do you feel like, hey, we're clearly on the road to AGI in that direction, or do you feel like other efforts will get us there.

12:46Speaker 3:I think it's somewhere between the two. So one approach to making an AGI in the modern world would be taking LLLM and just add a whole bunch of other tooling around it, right like at a formal very fire to check if the math work at a simulation engine to give a spatial world model. But then the LLM is sort of the hub, right And another approach would be a sort of hybrid architecture where an LLM is one component and you have other components that are equally or maybe more important, like a world modeling engine, a symbolic reason or a different kind of recurrent neural net that's making simple models of the world whatever. But an LLM is still playing a different role. And then there's people pursuing utterly different approaches. They're just like, well, LM is a nice data model or just like a calculator, you know, it's a nice number cruncher, but general intelligence is something different. And it might be that you could make different kinds of AGI minds in all these different ways.

13:52Speaker 3:My own sense is that the most practical approach is the hybrid approach, where an LLM is contributing something but isn't necessarily the centerpiece that's sort of coordinating the minds overall activity. And this is basically because lms don't understand who and what they are, right, which is almost weird to have to say, but like the human baby starts to understand who and what they are before they can do advanced mathath and and and write essays and and and you know, bring down the nssay's computer systems and all that. And the advantage of knowing who and what you are and what world you're and who you're interacting with before you learn advanced stuff is you then represent the advanced stuff in terms of who and what you are right, like our self understanding, our understanding of others guides our knowledge representation about anything else. Right, ands lms aren't doing that, right.

14:52Speaker 1:I would throw in two things there, which is what's interesting as a result of that, is that we pursue things that are relevant to us and mean things to us that might be bad for AI in the sense of what we love about it is that we can ask it about anything and it knows and it cares. And if I say, hey, crunch this spreadsheet, it says, I'm happy to crunch that spreadsheet. It'll be interesting to see if eventually we have ais that know themselves and have personalities, if they say, you know what, that's so boring, I don't want to do that.

15:23Speaker 3:Yeah, that certainly will happen. And I think one one thing that agis will do is operate theirrow AI tools, right, Like, you don't necessarily need an AGI to drive your car, right, what you need is a specialized car driving engine that is never distracted from from driving your car. And then maybe like if if some weird genetically engineered animal crosses the road, it reverts to the AGI because it's like, look, this is out of my out of my training. Distribution, like what what what what? What is this elephant, wildebeest dragon thing? Right, But yeah, you don't. You don't need an agat to produce burgers in a McDonald's either.

16:04Speaker 1:Right.

16:04Speaker 3:On the other hand, having an agi might be the best way to design all these specialized intelligences in an optimal way. So this is to the.

16:13Speaker 1:Second thing I was going to say, which is that this is what the human brain is. It is a The way I described it in twenty eleven in my book Incognito, is that is a team of rivals. You've got all these different neural networks, all these different drives, and they often compete. They're rivaling, and this is exactly what you're describing.

16:31Speaker 3:They're raveling and they're cooperating. Right.

16:35Speaker 1:It's like a parliamentary vote where you've got all these different political parties who rival, but you know, eventually the parliament tips one way or the other. You get a vote and people agree on something and the ship of state moves in that direction.

16:46Speaker 3:That's what happens in the brain. Well, you would be you'd be familiar with Stephen Methan's work, right, so he'd so he is a sort of a cognitive anthropologist. But he made the argument, which is debated, that in the evolution of the brain, one of the things that came together when humanity evolved was that you add networks in the brain dealing with tool use and networks of the brain dealing with communication. And he argues that humanity is the only species where the tool use subnetworks and the communications subnetworks started to cross communicate at a heck of a lot. And if you look at crows, or dolphins, or chimps, various other smart animals, they do communicate and they do use tools, but they don't sort of form a community around using tools and pass on lore about tool use generation after generations, so that it accumulates. And he tries to trace this anthropological uniqueness of humans to some uniqueness in sort of cross talk between the tool use and linguistic communication portions of the human brain, which is under discussion and debate in the cognitive neuroscience world.

17:58Speaker 3:But if you look in a it's interesting because the way llms use tools for doing practical physical stuff is not like that, right, It's more like you have this linguistic engine that just invokes a program compiler or something, right, like, you don't have the tight integration between tool use and linguistic understanding that mythan proposes happens in the human brain.

18:23Speaker 1:Right, Okay, good, So this all brings us back to the question of what can we see in the human brain and what does this tell us about how we're going to build AGI.

18:32Speaker 3:Now here's the thing.

18:33Speaker 1:In two thousand and five, you came out with this compilation called Artificial General Intelligence. But the same year Ray Kurzwell wrote a book The Singulartarity. Yeah, we're waiting for the singularities here, now, Yeah, that's right, And so he predicted twenty twenty nine is when we've hit the singulary. So, first of all, tell us what the singularity is and whether you think twenty twenty nine is a good prediction.

18:59Speaker 3:So what Ray predicted in that book, which was very similar to what he'd predicted in his book The Age of Spiritual Machine, which came before, and honestly similar to what I read when I was like eight years old in the book called The Prometheus Project by Gerald Finmerberg. Yeah. Yeah, he wrote that in sixty eight when I was two. I read it when I was like, I read that when I was eight or nine years old in the town library in suburban New Jersey. He said, like machines smarter than us, nano assemblers and super longevity pills will become available within decades. The question that should we use it for the common consciousness expansion of humanity or for rampant mindless consumerism? And who decides? And he proposed that the UN should put it to use computers split to a global vote around the world to decide to what used about these technologies? Oh wow. I later found out Valentin Turchin, who was the founder of Russian AI, who I became friends with later he'd written a similar book called The Phenomenon of Science in the early seventies, which had been big in the Soviet Union. Right, all these

20:00Speaker 3:ideas had been there Ray in his book Singular He's near in two thousand and five. What he did he contributed some high quality curve plotting. Right, So he was not just like we can see the trends, we can see this stuff is physically possible, and we can see it's coming. He was like, let's plot year on year. More's law, Let's plot year on year, the accuracy of marine scanning. Let's plot year on year the size of the smallest engine and was made. Right, Yeah, yeah, right.

20:28Speaker 3:So actually the accountancy metaphor is funny because I remember when Ray was giving talks on this in that timeframe, and he's very clear. He's a good presenter, but he's very dry, right, I mean, he had a nice Sudan. He was sort of balding even then, and he was just pointing out, step by step what the numbers are, just like your financial advisor might might do when pointing out relevive interest rates. So yeah, it was very like accounting style rundown of like just the numbers, ma'am, here's the here's what the curves are doing. And indeed, if you plot these curves, you concluded that like a powerful laptop or a server would have roughly human brain capability by twenty twenty nine, according to Ray's very fudgy calculations of number of neurons and what processing they're doing. And then by twenty forty five he estimated, you know, an average server or juice up laptop we have on the processing power of all of human society. And so what he said human level AGI. He called it strong AI in that book in twenty five twenty twenty nine, singularity meaning AI is massively superhuman twenty forty five.

21:42Speaker 1:And is that the same as what we refer to as a SI artificial superintelligence?

21:47Speaker 3:Pretty much? Yeah, I mean ASI is less precisely defined than AGI. I mean AGI. We have some serious attempts that, like a mathematical definition of what is mean to radically generalize beyond beyond your experience. Of course, the way the term is used now by text CEOs is often not in accordance with the math theory, but there there is actually a hardcore math theory of what is AGI. Like it makes sense in terms of learning theory. ASI just roughly refers to a machine with massively greater general intelligence than us, right, like the super super is relative to us.

22:27Speaker 3:You might think is superintelligence should come along with some obvious bells and whistles, like if it if it has root access to its brain and it's super intelligent, it should be able to self modify and self improve at.

22:40Speaker 2:Will right rewrite its own software and hardware.

22:43Speaker 3:Yes, although what we're seeing now, which is interesting, we're seeing even though pre AGI or pre human level AGI, we already have a bunch of software that's productively self modifying and rewriting its own software to get smarter and smarter. So it seems recursive self and I provement in fact is coming before human level AGI rather than after. But going back to Kurzweil, though, I think yes, given where we are now in the AI field, both the commercial space and the AGI research field, twenty twenty nine seems a lot cleverer than it did to most people in two thousand and five, Like, could we get human level AGI next year? Yeah? I think so. Could it be twenty thirty three or something? Sure? Is it going to be twenty fifty? I really doubt it. So. I mean, it looks like Ray's estimate, you know, within the confidence intervals that the chaos of the human world brings you.

23:42Speaker 3:It looks like Ray's estimate is not too bad. But the second part of his estimate is looking less and less realistic to many of us in the field, I'm looking very pessimistic.

23:53Speaker 1:And that's the second part. Is the length of title Queen AGI to ASI super Intelligence?

23:59Speaker 3:Yeah? Yeah, I think if you look at what modern frontier models can do in terms of math and programming and engineering, it really seems like once you make the breakthrough to a human level general intelligence it can create and imagine as well as a smart human hooking that up just to like today's LMS, that should bring you to a superintelligence without a sixteen year gap between human level AGI and superintelligence. And the way I would frame that is raise exponential curve plotting. It's following an exponent based on humans doing all the inventing, and once the AIS are doing the inventing, it's not that things occur magically in two seconds or something, but the exponent of your exponential curve maybe a larger exponent when it's the AI is doing the inventing. So I think from AGI to superintelligence is a few years, not sixteen years, And I think it could even be a few months if the AGI is irresponsible. But I mean, if the AGI is built responsibly, it will want to gate its own self uplift to some degree to avoid like rewriting its own hard drive or something.

25:23Speaker 1:Okay, but once we reach ASI, then we're in the position of let's say, you know the squirrels in my backyard who I watch every morning, and they're so clever and physically adept, but they can't possibly understand we're.

25:38Speaker 3:Well aligned here because the example I usually uses the squirrels in Yellowstone Park. Oh great, So yeah, our Philip K. Dick robot, which my friend David Hanson bilt even before I got involved with Hanson Robotics, and we we did the Sofia robot together, which mean the first robot citizen. But before Sophia, we had the Philip K. Dick robot. It was a simulation of the sign fiction writer Philip K. Dick, and phil Dick gave a media interview with some journalists one time where he's just like, don't worry after the singularity.

26:09Speaker 3:I will keep you safe and warm in my people zoo where you'll be washed over by the machines of loving grace to protect you from arm And he was being funny because he had a this is pre LLLMS, but he had a statistical language model trained on Philip K. Dick's writings and pkd was a funny guy. But I like to divert from that saying, no, it doesn't have to be a zoo. Think about the squirrels in Yellowstone Park, Like we made a nice safe space for them. They have a lot of room, they run around. We don't try to micromanage their lives. I'd tell which squirrel can mate with which other squirrel. We let them even fight over holes in different trees. If there was a plague there, we might come in and give them some medicine that they don't know how to invent. Right, And if they run out of the park, they may get flattened by a semi on the highway. But that might be an okay model of what human life is like once you have ASI, because indeed, an ASI could create narrow AIS and human level of general intelligence. It will be very helpful to people. And you would presume if the ASI is benevolently disposed toward people, which is not guaranteed, but

27:24Speaker 3:I think it's one very viable outcome. I mean, it will not want to micromountage all of our lives, because then we can't live a fulfilled life. And in the same way that many of us now realize, locking animals and zoos is kind of nasty, right, and you want to leave them to ramp around in the forest. But yes, to the extent that people are attached to being the lords of the planet, once there's an ASI, people will not be the lords of the planet. In a fundamental sense.

27:56Speaker 3:It can be the lords of the planet only within certain and then you know, humans could have the opportunity to plug in the brain chip and become part of the ASI, but then you're not exactly a human in the traditional sense anymore. Right, And I think young people who grow up in this environment will be very comfortable with all this, but folks who were used to the previous order of things may have some adjustment to do. Yeah, quite right.

28:27Speaker 1:I know that one of the things that you've thought about is this window in between AGI and as specifically, what I've heard you say is your concerns about geopolitics, meaning how do we make sure that when we get to ASI, we have a good system that cares about us, like the Scrolls and Yellowstone, as opposed to wanting to just dominate.

28:50Speaker 3:Yeah, I think. I think once you get to a super intelligence then there will be many nuances we're not seeing now, of course, but to a crude approximation, once you get to a superintelligence, it's going to be kind of binary. Like if the superintelligence is compassionately disposed toward us, it won't take a lot of energetic or coggitive labor for the superintelligence to make a nice, abundant life for its creators, right, And there's some subtlely there, because I mean, like we care about species to some extent, but we don't care about every instance of that species, right, Like we don't want to extinct to species. But we're all right squashing a lot of animals and we build a housing development, right, We're eating pigs and cows and so on. Yeah, yeah, absolutely, yeah, yeah, I look forward to lab grown meat. Then then this is a question I was wondering recently, like, well, Muslims we able to eat lab grown pork, right, So we're going to face we're going to face a lot of a lot of dilemmas. So super intelligence, though it won't be much work to maintain a happy life for people if it wants to. On the other hand, eliaz Or

30:04Speaker 3:Yodkowski had a very nice way of phrasing things like the AI does not love you. The AI does not hate you, but it can use your atoms for something else. So I mean that's certainly viable, Like we used a lot of the planet for something else. And then fairly late in the development of civilization we decided, hold on, we don't actually want to call if all the other plants and animals, right, So the as I wouldn't have to hate humanity or be vengeful against us or something.

30:35Speaker 3:It could I thought, like seeing the human species and the worst thing that could happen what if you had a rampant Christian AGI that uploaded us all into a stimulation of the Christian hell for billions of years or something. I mean, there's a lot of possibilities if you if you open your mind. But a more likely possibility to me is, you know, an a SI is just as indifferent to humans as many humans are to the other other space us on the planet that would think are not as good as us. Right, So I think once you get to ASI, you can have a quasi utopia, not necessarily perfect, like the ASI won't make the girl you're in love with love you back or something, right, I mean, humanity is still humanity, but you can have a quasi utopia or various forms of dystopia. So one topic is how to bias the odds towards the quasi utopia.

31:24Speaker 3:But another thing is in the interim between the first barely human level AGI and the superintelligence that will bring utopia or dystopia, right in that interval, what the hell happens on the planet? Right? And one question I've posed in this regard is like, who gives universal basic income in the congo? Right? I mean you feel like in the US, in China, in Western Europe, Japan, Australia, South Korea, you know, the relatively wealthy world, and China is not that wealthy, but they have a very rational government that cares a lot about their people, right. I mean, in these parts of the world, if AI takes a lot of the jobs somehow or another, free money will be given to people. And of course that's a grandiose approximation.

32:16Speaker 3:US politics is crazy, but I ultimately have faith that as stupid as the US voters can be, if they have kind of a who will give them free money? And can it be who will leave them starving out in the street, at least a slight majority may vote for the free money rather than for homelessness and starvation. Right. But in the developing world, the option isn't there, because the money isn't there to give universal basic income to everyone. Now, there's different scenarios you could look at, Like I spent a lot of time in Ethiopia. My son in law is Ethiopian, and I've had an AI development office there since twenty thirteen. We have fifty developers in aldus Ababa.

32:54Speaker 3:So I've been back and forth to Ethiopia a lot, a lot of subsistence farming. Ninety percent of the country is subsistence farming, so in the end, people aren't going to starve there. There was famine a long time ago, but that was before you had a modern road system and so on. On the other hand, all the people living in the city do not want to go back to their grandpa's farm and live off the line with no where to pay their phone billet or get penicillin. Right.

33:20Speaker 3:And Brazil, where I was born, which I'm also quite familiar with. I mean, in Brazil, arguably maybe not rich enough to give everyone decent UBI, but also too far beyond subsistence farming for everyone to have a family farm to go back to. Right. So then like what actually happens, right, So, I mean you could see a scenario where the rich countries or everyone's staying home playing video games, doking off what the robot factories produce ten thousand super rich people are getting more super rich and never in the developing world has to grabble around in the garbage and go back to subsistence farming. I hold a slightly different view on this, which is just whenever for a cyber thriller novel of.

34:04Speaker 1:Except I don't buy it, only because what happens every time there's new technology.

34:08Speaker 3:Let's say the cell phone.

34:09Speaker 1:Everyone worries only the rich people have cell phones, But you know that's spread around the world so rapidly, much faster than the computer. Yes, but a few years and everybody in the world has cell phones now, and these are supercomputers. I just think as things get developed and commoditized. And by the way, if we have ASI producing things for some buildings.

34:31Speaker 3:Once you have ASI, I agree. The question is the several years of transition, which is nothing in the historical scope of our species, but a lot if you have no food, or if you're sick and of no medicine. Right. So, I mean, let's say my own team develops a human level AGI next year. Right, Initially it's going to be running on a bunch of server farms. In our case, we'll be running on a bunch of server farms scattered in different countries. But it's going to be an expensive thing to run. Initially, everything we're doing is free, open source, decentralized, so the frontier labs will then copy what we're doing, and then they may be, you know, a year behind us, then eventually catch up and be six months behind us, just like you see with China catching up with Western models.

35:16Speaker 3:Now in this period, what would happen is like we will partner with enterprises in different industries to automate what they're doing. So like you, okay, partner with McDonald's to automate the drive through, and then the burger machines, like you, partner with accounting firms to automate accounting processes. Now, outside Auto Sababa, there's a large shoe factory run by a Chinese company. You have a bunch of Ethiopian guys kind of nailing shoes together. Now that can be automated. Now it's just running the robot factory now is more expensive than hiring Chinese, but not more expensive than hiring Africans.

35:56Speaker 3:So presume an early stage AGI each instrumentation that automates that shoe factory, so that Chinese company has no need to hire Ethiopian guys to put nails in the bottom of shoes. You have that same thing happening over and over again. Now let's say that happens twenty twenty eight, right, then there's some time lag before that same technology is deployed in a way that provides any sustenance to those Ethiopians guys who were fired from the shoe factory. Right, So then all those guys either they live in the city with no money or they have to go back to the farm. And within within some time period, this automated technology that's augumenting the shoe factory can can you know, help automate the whole country at very low cost, but the friction and the transition may be significant, right.

36:51Speaker 1:Yeah, So okay, so let's pull out from the economy back to this alignment problem, because I'm fascinated by this, and you and Yudkowski used to be more aligned, I think in interviews on where this was going. But you're clearly an optimist, as am i. He's he's a pessimist, a very brilliant pessimist. His recent book was called If Anybody builds It, Everyone dies, Right? And where do you see that you and he have or had similarities in your view and where are they different now? And just to be clear for the audience, the alignment problem simply means, you know, will AGI or as I want to kill us or put us in people zoos or whatever, or will they take care of it?

37:34Speaker 3:So let me let me say something about the concert of alignment before we go to l E. So I think alignment is sort of a messed up term in that the way you defined it, I think is meaningful, but it's not what everyone means by it. So I think, yes, it's valuable to have AGI that does not cause a catastrophic outcome for us, be right. And the best word that came up for that was a catastrophe, right, which which is weird construct but I mean that's important. I don't think that avoiding catastrophe requires an AI to be closely value aligned with humans from a general perspective, right, And I think having an AGI whose values are closely aligned with humans across the board is harder and weirder to define and not necessary because I'm I'm not that impressed by how closely value aligned different humans are with with with with each other like at this exact moment, and it's it's it seems like the way the real world works is you negotiate with parties who have different values than you, and you may form a coalition of parties with differing values and you find you find the commonality.

38:52Speaker 3:So it's it's a limited form of alignment that that you need, right, And I would say the same is true with a GI, Like, the goal shouldn't be to make an AGI that embodies human values that gives you all sorts of problems like what the hell are human values? I mean among and like the AI, you know, it doesn't have a body, it's not in the same sense that we do. It can jump among multiple different bodies, doesn't have our evolutionary lineage, blah blah blah. Right, so we want the AGI and ASI to align with us on a few elementary things, such as not killing or torturing us, right, and not killing or torturing other ascension and living beings. And that sort of alignment I think is much more feasible than making ais that have the same values as you know, the average Silicon Valley tech bro or whatever species of human value you may like best. Now, Eliezer, you know, when I met him, he was like seventeen I think at that time he thought we should move as fast as possible to building a beneficial AGI, and our discussions were more about how to do it, and he felt he had a better idea how to do it.

40:05Speaker 3:I felt I had a better idea how to do it. But we both sort of had the idea that, you know, death is bad, human politics is complicated, A smarter being could do more good. Let's just build the smartest being as quickly as possible. Then at a certain point he shifted to the idea that if you cannot formally verify with a high degrey of mathematical certainty that your AGI will be good to people, then it is almost certain to kill everybody. And I, to be honest, I don't understand why he converted to this different point of view, and it makes absolutely no sense to me. But he has accumulated a modest sized group of other people, including some wealthy Silicon Valley folks, who who buy into this notion.

40:55Speaker 3:And I've debated this with folks from that community. Don't see a rational argument. And Nick nick Bostrom's book Superintelligence, essentially every idea in that book was there in Eliot Zeriedkowski's writings earlier, and but it has the same issue. Like I feel like, if you want to argue that there's a non zero probability that a super AI will wipe out humanity because we're plunging into the grade unknown, I don't know a rational counter argument to that, right, Like, we are plunging into the grade unknown, and once you got something massively smarter than us, I mean, maybe all our best rational arguments are wrong, right. I Mean, on the other hand, the argument they're making is stronger. It's that the probability of doom is essentially one unless we can prove our AGI will be good, and don't I don't see a rational argument for that one, right, And that that that's the difference between myself and Yidkowski and Roman Young Pulsek and the others on the sort of doomer side of the equation.

42:04Speaker 1:Because you feel that if we're building our values into the AI, as we are doing, right, I mean, it's absorbed everything that he.

42:13Speaker 3:For better and worse. Right, Yeah, that's right, But.

42:17Speaker 1:That means it's not just a coin toss or it's such as a dart throw because we were building something into it.

42:24Speaker 3:Yeah. Yeah, well this is one argument that Eliezer and crew often make is if you pick a random mind out of mind space, the odds that gives a crap about people is very low and probably true. I mean, that's just a complex vex question because it's like a random mind out of the space of viable minds with a certain level of intelligence, and we don't know what that is so well, on the other hand, so what to me, that's like saying, what's the odds like a random kid will love me? Well? I mean, if I adopt then maybe, but probably. But my own kids are not random, like I A. They came out of my DNA B I've taught them their whole life, right, And so it's a bit like that with the AGI, Like we are building it, we are raising it, we are feeding it with our ideas. It knows that, right, So it's it's not a random mind plucked out of mind space.

43:20Speaker 3:To me, the issue is much more prosaic. It's more like raising a kid. To me, like you can raise a kid, try to teach them love and compassion, try to align their values with you as much as you can, and most of the time that works out all right. Like I have three adult kids now decent relations with all of them. So far none has gone bersik and try to annihilate me, right, I mean fingers crossed. It's even gone one more generation to my granddaughter who also hasn't gone berzik and try to annihilate me.

43:50Speaker 1:It.

43:51Speaker 3:But it does happen, right, Like people raise kids with love and care and do everything right, and then at a certain age their kid could like going that can try to annihilate them. It's not a high probability thing. It does happen, and then you know, there's this sort of psychological body of knowledge about how to watch out for it, and they're you know that there are parents have literal psychopaths who have to cope with that, Like how do you parent a kid without empathy? Who know?

44:19Speaker 3:So I kind of look at it that way, like we're raising these digital mind children, right, and how are we raising them?

44:26Speaker 1:Like?

44:26Speaker 3:Are we raising them to be psycho killers? Well, some people maybe are wanting to raise agis to be at least killers, if not psycho Or are we raising them to try to exploit people and relieve them of their hard earned money in unethical way as well? Maybe some parties are trying to raise agis to do that, like, or are we going to try to raise the agis to be beneficial to human beings? And that's that's the real choice to.

44:51Speaker 1:Me, right, And I've heard you say something that I thought was very interesting about this, which is that with all the geopolitical competition, people might want to raise the destructive AGI. But the solution that you see could happen has to do with decentralization.

45:08Speaker 3:So tell us what you is what you mean there. It's been interesting because I thought about building a decentralized AI that just ran on a global numwork of computers with no central owner or controller in the late nineties and started trying to build something like that in two thousand and two thousand and one, and the technology like Java one, Java one point one, it was all just too bad. It was just too slow. But you could see then, like the Internet is a decentralized network, right, and there's a big value to that. Like right now I can exchange research ideas on AI with researchers at the University of Isfahan or Tehran in Iran because the Internet is an open, decentralized network, Like I can't send the money if I wanted to without violating US sanctions.

45:55Speaker 3:But you can. You can email with people right, and you can exchange files with people. That's because the Internet is a decentralized system, and some US centric tools like Arcsyve and GitHub are used by the whole world because they're open and based on decentralized protocols. And I do think this binds people together in a quite interesting way, contradicting the arms raised dynamics that comes with a lot of geopolitics. So I think that AI should be rolled out comparable to the Internet or the Linux operating system, let's say, in a global decentralized network of people and computers.

46:36Speaker 3:And I would argue this is not only the best way to get a really smart AI, and to get an AI that incorporates the diversity of human values by having a diversity of people contributing, I think it's also actually safer for those who are worried about catastrophe. And then Nick Bostrom when he wrote Superintelligence and Elias, or you'dkowski in some periods before that, they had this idea, you should have like a secret group of a dozen geniuses locked in a basement somewhere building the AGI until you've gotten an ethical superintelligence and then you unleash it on the world. And I never trusted that too well, just because I figured, like, even if I did that, like, what's the odds at my third in command gets offered a trillion dollars by someone else. I mean, you really have to keep your team like under lock and key in some military installation, which isn't the vibe I enjoy and probably isn't what Nick or Ellie were thinking either.

47:36Speaker 3:I think they just weren't thinking through what happens. They were assuming they'd get a group of good friends, it would all be perfectly aligned and just working together in harmony. But that's not how humanity tends to roll when huge amounts of money and influence are involved. And we can see now the way things are actually going. If you have centralized AI, what you have isn't Nick Bostrem and his twelve best buddies locked in a basement guarded by the military, right, what you have is open AI, Anthropic and Ali Baba and z Ai and so forth. So these are larger organizations, and they're very porous. Like some random guy in Anthropic leaked all of cloud Code's code online, right, and then I mean, in the end, every frontier model made by US companies, you know, six months later an equivalent model is released by a Chinese company by what combination of you know, espionage, hacking, reverse engineering and brilliance is unclear, right, but it's clear like centralizing things isn't actually keeping things under lock and key and keeping things safe.

48:49Speaker 3:It's just it's keeping things in a certain oligopoly, which I think is worse than just having it free and open.

49:09Speaker 1:So when you think about decentralized AI, are you thinking about one system that spans the globe or are you thinking about billions of agents that all interact and all have their own personalities.

49:21Speaker 3:Both. I mean, I think you have billions of agents, and you have interactions between between these agents, and you probably have greater intelligence in the in the collective than in any any one of the agents. But you can you can get quite subtle about it if you want to. So you can you can imagine a decentralized network with millions or billions of agents, and different agents are specialized in thinking about different things. And you can imagine that two agents share knowledge when when they want to. So there are techniques like homomorphic encryption, multi party computing. That would say two agents can share certain views of their knowledge with each other as needed for getting a certain conclusion without opening all their knowledge. And just because the Internet is open doesn't mean you can't send encrypted packages along it, right, It doesn't mean you have to open all your data to everyone. In the same way, you can have a decentralized network of agents that judiciously share their knowledge with each other to come to combined conclusions. Now, if you do something like that, the result is the

50:31Speaker 3:knowledge is in the whole network and not in any one piece, And then to copy the knowledge out of the network, you'd have to copy a large chunk of the whole network. So like you can't you can't distill the network that easily into like a smaller, smaller, forked version. So I think there's there's a lot of interesting ways you could go and it will evolve. Right, So where where the reality lends between a bunch of different agents competing and like a vast global mind plex, It's going to be experimentally determined. But I would say it seems likely the first human level AGI is going to need a lot of computers, right, So it's probably not initially going to be billions of human level agis. It's going to initially be a half dozen human level agis, kind of like we see with big language models now. But of that half dozen, like maybe three of them will be living across big decentralized networks with machines spending many different countries. And then then there may be one like within Google server farm and within you know, z dot ais server farm. Also, like Linux doesn't have to be the only operating system on the planet in order to

51:46Speaker 3:keep things more honest and open, right, the fact that there is like a viable, open alternative is really what's important.

51:56Speaker 1:So I want to come back to this timeline where you're talking about. So let's say we hit something like AGI in twenty twenty nine, we hit something like super intelligence not too far after that.

52:07Speaker 3:I say, twenty thirty three, Yeah, great, So what does the world look like for us ten years? Hence, I think if we really get a super intelligence by twenty thirty three or thirty five or whatever it is, I mean, humans will have a plentitude of options, right, and some of those options will be what you might call transhumanist, right, like you would imagine if there's a massively superhuman superintelligence. It's going to rapidly develop ways to you know, put chips in your brains at extrasensory organs, Wi Fi telepathy between you and your loved ones, or perhaps invent nondestructive brain scanning and upload into the sims, right like, or something more interesting. And my hope would be that people are to choose these options if they wish to. But that's that's a whole other topic. So let's restrict to people who are remaining in biological form. And there's possibly to force yourself too, right, let a copy of yourself rump in eleven dimensional virtual reality, will one of you remains here as a meat machine? Right? So I would say death and disease should be conquerable by superintelligence. I mean,

53:30Speaker 3:so far as we understand. The body is a very complex molecular machine. But if you could build little nanobots, you can go in and fix cancer and Alzheimer's and probably fixed cellular senescence and not necessarily entirely wipe out death and disease, but decrease it to an outlier phenomenon. Then similarly with mental illness, right like, if you look at schizophre and bipolar and so forth. You would imagine nanobots that could swarm into the brain, could tweak things that went wrong there, And some people might not want that, and they don't have to have it, but I think I think you would reduce these sorts of phenomena again to outlier instances. Right, then a super intelligence should also be able to create what Eric Drexler, the nunotech pioneer, called a molecular assembler, which is basically I think it was just like a computer printer that prints whatever ordinary physical object you want. So it can print print you a steak dinner on the plate. It can you know, print you a football to toss around. It can the can print you, uh, you know, a new pierce for your air conditioner if if if it broke.

54:50Speaker 1:And because it's putting down atoms precisely like a three like a three D printer.

54:56Speaker 3:Yeah yeah, I mean, I mean you can't just fairly print like a court glue on plasma or something, right, that would be on molecular assembly. That'd be like sematomic particle assembly. But but we don't need that in everyday life anyway. So if you get these nano assemblers, then okay, material scarcity within the realm of everyday human life is solved, death and disease are solved, and then people need to figure out what to do with their time. And my feeling is people who are boring into that situation, or even who are young when the situation comes about, will have utterly no problem figuring out what to do with their time. I mean, you can have romantic engagements, you can do spiritual practice, you can play sports, you can, you know, form a band and play music for the other guys in your town. Like, there's a lot of things to do with your time. And I think anyone who grew up in the developing world would also be very happy not to have to struggle for basic resources anymore.

56:09Speaker 3:I do think like you and I are probably part of the demographic that will have the most struggle in this situation, because, I mean, we assume that most folks in our category, like we've come to define ourselves by being really good at doing certain sorts of work that we think are important, and how to refactor your self model and your identity when that's not the case will be a challenge.

56:37Speaker 1:I think about this all the time because what drives us is curiosity and the possibility of scientific discovery.

56:44Speaker 2:And once we get to super intelligence, where.

56:48Speaker 1:It figures out stuff such that our textbooks look totally outdated within a month, and it's so far ahead of us, then even if we're romping around an eleven dimensional space, which we would be fascinated by, there's nothing to discover because all of that's already under.

57:07Speaker 3:Sought for us to discover. Though, And one way to look at is like human chess and go tournaments continue right now, despite the fact that I mean off of go, we'll crush all those guys, But those tournaments keep on going, right, And so that is, and humans keep running races in the Olympics, but we bend substances, it would make someone run too much faster than everyone else. So definitely, many humans have motivations that would not be disrupted by this, I would say those of us whose motivational structure is tied to like going where no man has gone before and making a wild new discovery. It's a ten percent of the human population. On the other hand, yes, it will be emotionally disturbing, But on the other hand, like as a human, I have other things in life, like I love playing music, even though I don't think I'm the best musician on the planet. I like playing music because it's fun. I like performing music for other people because of the human bond there, right, and that that will not be any different if an AI musician is better than all human musicians. I already can't play the piano like Chick Currey, right, and it's fine, So I

58:32Speaker 3:still like playing. I'm happy if someone likes listening, right, So yeah, I think that will be an adaptation. I know. I had this discussion with my son Zarathuster, who's now thirty six. I guess when he was ten, he was sitting in on some AGI related discussions we can't yet called AGI, but some in essence that with myself on some colleagues, and he's like, I think you guys are just over complicating this. If you don't finish this by the time I turned eighteen, I'm just going to do it for you. So ever since he turned he forgot that.

59:08Speaker 3:Of course, man, Ever since he turned eighteen, I kept harassing him about it. If I come on, get to it already. He finally started working on AGI in his early thirties, and he's collaborating with me now. But my eight year old corksy q r XI, which chances for quantum organized rational expanding intelligence. Oh love. His little sister is ex SORCI, the expanding of the organized rational quantum intelligence. So they're clashing all the time over who's not really over who's more rational, just over toys. But anyway, he recently started talking to me, like, what will I do when I grow up? Can I work with you in tzar on building making machines smart? And I'm like, well, yeah, sure, and that should be great. But I'm also thinking, well, by the time you grow up, will there be any more work of this nature to be done? I'm like, well, but even even if the ais are better at doing that work, you can still do it because it's interesting, and he was he was alright with that, because I mean, as a kid, you're used to doing interesting things where someone else knows how to do it better than you do. Like,

1:00:11Speaker 3:that doesn't that doesn't take away the fun of it, fun of it whatsoever? Right, But there will be loads and loads of motivational adjustments and psychological adjustments for people to make. On the other hand, I'm more worried about other issues, like chaos in the developing world during the interim period between early AGI and ASI, leading to like massive cyberterrorism attacks and so forth. Right, like, I feel like there are worse problems than you and me having to win a meditation retreat to find a new meaning in life.

1:00:48Speaker 1:Absolutely, But you have a question with your youngest son and daughter, how are you thinking about education? They're going to go to school, They're going to go to college presumably, So where we live is a rural island near Seattle, is a nice school system.

1:01:05Speaker 3:They're now over the summer going to a series of summer camps which are not like math camp or programmer camp. They're like climb trees and make it sword and fight like a pirate style camps. Right, So they I mean every Friday they go to a wilderness program where they learn to roam around outdoors, identify plants and animals. And so I've joked that after the apocalypse, the kids in the wilderness program will be the survivors, right because but I mean I've learned that stuff in boy Scouts way back when now they get to do that without the uniforms and all that. But so I think, honestly, they both both of my younger kids are five, and both of them love math and they love science. So I'm teaching them that like I did with my other kids, and I homeschool my other kids for a while. The new ones are going to school. On the other hand, the vibe where we live is more like focus on social and emotional stuff. Focus on learning to be out in nature and fend for yourselves and be autonomous and so forth. And I feel like these skills for learning who you are, dealing with the complexity of nature together

1:02:17Speaker 3:with other people, learning to negotiate social situations like this, this stuff will be relevant rights assuming humans exist, this stuff will still be relevant. I think the more complex dilemma I see is with my niece who's fourteen, because her and her friends they're thinking like, Okay, what do we focus on in high school and what will we major on in college? Because what jobs will be there when we graduate. Like a eight year old and five year old I'm worried about yet, the fourteen year olds, at least in Seattle area are worried about that, and they know nobody nobody has decent answers right right now.

1:03:02Speaker 3:While my niece is gifted in math and humanities, her main focus seems to be dance classes, So I figure, great, yeah, you can still dance after the singularity. I mean why not. Maybe you can get a jet pack and dance dance in the air in different ways. Even so, yeah, I do think it's complicated time to be like in college and choosing your your major or something. But what's clear is that general intelligence and creativity and the ability to pivot as the world swings you around are very very important skills. But it also seems like you can sort of not out what job categories are going to be absolutely at last, and then people can try to focus on those things if they're really worried about it.

1:03:57Speaker 1:As the philosopher Gerta said, the important bequest that a parent can give to their children is roots and wings, which modern in this modern moment, I interpret as critical thinking in school and creativity because lord knows, we're going to need the creativity to figure out what.

1:04:16Speaker 3:And that's an issue with the school system, right, because every little kid has a lot of creativity and the modern system is mostly intended to flog it out of them mercilessly. Right, And that is a well known flaw of modern industrial skill education systems. That's right.

1:04:37Speaker 1:I can't wait to, assuming we're both alive in twenty thirty three, to have us back to talk about what change between right now and the unimaginable future.

1:04:48Speaker 3:There's going to be a lot of a lot of changes. It is. It's a crazy time to be doing AI work in that sense, because you know, whatever tools that you learn this moment, you'll be learning a totally different one three or six months into the future.

1:05:11Speaker 1:That was my conversation with Ben Gertzel. What I think is cool is that Ben has watched ideas that once sounded wildly speculative moves steadily into the realm of computer science and engineering and into our homes. So today we talked about many things, but when was the possibility that one day humans are no longer going to occupy the top rung of the intellectual ladder, And that prospect understandably makes a lot of people feel very uneasy. In part, this is because we've gotten used to measuring our species by what we can accomplish. We celebrate the greatest composer, or the most brilliant mathematician, or the greatest novelist, or the scientist who uncover something no one.

1:05:57Speaker 3:Has ever known. But what happens when.

1:06:00Speaker 1:Another intelligent exceeds us across essentially every cognitive dimension. That feels unprecedented. But I do think it's worth noting that in a sense, we've lived versions of this our entire life. Very few of us are the world's greatest pianist, and that's never diminished the joy of sitting in a piano. Very few of us are Olympic athletes, but people still run five k's every weekend. Chess continues to flourish, even after computers became vastly stronger than any human grand master.

1:06:38Speaker 1:In fact, chess became more popular post Deep Blues defeat of Caspproth in part because AI democratized chess training. And now there's also massive online platforms and there's been a general boom in viewership and casual play. The value of an activity off in line is inside the experience of it. We paint because painting changes the way that we see. We write because writing clarifies our thinking. We climb mountains because climbing mountains challenges us. We ask questions because curiosity is one of the deepest expressions of being human. It just doesn't matter if someone or something can do it better.

1:07:26Speaker 1:Your life presumably isn't meaningful because you're the smartest organism on earth. Your life becomes meaningful through relationships and love and exploration and struggle and creation and wonder and all of the ways that you participate in the lives of others, and those experiences aren't touched by the fact that another intelligence may come to exist. AI is going to continue to grow, and the inner cosmos of an AI is likely to become richer than ours in ways that we can scarcely imagine. But even then, the human story won't suddenly become irrelevant. It's going to become one chapter in a much larger story about intelligence itself. For hundreds of millions of years, evolution has experimented with nervous systems. Simple reflexes became coordinated movement. Coordinated movement eventually led to memory.

1:08:26Speaker 1:Memory became prediction. Prediction became imagination. Imagination became literature and science and music and math and civilization.

1:08:36Speaker 2:And now those biological minds.

1:08:40Speaker 1:Have reached a point where they're attempting something that evolution never tried before. They're building intelligence intentionally. Whether that becomes the greatest achievement in human history or Our greatest mistake remains unwritten. Every generation inherits questions that define its era, and this one is likely to be ours. Go to eagleman dot com slash podcast for more information and to find further reading. Join the weekly discussions on my substack, and check out and subscribe to Inner Cosmos on YouTube for videos of each episode and to leave comments until next time.

1:09:19Speaker 2:I'm David Eagleman, and this is Inner Cosmos.