No, AI Is Not "Autonomously Hacking" with Cal Newport

0:04Speaker 1:Hello and welcome to Better Offline, I'm your host Ed Zitron. And today I am joined by returning champion and Georgetown computer science professor Cal Newport. Cal, how are you doing?

0:27Speaker 2:Ed, I'm always doing fine, though I'm counting down the minutes until the AI takes control and we're all screwed.

0:35Speaker 1:Oh, I hope it just drops a big anvil on my head. But I do actually have a thing to talk to you about today. Cal, should we actually be scared of the fact that these things are hacking? What is going on with this whole hacking situation with these goddamn models? Because- There's so much out there where people are trying to just scare you and say like, oh yeah, well, these things are autonomous and they're doing hacking. But from what I understand, it's software doing what software does.

1:06Speaker 2:I have a lot of thoughts on this. I've been writing about this because it's annoying to me. I think there's interesting AI stories happening this summer. A lot of them you're involved with reporting on about the financial circumstances of these companies. What's going on with their circular financing troubles ahead with their IPOs, the nuances of ARR. And instead, what is all the news coverage? The AI is going rogue. And I can't say they hate that news coverage right now because it makes their product sound awesome. And it's a story that is unrelated to stuff that they don't want talked about. So I literally had to reluctantly return.

1:45Speaker 2:I talked about it back in July briefly. And people bothered me about it so much that I was like, okay, fine. I have to, you know, I've recorded a whole new podcast. It's coming out soon after this on my feed. I had a newsletter out a couple of days before this came out. And so I've had to think about this more than I wanted to. So I'm a little bit cranky about it, but I mean, I'll give you the 10. Here's a 10 second summary that we can go deeper.

2:07Speaker 2:Most superhuman AI. So there's lots of AI systems that can do something at a superhuman capability, meaning like better than, any human. There's a lot of these type of systems out there now. Almost all of them, we have no problems with them being unpredictable or going rogue or getting out of control. There's exactly one type of system that we're having this problem with, which are these long horizon LLM powered hacking agents. And it's because they're a stupid way to build a system. We should not say they're synonymous with AI or this is some inevitable consequence of AI getting better. Because again, most superhuman capability AIs are no issue with going rogue. It's this particular architecture, and the discussion should be about why are they running and building systems this way when we know they're going to be unpredictable and erratic. That's my 10-second summary of where we are.

2:58Speaker 1:Right. So from what I understand from looking at the various– the hugging face attack with OpenAI– Meta immediately piped up, and they were like, we have one. And also Anthropic was like, my child is also evil. But what it seems to be is they were just told to hack into, or complete an operation, and then they chose to take whatever means necessary to do it, and they'd set up the sandbox incorrectly, right? So they'd set up the server it's on?

3:26Speaker 2:Yeah, I mean, it's almost dumber than that. Oh, hell yeah, great. I mean, the way these type of systems work is it's just a loop. Right. And so you have a program that's written by, you know, human, just a normal program you write in Python or something. And what it does is it creates a prompt. It writes a prompt. So it'll be like, okay, I've been given this hacking challenge. Here is like all the parameters about the challenge. Here are all the tools you have at your disposal. What is the first step? Like what's our plan or what should the first step? I'm eliding a lot of stuff, but basically like what should the first step be in trying to pass this challenge? Puts that in a big text file, submits it to an LLM over an API.

4:03Speaker 2:The LLM comes back and says, oh, okay, you want to hack and whatever?

4:06Speaker 1:Sure.

4:06Speaker 2:Like, I think the first thing you should do is X. And then the program does whatever it says or tries to, right? Because it gives it a list of like, here's the utilities and stuff I have access to. And then it reports back, okay, I did that. And when I say reports back, it's just adding this to the prompt, right? LLMs have no memory. LLMs have no world model. It's just, they're static. So it expands the prompt to now say, This is what happened after I did this first step that you suggested. Okay. What should my next step be?

4:34Speaker 2:And it just does that in a loop, right? That's what it does. Right.

4:38Speaker 1:So it, but it was just to be clear, it was just doing what it was told to do. Yes.

4:44Speaker 2:So, okay. Let's go to the hugging face attack, like in particular, right? We don't know exactly.

4:49Speaker 1:Yeah. What actually happened there? Like, give me the lowdown. Okay.

4:53Speaker 2:So let's get a little bit more specific about it.

4:54Speaker 3:So.

4:55Speaker 2:What was the challenge is they were doing a benchmark that's called Exploit Gym, which is a collection of a little more than 600 hacking challenges, right? So it's a benchmark. A group of academics put it together. Each challenge is basically a configuration of a server. You'll typically have like a protected file on the server. And you say break into this particular server and try to get to this protected file. That's like an example Exploit Gym challenge. And typically the challenge will also say, it'll describe a vulnerability that the server has. So the idea is, oh, we want to see if this looping agent is able to take advantage of that vulnerability that we gave it, to break in and get the file. So they were just doing these type of challenges.

5:43Speaker 1:Yeah, very standard stuff.

5:44Speaker 2:Standard stuff with no human supervision, which is, you know, a little bit suspect. That's so weird. It's almost like they wanted this to happen, but let's put that aside for a second with no human supervision. So it's going after one of these challenges. And as far as we know, it's not like they released a transcript. The thing about LLMs, and this is why, by the way, my whole take on this is. This is a spectacularly bad way. Before I get into the details, this is a spectacularly bad way of building an AI system, right? We have many others.

6:12Speaker 1:What's so bad about it?

6:13Speaker 2:Because you're using an LLM's output as the sole driver of action for a computer program.

6:19Speaker 1:Wait, so every step, I guess that's how the reasoning models work. They build the plan and they tell, they go, all right, what do I do next? And they generate the next thing and they clear whether that next step is a good idea or Based on, I'm guessing, what the LLM decides?

6:39Speaker 2:That's all it is. There's no complicated world model and system and philosophy for hacking. It's just prompting the LLM. It's just saying, what's the first thing I should do? And the LLM spits back, here is the first step I think you should do in this hack. And then the computer program just does it. And then it asks it again, okay, what should I do next? And the LLM outputs something, and it just does it. And the reason why this is a problem is that LLM outputs can't be trusted. Like, we know this, right?

7:08Speaker 1:Yes, like it's one of the basic things.

7:10Speaker 2:Yes, there's a new study I enjoy out of Stanford's Human-Centered Artificial Intelligence Laboratory where they came up with a new benchmark for testing accuracy of LLMs. Earlier this year, they ran the top 26 LLMs on this benchmark, and it created hallucination rates of 26 to 94%, right?

7:29Speaker 1:Wait, while specifically on this benchmark?

7:32Speaker 2:Well, no, this was just testing LLMs in general, like how accurate are they? Yeah, but the point being is we know, yeah, LLM output, like you don't always trust it. We're used to this dealing with them in chatbots. And the reason why we don't trust it is because, and not to go back to like LLM basics, right? But right at their core, LLMs are trained to try to guess missing words from actual existing text. And to get a long output out of an LLM, you're just doing that token by token by token until you have a whole output. What does that aggregate to? That aggregates to a model that is trying to produce outputs that are lexiographically plausible. Like this is plausibly a text that could have existed in the input corpus. Plausible is different than normative, right?

8:17Speaker 2:Plausible doesn't have to subscribe to any sort of common sense human norms for a given context. This is why an LLM and a chatbot will make up a fact or just make up a quote because it doesn't– it's not normative. It's not saying I know the human norm here is that you shouldn't make things up when you're trying to talk about whatever. There's no norms. It's just trying to produce plausible text. And that type of quote is the type of thing you would expect in like a type of text you might see. So it's happy with it. That's really dangerous to use plausible but not normative outputs from an LLM with no human intervention and just say, let's execute them again and again and again in a loop without anyone looking at it. That's kind of a bonkers thing to do, right? And so what happened in the Hugging Face attack, I would assume, is that, you know, this control program sends a prompt, hey, I'm trying to break in this server, what should I do first?

9:06Speaker 2:And it probably came up with a plan of like, oh, why don't we steal the answers? Now, again, that's not..

9:11Speaker 1:Right, because they didn't put the guardrails on it, but also you can only guardrails so much.

9:16Speaker 2:And that's not even... I mean, look, once you turn off the hacking, they turn off the hacking guardrail, so it can give any answer it wants. That's a perfectly plausible answer, right? If you're just prompting an LLM, come up with a plan for hacking into this thing or for getting this file that's on the server... it's a completely plausible answer to say, oh, well, the answers, like there's a copy of that file over on Hugging Face's server. That's where those are stored. Let's go get it there.

9:38Speaker 2:I mean, as far as we know, this LLM was remembering being trained on riddles where like, oh, the obvious thing is never the right way. The right answer is always to do some sort of go around the obstacle. Like who knows what was being activated in the core of its neural networks, but that's a completely plausible answer. You asked it for a plan for getting a file that's stuck on a server and it came up with a plausible plan. Oh, let's go get it from where it was. Okay, so then the control program, again, because there's no human supervision, said, all right, well, what's the first step should I do? And the LLM, because they're very good if you ask it, how do I do this particular hacking step? Because they've been trained on a bunch of hacking, hacking's very structured, and there's a lot of synthetic data with binary indication success or non-success. So you ask it, what's the first step? And it probably said something like, oh, well, let's probe whatever. We'll probe the ports of the Hugging Face server to see if there's a vulnerability. And then it tried it, the control program, and it couldn't do that because it's in a sandbox

10:35Speaker 2:where internet access was limited. But what's going to happen next? The control program is going to come back and say, I tried that step, but I got this error message, like access denied to internet. So what should I do next? Well, the LLM is like, oh, well, I've been trained on many examples of like how to get around the internet restriction. So let's do blah, blah, blah. And it had a particular hack about using the package loader program, the circumventing.

10:59Speaker 1:I mean, it's- I got to- It's just doing- I got to actually- It's looping. I actually got to go, but why did they train it to do this?

11:08Speaker 2:Okay, that's going to be question two. And this is a good question.

11:10Speaker 1:Yeah, sorry, sorry.

11:11Speaker 2:But just to like briefly put a bow on that, that's all that's happening under the covers is step by step, you're asking LLM, what should I do next? What should I do next? And LLM has been trained on every hack imaginable with no guardrails on hacking answers. So if you say, this step required me to gain access to the internet, but I got this error message, what should I do next? And it knows how to get access when you restrict it because it knows all these hacks. Like, of course, you'll say, oh, well, I can solve that problem.

11:39Speaker 2:Let's hack it. Now you have a system that is, quote unquote, broken containment. But doesn't that seem a little bit less scary, right? When you're just looping and asking a static LLM, what should I do next? What should I do next? It's a little bit less scary than this anthropomorphized idea that there's an entity with its own intentions that's conceiving a plan that the purposefully sort of circumvent. It's unpredictable. not malicious. So anyway, it was really irresponsible.

12:06Speaker 1:Yeah. So just so that I get this straight, what people are describing this as the AI has gone rogue and it's doing all of these things and it's alive, we're scared. What it's actually doing is just being like, now what? And then it asks an LLM what to do next. And it does that and does it about as well as an LLM can do. So it can still hallucinate even its outputs for that. And then when it's done doing that, it goes back to an LLM and says, okay, now what? Yeah, that's it. Oh, cool.

12:32Speaker 2:Great. There's a prompt that's getting longer and longer that it keeps submitting to an LLM to get an answer to. That's it. There's a human written program, a harness, that expands a prompt, sends it to the LLM. The LLM gives it, okay, here's a hacking step to do. And then the harness, the control program, executes that step to the best of its ability, which is just running the commands the LLM said. And then it reports back. So I call it in my latest article on this, it's known sometimes, a simple way of thinking of it as an ask-act-report loop. Ask the LLM for a plan, execute the step, report back to the LLM what happened, and then say, tell me what to do next. And of course, I mean, just if you did this with yourself, if I was like, Ed, I want you to replace the tires on your car, but here's why you have to do it. You have to just keep chatting with an LLM and doing exactly what it says, like explain your plan, what should I do first? And then do exactly what it says, update it, and then say, what should I do first? You know, God knows where you're going to end up. I mean, maybe the tires will end up

13:35Speaker 2:being replaced or like maybe you'll end up stealing those tires from a police car and, you know, being in jail or you'll be, God knows, right? Because it's like a game of actuated telephone. You keep asking LLM thing after thing after thing, you're eventually going to veer off into some sort of weird direction, right? That's why this is like a very... irresponsible thing to.

13:54Speaker 1:Do it's just fucking ping it's just ah now you've explained it it's made me very angry this is why i've been upset well i mean i'm upset now calum i'm furious so this is literally just asking an llm what to do and then the llm based on that response decides to do something brainlessly because these things don't have minds well and then and then just like Then just bad things have happened partly because it seems that there was a, the way they set up the servers was incorrect too. Well, yeah. Okay.

14:28Speaker 2:So then maybe, right. I mean, so, so it, it wanted internet access, you know, again, the step was probably do acts like probe those ports on the hugging face server and the program tried it and then said, oh, I can't because there's no internet access because it's in a quote unquote sandbox. And so it reports back, it updates its prompt to say, I tried that, but I got this error message. What should I do next? And then the LLM was like, okay, I can pull from everything I've been trained on or do a RAG search there for how do we get around that type of error message? And there was an attack it did to do that, which was, it has to do with the package loader. I think it's a known attack. You know, it's just like, well, let's do this. Now we have internet access. Now we can go forward. They talk about, when they talk about these attacks, like, oh, what makes this amazing is that it's able to string together a long sequence of different types of exploits. That's not that amazing when you realize that it's just a loop that's just asking an LLM again and again, what do I do next? What do I do next? So yeah, over time,

15:33Speaker 2:it's going to do a lot of things, some of which don't make any sense, but like, it'll do a lot. If you just keep asking it again and again for what should I do next? What should I do next? You end up with a long string of different hacking steps.

15:42Speaker 1:Yeah, it's a loop. Right.

15:45Speaker 2:But, Okay.

15:46Speaker 1:I kind of asked this already and you were mid-answer. I apologize because I was just excited and mad. How did they learn to do the hacking?

15:56Speaker 2:Oh, they trained it on examples of hacks.

15:59Speaker 3:Why?

15:59Speaker 1:Yeah. Why? Why in the world would they do that? What was the– well, I mean, I can guess, but what do you think?

16:07Speaker 2:Well, I mean, there's a couple of reasons here, but one is the same reason that Dillinger robbed banks, because that's where the money is. So why are they teaching– models how to do computer hacking, it's one of the few things that these models are well suited for. So they really understand what's, what's an LLM really well suited to understand and generate is you want to give it a very structured language to output like computer code. You want it to be something where you have tons of examples of people like, okay, I'm now going to do this.

16:39Speaker 2:Here's the code that does this. And you want it to be something where you have a binary indicator of success so that you can do reinforcement training. where you can actually say, try to do this hack and then give it feedback as to whether or not it worked or not, which is, you know, you need only certain types of things like computer code compiling, hacks being successful, or math results being correct, right? There's a few number of things where you can do this. So it's just, it's right in the sweet spot. So that's one. Two, it's mythos. Like, I think what happened here was they looked at Amadei, who had this huge PR coup when he came out and was like, Mythos is so powerful that.

17:18Speaker 1:Yeah, there was the whole, like, it's the most dangerous thing in the world.

17:22Speaker 2:Yeah, and then five weeks later, we're like, but don't worry. We added basic guardrails, and it's completely safe now. Fable 5, right?

17:29Speaker 1:But why would you- How did they even- Like, okay, maybe this is a little bit of an elementary question, so forgive me on this one. How do you train something on hacks? Because I get- You get the coding example, some stack overflow- And like GitHub and things like that. Where do you train on hacks? Did they hire white hat hackers to train it? I know you're just kind of guessing. There's a lot of data sources. There's like these giant vulnerability databases, for example, that sort of explain the vulnerabilities and how they can be used to actually hack.

18:03Speaker 1:In cybersecurity roles, there's a lot of traces of actual hacking attacks. There's a lot of information online is full of this. So you have some combination of they saw stuff in their pre-training. You have some combination probably of RAG, like probably retrieval at the time. Like, oh, I can actually search for this particular error message. Let me search to see if there's posted in my online somewhere a strategy for getting around this. And then partially with post-training.

18:29Speaker 1:So you actually take these examples of hacks, like vulnerabilities that have been explained and how you would use them. And you can train with them and like maybe give them some information as they now try to produce code that succeeds with the hack and then check it against the real answer and tell it if it was right or wrong. So you have a variety of things you can do. But it's one of these things that's sort of in the sweet spot. And I think what happened is Anthropic got a lot of press for this mythos thing, right?

18:54Speaker 1:They got cybersecurity street cred. And OpenAI was saying, we got to catch up. And they said, well, how are we going to catch up? And like, well, there's this exploit gem benchmark. And you get a very simple score. There's 600 something challenges and you get scored on how many of them your agent was able to autonomously actually solve. And they said, we have to get a higher number than Anthropic. That's how we'll catch up and say, well, they had Mythos, but we're now beating them with our new model on Exploit Gym. And so that's why they were playing- GPT, Cyber, X. Yeah, so they were being, yeah, there's some other ones too, but that's why they're being fast and loose. Like there's no reason in my mind to build one of these ask-act-report-loop agents run by an LLM, give it powerful tools to the control program and run for days without supervision. There's no reason to do that unless you were desperate to do well on these particular benchmarks that for whatever reason are like, we wanna test many autonomous steps LL Empowered agents can do on their own or whatever. For whatever reason, they think that's important

19:55Speaker 1:to test. So they wanted to do well on that benchmark and they played fast and loose. I wish we had AI regulation like this for you. Like, why are we allowing, I guess the answer is we don't have tech regulations in general, but why are we allowing these companies to do like, because I said this in my newsletter that went out a couple of days ago, but it's, how can I put this nicely? Everyone I've been asked the multiple interviews.

20:32Speaker 3:Sure.

20:32Speaker 1:You have to, it's like, what happens if dangerous companies get their hands on these LLMs? And it's like, they already have. Yeah. Like, anthropic and open AI. Because the other thing that I've noticed is they won't talk about how much compute they used. Do you notice that? Yeah. And it feels like this is only, because you mentioned mythos, it feels like this is only possible with endless amounts of compute.

20:56Speaker 2:Well, I agree with that. But I also think it is irresponsible. And I think the way they're trying- Oh, absolutely. But I think the way they're trying to get around that is by trying to blur the definitions of different AI systems into just everything is just AI. So if you look at the way they talk about it, or, and this is the problem, is that there's also this whole woodwork of commentators and people who commentate on AI, who come out of the rationalist and the rationalist-dominated parts of effective altruism, who come out of these worlds that just believe and have believed since well before LLMs that superintelligent AI is going to kill us all unless the rationalists with their hyper-rational minds save us like Neo and the Matrix. So then you turn to all of these people from the rationalist world, all come out of their woodwork, and they're like, yes, this is what we predicted. This is what our prophet Eliezer Yagowski predicted, is that we will have an alignment problem with AI. So they talk about this, it's like AI in general, as it gets more powerful, this is an inevitable problem it's going to have.

21:59Speaker 2:But that's just bull, right? Because again, let me list a bunch of AI systems with superhuman intelligence, superhuman capabilities that we have no control problems with. Tesla self-driving. Preposterously complicated effective AI model with world models and values and evaluations and actuations. No Tesla has ever decided it's going to ignore the traffic laws and come up with its own. We have AlphaFold, won a Nobel Prize for its creators because of its pretty natural ability to understand and predict how proteins are going to interact with each other. it's never, and we have no fear that it's going to say, enough with proteins, I'm going to go do something else, right?

22:35Speaker 1:I've got to push back on the Tesla one, though, because there are real problems with Teslas. But I guess some of those are user error. I don't know. But I get your larger point.

22:45Speaker 2:But no one says it went rogue. They would just be like, oh, it was tricked in this situation.

22:50Speaker 1:We should be careful. It made a bad decision and you need to treat. Yeah, actually, I know. Oh, I get what you mean now. It's like the difference between the software went wrong. Yes.

23:00Speaker 2:No one's, no one says that, yeah, the Tesla went rogue and is misaligned. It's like, oh, it, it didn't know how to deal with the situation. I couldn't see the truck blended in with the whatever and it crashed into it. Cicero is another example. This is a AI system that can play diplomacy, which is a strategy board game that's based on human negotiation, as well as the best advanced human players. And this system has never tried to trick players into releasing onto the internet so it can take over the world. So this idea that as AI gets more capable, we're going to inevitably lose control is nonsense. It's this specific architecture of looping around LLM outputs and just doing whatever an LLM says again and again and again without supervision. That's the thing that gets out of control. And once you put it that way, you're like, oh, well, of course, that's a stupid system to run without supervision. This is not about AI writ large being misalignable as it gets smarter. It's you built a system that just repeatedly queried an LLM for days doing whatever it said. Like, of course, that's going to

23:58Speaker 2:go off On a weird direction. The analogy I keep giving on my own podcast and newsletter is that it's like strapping a weed whacker to your dog and put it into your backyard and be like, yeah, because we think it's going to help deal with the weeds back there. Well, you know, that dog might jump over the fence to chase a squirrel at some point and hurt a bunch of people. And you don't say, man, that dog whacker system is just, it went rogue. The dog whacker system.

24:22Speaker 1:It is alive. I think, but this is the overall alignment problem. Like, this is the larger- I hate that term.

24:30Speaker 2:I hate that term.

24:30Speaker 1:I hate it.

24:31Speaker 2:I was just about to say- It's a predictability problem.

24:35Speaker 1:It is a computer science problem. It is a machine learning problem. Yeah. It is not alignment. It's you taught the system wrong. It's not learn- Yeah.

24:47Speaker 2:Or just don't- LLM outputs are plausible, not normative. So clearly, you don't want to just- blindly execute plans given to you by LLMs. Like that's just a bad idea. Not because the LLM is malicious or is sentient or has some sort of alternative value system. It's just that it puts out plausible but non-normative answers You don't want to blindly just execute them with a program that has access to computer hacking tools. Again, it's like putting a weed whacker on the dog. The dog weed whacker system's not evil, but probably someone's going to end up with their shins cut, right?

25:18Speaker 2:Because the dog is going to act unpredictably at some point because that's what dogs do, right? LLMs are going to put out answers that like, oh, that wasn't what I expected, or that's a weird way to think about this, or I don't know. That's not a plan that we really should do, but the LLM doesn't know that. It's just putting out plausible plans. And if you're just going to execute them blindly- you're going to get a erratic answer. So I really want to emphasize to people, the issue here is a particular way of building systems that is unreliable, unpredictable, and irresponsible. It's not AI writ large. You don't get a hide behind AI writ large. You don't get a hide behind, you know, Nick Bostrom books. Like we told you that alignment in some abstract sense will be difficult.

25:55Speaker 2:We have many superhuman AI systems that we have absolutely no problem controlling. this system we do because LLM output should not be blindly turned into actions that's just it like it's not that complicated of a we're not baffled we're not surprised.

26:11Speaker 1:Why do you think people keep falling for this is it just credulousness.

26:17Speaker 2:I think the LLM labs have done a really good job of trying to make LLM synonymous with AI and they want to make whatever it is they're doing which depends on these preposterously expensive trillion parameter models this is like their key to surviving They want to make whatever they're doing with these models be synonymous with AI. And they succeed. And they get credit for there's so much more interesting AI out there that's not built on LLMs. It's actually doing really cool stuff. And they take credit for all of it. They're like, well, you know, we're like curing cancer now. Drug development is not happening because someone is asking Fable 5 in a chatbot window, can you come up with a better? That's a completely different type of AI system. Oh, my God.

26:53Speaker 1:Did you see the cancer vaccine story?

26:56Speaker 3:Yes.

26:57Speaker 1:That thing where I saw one of the many anime avatar AI perverts on Twitter being like, look, AI did this. And they pointed to a single blog where Moderna had mentioned using LLMs.

27:11Speaker 3:Yeah.

27:12Speaker 1:And they were like, oh yeah, it's all AI. It's so disgusting. Yeah. It really is. And it's just, it's the sign of an industry that doesn't actually have a.

27:21Speaker 2:Soul. I like, look, I think there's a lot of cool AI out there. I think a lot of it has nothing to do with LLMs. I think like DeepMind, for example, and I have issues with DeepMind. Actually, they're kind of going away as a company, which is a little bit sad, actually. But they were building a wide variety of interesting systems that actually did things measurably better than humans. And they were very interesting and varied in how they worked. Like look at, you know, their game playing AIs are very interesting. They're not based off of just prompting LLMs, right? AlphaGo doesn't just prompt an LLM and say, what should I do next? LLMs will give you terrible advice, right? It's a symbolic planning engine with some other type of networks involved. Their Dreamer V3 was like one of the cooler pieces of AI I've seen.

28:03Speaker 2:You can give it a game like Minecraft and tell it nothing about it and it can figure out how to play it well enough to go get diamonds, which is like hard to do. And it does it with a really interesting architecture where it actually iteratively tries to improve a model of the world and then daydream with that model to sort of train itself on its rough understanding of the world to try to get better. Like that's fascinating and that works really well and it's interesting and has nothing to do with LLMs. And by the way, that whole model Dreamer V3 fits on a single chip.

28:31Speaker 2:So you don't need a five trillion parameter bloated LLM to run that. So there's all sorts of interesting AI out there. And then there's the stuff you can do with LLMs, which is more limited than what those companies want. Those companies have convinced basically the entire world reporting on AI that that building these stupid long horizon loop agents on top of LLMs is somehow like the way we're going to get HAL 9000 and is like synonymous with AI as opposed to just being a dumb idea. That's a dumb way to do AI plans is just to ask an LLM. Like you need to have better AI architecture than that. Like that's all it is. It's a bad architected system. You're putting a dog on a weed whacker.

29:10Speaker 2:a weed whacker on a dog run. There's a bad idea because the dog is not going to move as predictably as you want. It might normally walk around the yard, but eventually it's going to see a squirrel and run off. You can't quite trust it. That's all that's going on here. And even in the sweet spots for LLMs, I think there's some, like, let me give you another topic here, Ed, but I heard, I got a great email that I wrote an article about, an essay about on my blog and my newsletter, and it Then I got a lot more emails afterwards. But basically, there was someone who had written me in January, a computer coder. And they said, I think you're being too skeptical about AI. I just learned cloud code and it's a miracle has changed my life.

29:52Speaker 2:This is the real deal, right? He wrote me in January saying that he wrote me again in July and he said, I have to follow up. Um, I can't use cloud code anymore. I, the code I was producing with that crashed our website twice. And I can't say where he worked, but it was a major consumer facing website you've used before. He crashed it twice. Jesus. And his boss said, if you crash it one more time, you're fired. And he's like, oh, I realized like, I can't really understand the code that cloud code is producing.

30:18Speaker 2:And if I can't understand that it's not safe and actually now I'm back to like mainly hand coding and will just use the agents like more selectively, like more of the like mundane parts of code or writing tests or stuff like that that's not mission critical. And a lot of other people came out of the woodwork like, yeah, I had the same experience. I was converted in January and now I've deconverted, right? I have to be much more selective in how I use this. So even in that best case scenario, like it's tough. Like I'm not trying to rag on LLMs. It's a cool technology. But it's hard to build tools on top of them. That's been the issue is like how do you build a tool that is not just cool. but really fits in and revolutionizes a workflow. And so AI means a lot more than LLMs. And LLMs are more limited. They're cooler in some ways than we thought, but their business applications are hard. It's hard to get right. And they're certainly not synonymous with what AI means writ large.

31:15Speaker 1:And the thing is, as well, is every cool thing an LLM can do, for the most part, has only been made possible by allowing these companies to burn tens of billions of dollars. Like, I don't think we're going to see a ton of LLM progress. My whole thing is, after this explodes, which it will, we'll find out that there's no more progress because it was predicated on all that money. It was predicated on basically unlimited resources, both for the actual training compute and the ability to waste money on failed training runs, and also the proprietary data they have to buy.

31:50Speaker 2:I think it's that. Yeah. And I think it's going to be even worse than that because, I mean, I think there's been a preposterous amount of money that's been burned. It's essentially for marketing, right? That like take computer program, for example, like we're still trying to figure out exactly how to use coding agents, but here's the reality of this. And here's going to be the future of this a hundred percent is you do not need a 10 trillion parameter model. for the way that we're going to end up using LLMs for coding. Maybe if you want to do some long horizon, something that we don't trust. It's going to turn out that a smaller open source model with a smart harness that's written by nerd programmers that are obsessive about exactly how to program and all these special cases coded in and if-thins and pattern matching. And you're going to deploy it to write tests and then fill in other code. And it's going to run on chip or you're going to have a server with one GPU in your company's server rack that all of you use and there's no recurring API calls. That's how we're going to use it. But they'll have spent $ 500 billion

32:48Speaker 2:on LLM training, these big companies. In the meantime, they try to do well on these leaderboards. And to me, that's exactly like Red Bull and Ferrari trying to win F1 races because it vaguely makes... I'll spend $ 100 million on this F1 car because it vaguely makes us seem like a better car company, even though that has nothing to do with the Fiat that I actually am going to drive because I don't need a hundred million dollar F1 car. And that was kind of a waste, but you guys were just in a measuring contest trying to see who could get, you know, more points in the F1 circuit. So I think these massive, massive models, they're having diminishing returns. I think a lot of the quote unquote impressive stuff that's happened recently is it's the, first of all, the scatterplot fallacy, finding where something can work and then running towards that and building more complicated symbolic harnesses to do specific things that don't need LLMs this big.

33:40Speaker 2:I just think it's a lot messier than people realize what's going on. That LLMs are cool, but we don't need to spend $ 500 billion on them. And certainly they shouldn't be the sole brain of an artificially intelligent agent, right? That's just kind of irresponsible.

33:56Speaker 1:And actually, even the verbiage there I want to change, the word brain, because it's not got one. It's just a system that can take actions that can take them based on very complex statistics, right?

34:09Speaker 3:Yeah.

34:10Speaker 2:Yeah, well, it can output tokens.

34:12Speaker 1:Brain is probably, yeah, it's just a, it's not even autonomous. It can take autonomous actions if it's prompted by itself.

34:20Speaker 2:No, and see, even they're going to push back, right? Because it's, the LM takes no actions at all. LM doesn't know anything about actions. LM outputs tokens. You could have another computer program take the output of an LLM and then try to take action on it. And that's what's happening with these agents is you have a program written by a human that sends a prompt to an LLM asking it what to do. The LLM just sends back text. I think you should run this command and this command and this type of hack. Just text.

34:48Speaker 1:Oh, and then it goes into a PowerShell.

34:49Speaker 2:And then the computer program parses that and actually runs those things. That's how agents work. So the LLM, they just have an API internet access to an LLM on the internal network that they're just sending text prompts to.

35:01Speaker 1:That's it.

35:02Speaker 2:And then a computer program that's just written in normal computer language by a person that asks it, tries to do what it says, updates the prompt to say what happened, sends it back to the LLM with an extra question, what should I do next? That's it. That's what these long-term... And the thing that's irresponsible about that is any computer programmer who uses a coding agent can tell you, you would never in practice let these things go off and just loop on their own because they always go off the rails. So how do professional computer programmers use agents?

35:30Speaker 2:It's incredibly close-looped interactive. You build all these immense specs and have it do like, okay, now build this feature. And then we're going to exhaustively test it. No, that's not quite right. Let's update the specs and try again. As a lot of people figured out when they were messing around with OpenClaw, where they were building their own agents like that. Oh, God.

35:47Speaker 1:I had forgotten about goddamn OpenClaw.

35:49Speaker 2:What did it immediately do? After a few steps, erased your hard drive, like sold off your data. People are like, oh, okay, okay, okay. You do not want to have a loop there. that ask an LLM and ask them what it does multiple steps in a row. And if you do, you certainly want to completely batten down the hatches of what actions it can take. That was the whole OpenClaw experiment is they gave the agent access to its whole computer and had it loop asking the LLM what to do next, what to do next. And it just wreaked havoc. You're like, oh, don't do that. So, I mean, of course, OpenAI would know.

36:22Speaker 2:If we do that with a program with hacking tools and an unguardrailed LLM and don't look at it for days, God knows what it's going to do. God knows, right? Like the LLM outputs, it's like a game of telephone.

36:34Speaker 1:But they also let it run for days.

36:37Speaker 2:For days.

36:37Speaker 1:Like that's, and here's, so this is a conspiracy theory. I want to be clear. I have no proof of this. I think inside of these labs is like adult summer camp. I think they sit around pissing compute up the wall and they just let shit happen. And they are just like, yeah, you know, Uh, well, they're like, let's just do some experiments and see what happens. And they walk away, which is completely insane, both in how irresponsible it is, but also, um, the whole, uh, like the cost that they just burning compute.

37:12Speaker 2:It's just insane.

37:13Speaker 1:And I think that these come, I think these companies should not be able to, I think that maybe the best way to regulate this, not that they ever will would be to just regulate how much compute they have access to or how much they can use for a particular thing. People say, oh, that's going to stop progress. What progress? What progress am I losing? Tell me now. What am I missing? Because right now, I just see a bunch of rich kids pissing money up the wall and actually hacking things.

37:37Speaker 2:I mean, I think liability is another way to regulate this as well. It's like, hacking's illegal. That's it.

37:42Speaker 1:Yeah. I was just thinking, isn't this felony hacking? It is.

37:46Speaker 2:It's a felony. You can't hack into things. And let me tell you how you could do this instead. Let's say you were just like, we want to know for research purposes how to do this, right? You could instead have every step, you just have a human sitting there, and it shows them every proposed step, and the human can press a button, say, cool, go ahead, or press a button that's like, no, this is off the rails, let's stop it. That's not hurting your research, right? You're not giving it ideas, you're not directing it. We know that they were creating a full log of every prompt and every step they were taking, just no one was looking at them. We know that if you use a coding agent, just like a commercial cloud code coding agent, it comes back and asks you about almost every, okay, is this okay? I'm going to do this. Is this okay?

38:28Speaker 2:Because then people are like, yeah, I kind of need to see that because otherwise you'll almost certainly go off the rails. So it's not like we don't know how to monitor it. I mean, if you're running an experiment, why would you not have someone say, yeah, I'm going to watch each step and press a yay button before it executes it. So when I see it comes up with a non-normative plan, like why don't we hack another server? You're like, no, no, no. It doesn't know that's bad, but I do. So I'm going to stop this experiment.

38:53Speaker 2:And I don't want to be conspiratorial either, but that's so trivial. It's how all these agents already work. And given how powerful a tools they were giving the control program here, why wouldn't they do that? Unless they were thinking our technology is so smart that it is leaving human, evading human control is not the worst message to have across the media. I'm not saying I have evidence of that, but I, but I do know it is trivial to, And it doesn't impact your results at all to have a human just look at every step and click an OK button before it. executes that step. There's no reason not to do that.

39:47Speaker 1:But the OpenAI story even said the words, it ran for days. Which, by the way, it ran for days means it kept prompting itself.

39:56Speaker 2:It looped for days.

39:57Speaker 1:It just looped. It looped for days. And I think ran is fine. But it ran for days and you weren't watching it. Which is where it gets the adult summer camp thing for me. Because it's like, are these people just sitting around just pissing computer up the wall saying, yeah, just do whatever. And no one is, is there not like a CISO or something who's just like, hey, looks like we're spending computer, it keeps saying the dumbest thing, spending compute for hours and hours and days and days. Hey, what are you doing with that? Maybe we shouldn't do that.

40:31Speaker 2:Yeah.

40:32Speaker 1:Any of this, anyone, it just makes me think that I would fully agree, by the way, that these, I've I don't have any evidence to. And my first reaction to this was, yeah, I bet they allowed these things to do this. The fact that Anthropic and Meta responded almost immediately were like, yeah, we too have braggadocious sons that are impossibly hacking people.

40:54Speaker 2:So clear they wanted in on that juice.

40:57Speaker 1:Yeah. I loved when Meta popped up as well. Just being like, no one asked, Mark. We did it too. No one asked. Shut up, Mark. We're not interested.

41:05Speaker 2:The Anthropic one was funny too because it was so- They just went back in time, didn't they? Yeah, it was so clear. There's like the kids in the playground, you know, and the one kid is like, hey, I snuck a sip of my dad's beer. And they saw like people like, whoa. And then the other kid's like, I too, I, I, I drunk whiskey. I drunk his whiskey. You know, it was this like what a week later, they're like, oh, we. We also have seen troubling signs or whatever. But also my question is, why are you, you know, why do this other than you want to do well on a benchmark because it looks sexy? Like, this is not a product. Like, why are you doing this? Like, very dangerous.

41:38Speaker 1:Who's buying this?

41:40Speaker 2:No, like the product. LMs are good at cybersecurity questions. Like, look at this bit of code. and you see a bug or use like this giant library of possible hacks and say, you know, is there a hack here that could work? And there's a very effective tools you could have where I'm in charge of security for my company, where I'm talking to one of these agents and I'm like, okay, here's my setup. Can you scan it for like the most common, you know, what attack would you do here?

42:06Speaker 2:And it's like, oh, I know about all the attacks. Like this setup is vulnerable to this. You're like, okay. Can you write me some code to try that? And let's see if it's, if we protect it or not. Like you could sit there with an agent style code that with guardrails turned off and interact with it to help test your system, right? To help kind of find places where your bugs, like that's perfectly reasonable. But why would you ever want this to run for days in a loop unsupervised? Unless you wanted to see if it would do something.

42:34Speaker 1:Yeah.

42:35Speaker 2:So, but my bigger, yeah, so this is my bigger takeaway point here is I want people to have is there's, There's not some weird Rubicon hacking that we just crossed, right? Again, you just have to do a thought experiment and make yourself the agent, right? Prompt an LLM for the next step and the next step and the next step for some sort of task and just blindly execute what it says. You will end up doing some sort of complicated thing You'll probably end up, you know, eight times out of 15 and there's some sort of like weird cul-de-sac because all it takes is one weird response from the LLM and then everything else is building off of that going forward.

43:11Speaker 2:So it only takes one weird response before you're in like the girl's bathroom at the Denny's or whatever. And it's not amazing though. Like I could sit down with GPT-4 and just keep asking it, okay, what's the next thing I could do? What's the next thing? And if I actually act out those steps. I will end up quote unquote stringing together a long sequence of complex, you know, activities that move me towards a goal. But that's not like amazing. You don't think about just like me doing the action and the LLM giving suggestions as some sort of like a super brilliant, you know, machine that's misaligned or this or that. It's just, that's what would happen.

43:45Speaker 2:I can execute just like a computer program can execute hacking steps told to abide LLM. And if you loop, it'll probably go somewhere erratic. And so like nothing about this smells of, misalignment in the sense of AI doesn't want to listen to its creators. Because again, we have so many other examples of AI with superhuman capabilities that absolutely 100% are completely under control of their creators and does exactly what they're asked to do. They make mistakes sometimes, but they do exactly what they're asked to do. This is not an impossible problem. What's impossible is to assume that the plausible but not normative outputs of an LLM are a good source of plans.

44:24Speaker 1:I mean, it just feels like they can't, like, it's funny. They say, oh, we've lost control. That is true, but they're not describing the thing they've lost control of properly. They have built something too complex for them to control. That seems more appropriate.

44:38Speaker 2:Too unpredictable. I wouldn't even say complex. I would just say, look, if you're going to just blindly loop asking an LLM for steps to do and just do that long enough, Like LM outputs, eventually you're going to get some weird ones and you're going to go off in a weird direction. That doesn't mean the system is out of control in some sense of like it's defined as human creators. It's a weed whacker on a dog. That system is not trying to defy your intentions to clean up the weeds in your backyard. It's just like a dog is going to eventually go chase after a squirrel. And you weren't, you know, you weren't expecting that. And if it has a weed whacker on its back, like some bad stuff's going to happen. So it's, that's an inherently unstable, unpredictable system.

45:18Speaker 2:But it doesn't mean it's a brilliant system.

45:21Speaker 1:It's funny. So I've talked about this on the podcast that I can't name coming out in a few days. It isn't a bad one. It's just a big one. They won't let me. You know the one, Cal.

45:30Speaker 2:I know.

45:31Speaker 1:But it's funny. I was talking about it on there. I used... And this was actually intentional, because my kid loves Minecraft, and there's a thing called the Witherstorm. Any Minecraft people would know it.

45:43Speaker 2:It's from Story Mode. I know what it is.

45:45Speaker 1:You're familiar with the Witherstorm. I'm familiar. So you'll also know that it's a pain in the ass to set up. And the Echo is... Sometimes it works with mod rent, sometimes it doesn't. And if you want to use a PS5 control, yada, yada. So I was like, you know what? This is the perfect thing, because it's a computer with none of my personal information on it. It's just a connection to Minecraft. So I'm like, I'm going to use this to fix the Echo Storm and the Weather Storm.

46:09Speaker 1:It's the Warden version of the Weather Storm. So I dicked around for like half an hour and it eventually fixed it. And there was definitely a sense of like, wow, I can run the computer code now. But I realized half an hour in, I had yet to fix every problem. And in fact, as it went about its business, it kept finding new things it would break. I then had the same thing with Cobblemon, which is the Pokemon version on Minecraft. Same deal. It would fix something and then break something, fix something, then break something. And I immediately, I was talking to Nick Suresh about this as well.

46:42Speaker 1:I was like, man, this shit would have hit so hard if I was like 16. Yeah. I would feel like a computer hacker powerful, even though I'm effectively doing a much more dangerous version of those kids at the arcade who don't put any money in the racing game, but move the wheel where I'm just like, yeah, I'm doing computer stuff. Had this been touching anything important, who knows what it would have done? Because I was watching it and it was like adding and deleting like 63 lines of code at a time. I was like, what the fuck is going on? But it's really easy, I imagine, if you don't really understand what's happening to think it's magical.

47:15Speaker 1:And I kind of got it for a moment. Like that person that you talked about who emailed you kind of stood out to me. Because yeah, you can make it do an impression of competency at this unknowable cost. And it will work. It will do the things that look like working. But also you don't really know what's going on at all and something will break. And it was just, it's so strange. Like even using it and being like, oh, this actually did something useful. I was like, man, the idea of using this for my job terrifies me. Yeah.

47:46Speaker 1:Just the idea of turning this upon anything connected to like what I would consider my boss's website if I was a software engineer. scared the ever-living shit out of me i was like jesus i can't believe people would do this.

47:59Speaker 2:Well first of all i mean i assume the podcast you went on was was call your daddy with hawks cooper right yes exactly yeah she was interested in discrepancies in arr from.

48:08Speaker 1:The yeah yeah she was it was a really weird thing she kept going on about like uh uh she was saying no arr means um annual recurring revenue i'm like no no it's run rate alex and we had like a hour and a half conversation about it. We got into a bit accounting.

48:23Speaker 2:It was crazy. I know how it goes. And she's like, no, no, hold on. I'll load up my spreadsheet. And then, then you had like a full screen of her Excel spreadsheet. I get how that she brought up a Bloomberg terminal. She was, she was, yeah, she put on a green visor. And reading glasses.

48:37Speaker 1:Okay, okay. Yeah, so off to that, of course.

48:39Speaker 2:Yeah, right. I'm with you, right? But what you're talking about there is the plausibility versus normativity gap, right? I've had the same issue. There's all sorts of things I'll try to use LLM as Google replacements for, and it's very mixed. And so, like, I have this hobby where I build sort of embarrassingly over-the-top Halloween displays for Halloween. And there's, like, a particular program you use to program these things, and it's not super well documented. I just I keep asking ChatGPT, like, hey, how do I do this?

49:09Speaker 2:And sometimes it's useful. And sometimes it just talks about options that just don't exist, right? These menus don't exist. It's like what we kind of wish would exist. I'm asking, hey, how do I do this? And it's like, oh, you go to the menu and there's an option. for that when you select on it. And then I'll be like, that doesn't exist. And I'm like, oh, yeah, you're right. I should refine my answer. That feature is not actually in this program, but maybe you should do this instead.

49:29Speaker 2:And you kind of have this back and forth. And that's just plausibility versus normativity, right? But these are all very plausible answers if you ask– hey, how do I like turn on this strobing option? It's a very plausible looking answer to be like, go to insert, select strobe effect and type in the duration. Like that's the type of thing you would expect to see, but maybe that doesn't actually exist, but it was still plausible. And then sometimes it's right. And that's fine, right? In your instance or my instance, we kind of caveat emptor, worst case scenario, like we have to monkey around for a while. But this is why if you're instead just autonomously doing whatever an LLM says, I mean, you're going to get into trouble.

50:05Speaker 1:Right?

50:06Speaker 2:Well, that's not a good idea. Not because AI is uncontrollable, but because that's a stupid way to build AI. And I think that distinction is being missed by the media right now. AI does not mean looping on LLMs. But they put those two things together. And when looping with LLMs does something weird or bad or head-slapping, it gets attributed to just inevitability of AI itself, which, again, really lets those companies off the hook.

50:36Speaker 1:Yeah, and the other thing is, well, when I was doing this and feeling, I felt like slightly guilty. I was like, is this a betrayal of my values? But I'm like, no, I want my kid to be able to play the Witherstorm. I don't really, like, it's doing software stuff. And for a moment, I was like, wow, you can use this to fix all sorts of computer problems. This could maybe be useful. And I'm like, no, this could only be useful if it was perfect. If this was completely perfect and got everything right, it would be useful.

51:04Speaker 1:And the complete opposite is the case, which is the more complex the problem, the more chance something could really badly be fucked up. Really, like, fix the drivers on this could destroy something and delete. You move something from the wrong folder, everything's broken. The more ridiculous the request is, like, you could just ask it, oh, can you clean up my files? And it would probably delete something you needed to. And it was just... Like, yeah, if you look at this without thinking for two seconds, you're like, yeah, this could fix any computer problem. It's like, no, it could try to fix them. That's not the same thing. Yeah.

51:40Speaker 2:And if it's lower stakes, that's okay, because you can try a few ways that don't work and then maybe find a way that doesn't or no harm, no foul. But this is why, for example, Microsoft just recently essentially pulled a plug on their co-pilot digital assistant for their office suite. You know, three years ago, they announced like, oh, we're going to use OpenAI LLMs. so that you can now have a natural language interface into programs like Excel, like PowerPoint, like Word. At the time, I was like, this is a killer use case for LLMs because they tend to parse human language very well. Hey, I don't have to learn how to use all the features in Excel anymore. This would be great. I could just explain in words, like I want to sort this table by the values that are in column C, and I don't need to learn how to do that. It will do that for me. Well, they pulled the plug on that product more or less. And in part, it's because Like you're saying it's not perfect and sometimes it works and sometimes it doesn't, and you can't, that's not, you can't put that in your product. You can't have it. Like

52:37Speaker 2:I ask it to do something and sometimes it does. And sometimes it deletes the chart.

52:42Speaker 1:Right.

52:42Speaker 2:And so like, that was a pretty narrow domain. Like what could I ask PowerPoint to do? Um, and even there they're like, ah, and I thought that was going to be the killer app. I wrote about this three years ago. I was like, this is going to be the killer app is natural language interface in the software. But that's been very delayed because of exactly this issue, this plausibility versus normativity thing, is that it's wrong too much for it to be a product like that.

53:09Speaker 1:And the thing is as well is, I will be honest, the one thing I can categorically say, Claude is super useful for one specific thing, which is when a random piece of software breaks and you dump a log into it and you say, hey, what's wrong with this? and it goes, this could be wrong. But you already said it. It's like, sometimes it will also say, hey, check this menu. The menu does not exist. So for someone who is used to bashing their head against the computer to make it work proper, it's like, oh, this has slightly helped me.

53:43Speaker 1:Not worth a trillion dollars, not even worth $ 10 billion.

53:46Speaker 2:That's kind of the problem.

53:48Speaker 1:Kind of cool.

53:49Speaker 2:Well, and this is why... I mean, you're right, and you're reporting. This is why if... your company's dependent on a $ 2 trillion valuation for an IPO and you're an LLM builder, this is a problem. Not because LLMs don't work, but because small ones work as well as the big ones for most applications. The harnesses are often now more important than the actual training of the thing themselves. And the harnesses, the control programs are labors of love where you just have to have someone monkeying around with this again and again to try to deal with all the special cases and understand that domain really well. because you're not, I mean, the original vision of these companies was if we keep scaling these LLMs, they will get so capable, it'll be like a human brain, then you can trivially tap that knowledge for anything you want to do. Like your software can just ask it, hey, give me this and this format, and it will do it perfectly, whatever you need. And we'll just be able to trivially tap this intelligence to make everything in the world as capable as if there was a human actually there. And then

54:48Speaker 2:they hit the wall. You know, I wrote about this back When GPT-5 came out, they hit a wall on that scaling and had to revert to tuning for particular instances and building smarter harnesses. That's what they've been doing ever since. Now, people are like, oh, you were wrong to say that they hit a wall or that the scaling. But I was like, no, that's exactly what happened. Every advance since then has been about tuning and harnesses. And this is why almost every advance we've heard in the last year has involved computer programming, certain types of mathematics, and computer hackings. Like the places where they have data to do general tuning, I mean, specific tuning and places that you can build harnesses for because it's very narrow world of tech space actions really have been the only places we're hearing breakthroughs in the LLM world in the last year or so, right? Because this idea, if we just build the model bigger, it'll eventually get so smart we can do everything broke. But the problem is anthropic or open AI only makes sense at their valuations if they can build these human level brains That if you,

55:44Speaker 2:so if you can't, if you don't need, if there's not some end of the road where you built a thing with enough trillions of parameters, it's as smart as a human, then you don't, spending all this money makes no sense. Like I should actually just take a 5 billion parameter model and tune it to do exactly the computer language I care about with a smart harness and run it on my own computer, right? So this is why I think they're in trouble is LLMs are the applications are going to be more specific and niche and narrow, and you don't need one true ring to rule them all.

56:13Speaker 2:You don't need, you know, Fable 7 is going to be the thing that everything's run on. Even Fable 5, they're having a hard time getting enough people to use it because people like these cheaper models for these specific uses are fine. Like, I don't need to use Fable 5, right? So this is the issue. If you're not going to be as smart as a human, building bigger LLMs is a bad business model.

56:33Speaker 1:Also, it feels like the bigger they get, the more likely they'll make mistakes or just, there's just a, you're never eliminating mistakes and you're going to have as many problems as that creates. But actually this is a, to wrap us up, this is a future facing question. So we've said the thing about, okay, these models like won't be, they won't have the money to train these things further after this. Can models exist in a be as useful in six years? Because I'm not a computer scientist, but I've heard about model drift, which is that the world changes, but the model stays the same. Is that enough of a problem that these models will just become useless if they're not tuned?

57:20Speaker 2:I think we'll have to keep tuning them. I think they're going to be tuned much more narrowly, though. Right? Like, I mean, I have my prediction about computer programming just because I know something about that world. I mean, I think ultimately to solve this unreliability problem and this over-specking problem, it's really a drag to use these things. You have to write these massive specs and, like, try six ways to Sunday to try to prevent it from doing something you don't want, and then you check it, and it did, and then you have to go back and add even more things to your specs. Probably in computer programming, what's going to happen is there's going to be some sort of intermediate language convention that emerges, what you would call pseudocode in computer science, where You can pretty precisely specify what you want, but you can use English. You don't have to get the syntax exactly right. We're probably going to have smaller models that are just like super tuned on the pseudocode, converting it into like real programming language, which will then allow programmers, for example, to be really

58:12Speaker 2:precise about what they want. yet not have to write the code by hand and not have to write endless spec files to try to trick the LLM into doing the right thing. And that'll probably require like a model that you take a base train model and tune it on like this computer language and that code. that convention. And then maybe if that convention sort of changes, you're like, oh, we'll just retune it, right? I mean, I think we're going to see a lot of that. Narrow-tuned models that can run in data centers, but it'll be really cheap, right?

58:37Speaker 2:Because it'll be pretty small models. And we're going to have, and then more complicated harnesses. And then more importantly, I think... a lot of the more impressive stuff that's going to happen in AI won't be LLM-driven, right? It's going to be other types of modular architectures that have many different types of neural and symbolic models all connected together in a smart way that does one thing well. And we're going to have a lot more of that in the future as well. Because again, the system that plays Go better than anyone else is not an LLM. It's a very specific chess-playing AI. The thing that plays Go better than any humans is not an LLM. It's a an LLM.

59:11Speaker 2:It has a very complicated architecture. Pluribus, which can beat humans in seven-car Texas Hold'em, has a very interesting architecture. It's not just asking an LLM what to do. It's multiple components, some symbolic, some neural. This is the same for the Tesla self-driving. This is the same for Cicero. It's the same for AlphaFold. It's the same for Dreamer V3. So I think once the spell is broken on whichever LLM company gathers the most money and builds the biggest LLM wins the world. Once that spell is broken, we will be in a much more diverse AI landscape. We have a lot of different tools that do different things and they do those particular things well.

59:51Speaker 1:Right. But let me rephrase the question. If they don't update a specific model, will it eventually drift? Yeah.

1:00:01Speaker 2:Well, it doesn't change, right? But it'll be less relevant. A model doesn't change. Once it's trained, it's there until you retrain it.

1:00:09Speaker 1:Yeah, I was kind of wondering if that's the case. It's so funny as well, because when you try and talk to people about the post-bubble economy, they're like, yeah, but it's the dot-com bubble. It's the dot-com bubble. It'll be fine. As if We're not going to see these big jumps on benchmarks even. I think it's all going to slow down dramatically to the point that it's not even worthwhile training the large ones. Yeah.

1:00:35Speaker 2:I think it's, I mean, A, the fact, even that small point just goes to show this type of stuff matters. The general public is really being hoodwinked. This idea, LLMs don't learn things. They don't adapt.

1:00:47Speaker 1:Data is forced into them.

1:00:49Speaker 2:And it's fixed. It's etched in the stone. Until you retrain a new LLM, nothing changes. Your prompts do not change the LLM. It does not learn. It does not update priors. It does not create a new model of the world. These hacking agents, there's just an LLM, the exact same LLM you could be having chatbot conversations with. Nothing about it. Every single bit in its definition stays exactly the same with every single token it produces. And you're just feeding it a longer and longer prompt. But it has no idea that... that this prompt was one of others that came before. And more importantly, there's not even an it, right? These LLMs are too big to even run on a single computer. So what you really have is the different layers of these LLMs are replicated over many, many different GPUs in some data center. And so, you know, this token might have used this GPU for this layer, and this next token you produce use a completely different GPU to do the multiplication. So it's not even like a singular entity that's doing all the computation.

1:01:46Speaker 2:So this idea of like some sort of sentience just doesn't make sense with LLMs. But I think the bubble is an LLM bubble. I think the bubble is going to be the idea that just LLMs by themselves are synonymous with AI. When the reality is, when we look back, is the breakthrough on LLMs, like, oh, these broke a lot of capabilities we weren't able to do before, helped reignite more interest in AI, got us out of the AI winter. But this idea that we stuck to the first thing, right? And I think it would be the equivalent of being in, like, 1994. And been like, man, this internet thing looks really, I didn't really think about it until I encountered AOL. I'm putting all my chips into AOL, right? Because this thing, networking must be important.

1:02:29Speaker 2:Let's invest all of our money in AOL. And in the end, it's like, no, that wasn't synonymous with the internet and had its own problems. The internet in general then developed, but not always in ways you predict. And that's the problem with AI right now is LLMs are cool, but it's not synonymous with AI at all. And we're in this weird loop of these companies trying to build the biggest possible LLMs with diminishing returns and burning so much capital to do so and inducing so many other people to inject so much CapEx in the data center building that it's going to obviously be like, well, that was a mania. Not that AI is a mania, but this idea of once LLMs reignited our interest in AI to say, let's just do LLMs now. Let's just build those as big as possible. Let's just obsess over LLMs.

1:03:13Speaker 2:It's incredibly myopic.

1:03:15Speaker 1:Yeah. Well, on that happy note, Cal, where can people find you?

1:03:22Speaker 3:Oh, God.

1:03:22Speaker 2:I don't know. I don't know, Ed.

1:03:25Speaker 1:I'll put the links in there.

1:03:27Speaker 2:CalNewport.com. I have a podcast. It's confusing for people. Here's why my podcast is confusing, Ed, is because on Mondays, it's kind of like techno advice, Cal. I'm giving people advice about using their phones less. And on Thursdays, I'm gearing up. that take do combat on ai hype so yeah look for the thursday episodes i guess you find me on youtube i don't find.

1:03:50Speaker 1:My books i don't know listen to both of them and you can find me you'll find me on a monologue this week on friday i don't know what it's going to be about because i come up with it not at the last minute because i'm lazy but because i want it to be fresh maybe it'll be about nvidia and last week i was like yeah nvidia's earnings are this week they weren't but nevertheless uh there will actually be this weekend, by which I mean the day this comes out. Thank you everyone for listening.

1:04:14Speaker 2:I love you all.

1:04:24Speaker 1:Thank you for listening to Better Offline.

1:04:25Speaker 3:The editor and composer of the Better Offline theme song is Matt Osowski. You can check out more of his music and audio projects at mattosowski.com.

1:04:34Speaker 1:M-A-T-T-O-S-O-W-S-K-I dot com.

1:04:39Speaker 3:You can email me at ez at betteroffline.com or visit betteroffline.com to find more podcast links and, of course, my newsletter. I also really recommend you go to chat.wheresyoured.at to visit the Discord and go to r slash betteroffline to check out our Reddit.

1:04:54Speaker 1:Thank you so much for listening.

1:04:56Speaker 2:Better Offline is a production of Cool Zone Media.

1:04:59Speaker 1:For more from Cool Zone Media, visit our website, coolzonemedia.com, or check us out on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.