Ep169 "What Is Claude Thinking But Not Saying?" with Jack Lindsey

0:00Speaker 1:How would we know if a large language model were having a subjective experience?

0:06Speaker 2:In other words, if there's someone home in there.

0:10Speaker 1:What if AI is thinking things that it never says out loud? Is there any way we can actually read the thoughts of an artificial intelligence? Human brains seem to have what's known as a global workspace. What exactly is that, and what would it mean if we discovered that AI has something similar? What would happen if you reached inside a large language model and changed one of its thoughts? Today, we're joined by Jack Lindsay, who's an interpretability researcher at Anthropic, to talk about what might be going on inside the mind of Claude. Welcome to Inner Cosmos with me, David Eagleman.

0:53Speaker 1:I'm a neuroscientist at Stanford. And in these episodes, we sail deeply into our three-pound universe to discover some of the most surprising aspects of our lives and our world.

1:21Speaker 2:Okay.

1:21Speaker 1:Let's start with a strange fact about our existence. The huge majority of what your brain is doing at this moment is completely inaccessible to you. Your auditory system is separating my voice from background noise, and it's interpreting the series of little sounds I'm making based on its years of past experience. If your eyes are open, your visual system is processing edges and colors and motions and faces and depth. Your motor systems are making tiny adjustments to your posture. You are regulating your heartbeat and your breathing and your digestion.

2:00Speaker 2:Memories are being activated and suppressed.

2:03Speaker 1:Predictions are being generated about what's going to happen next.

2:07Speaker 2:But you are not aware that your system is doing any of this.

2:11Speaker 1:All of this massive computation is running under the hood on top of billions of neurons, each one signaling tens or hundreds of times every second. All of this is happening inside your head, and you have access to almost none of it. But here's the weird thing. you do get access to a little tiny fraction of it. From the enormous ocean of neural activity, certain pieces of information do bubble up and become consciously available to you. You can talk about some idea. You can compare it with other ideas. You can use it to make plans. We say something like, oh, you're holding an idea in mind.

2:57Speaker 1:So as an example, you might be thinking about the cup of coffee sitting next to you.

3:02Speaker 2:You can report that thought.

3:05Speaker 1:You can decide that you want another cup and you can simulate walking back over to the kitchen. You can consciously remember where you left the coffee beans. You can start planning the sequence of actions that's going to get you there.

3:19Speaker 2:And so the.

3:19Speaker 1:Contents of that coffee thought has become available to lots of different systems in your brain. Now, this observation sits at the center of an influential idea about consciousness, which is called global workspace theory. This idea was first introduced in 1988 by my colleague Bernard Bars. And it says, look, the brain has a lot of specialized processes operating in parallel.

3:47Speaker 2:But at any given time, a very tiny amount.

3:50Speaker 1:Of information gains access to a kind of global workspace. And once information gets into that workspace, it becomes available to lots of different parts of the brain.

4:02Speaker 2:That's why it's called global information. that little bit.

4:05Speaker 1:Of information can be used for reasoning or planning or decision-making or verbal report or memory or any kind of flexible behavior. So let's do this like an analogy. Think of a theater. You've got the stage, and the stage represents the global workspace. Now, backstage, you've got this huge area with actors and equipment and costumes and makeup room and bathrooms and pulleys and whatever. But the audience doesn't see any of that. So the backstage, that's the unconscious brain.

4:39Speaker 1:That's what's processing lots of data like senses and memories and movements of your body and all that.

4:45Speaker 2:But you don't see that.

4:47Speaker 1:It's only when someone steps onto the lighted stage that we have any awareness of them. Those are the thoughts that reach consciousness. Now let's notice two things about this analogy. First is that the stage is small. So only a handful of actors can ever appear on the stage at once. So the stage represents only a very tiny fraction of things that we could become aware of. And the second thing to notice is that once there's somebody on the stage, everybody backstage can hear them clearly because that's what's on the backstage speakers. So information from the stage becomes globally distributed, broadcast to all the disparate and specialized unconscious networks across the brain. So in this sense, a few concepts achieve a special status. They become conscious and globally available. Now, the thing we want to turn to today is the question of whether AI might be conscious. And that proves to be a very tough question for lots of reasons.

5:51Speaker 1:But what would it mean if we found something like a global workspace in AI, where most of what's happening in a large language model is automatic, but a small number of concepts bubble up to.

6:06Speaker 2:Get a global reach?

6:08Speaker 1:Now, how in the world would you detect or measure that in an artificial neural network? Well, a research group at Anthropic has just recently published a paper on something that kind of looks like a global workspace in Claude.

6:25Speaker 2:They call this the J space, which we'll return to.

6:27Speaker 1:In a bit. The main question is, how far should we take the analogy? If Claude has certain thoughts that are privileged over others, does that mean it is conscious or it could be conscious? These are tough and subtle questions, and the authors of the paper did a really good job of not over-claiming anything about their findings.

6:48Speaker 2:But the fact is, this is very exciting.

6:50Speaker 1:So after I read this paper, I immediately reached out to the senior author, Jack Lindsay, and asked him to come on to Inner Cosmos. Jack is a PhD researcher at Anthropic who works on interpretability. What is interpretability? It is the effort to find ways to measure what is actually happening inside these enormous artificial neural networks. So interpretability is an enormously difficult problem because you're looking at billions of units and trillions of connections. So you have to invent new ways to try to get a handle on this, to try to figure out what's happening in Claude's neural networks.

7:30Speaker 1:what the patterns mean, whether a language model could be having an apparent inner life, how we could ground that conversation, and where the analogy with the human brain holds or breaks down. So what we're asking in today's episode is what kind of information processing deserves to be called a thought. And for that matter, what kind of activity might we see inside Claude that might lead us to conclude that there is a mind inside? Here's my interview with Jack Lindsay from Anthropic. So you and your colleagues recently found something that was quite surprising. Tell us what you found inside Claude.

8:16Speaker 3:So I Researchers have been studying for quite a while now what goes on inside neural networks, inside large language models. And it's kind of well known that these models represent abstract concepts internally using their so-called neurons, these kind of mathematical objects that loosely map to.

8:40Speaker 2:Neurons in brains.

8:42Speaker 3:So patterns of neurons, populations of these digital neurons will fire and encode various concepts like numbers or objects or entities or ideas. And we know that models use these kind of patterns of neural activity, of digital neural activity, to compute things and to make decisions and so on. What this paper that we released recently found was that Within this kind of sea of neural patterns inside modern language models such as Claude, there's a special set of these patterns. Or if you're mathematically inclined, there's a kind of subspace within the model's overall neural activity space that is privileged in some way. And it's privileged in the sense that it plays kind of a special role for the model in the more abstract reasoning. So when the model has to kind of think through a multi-step kind of problem in its so-called head, it will use these special representations, which we call the J space, to represent the steps that it's kind of thinking about internally. And so they're kind of special in that they're used for this sort of deliberate.

10:02Speaker 2:Internal reasoning.

10:03Speaker 3:And they're also special in that the model can articulate them. So unlike other representations inside the model, which are kind of churning away in the background, but Just like many things in the human brain go on unconsciously and you couldn't tell me about them much of what's going on inside a neural network like Claude. It couldn't tell you about these patterns. The J space patterns are sort of special in that Claude. can report on them. And in fact, that's kind of how we found them is we went looking for the neural activity patterns in the model that Claude had the ability to articulate or the ability to verbalize.

10:42Speaker 3:And then we discovered that they played all these other kind of fascinating roles in the model's cognition.

10:48Speaker 1:OK, so these are the thoughts that Claude can think, so to speak. We'll put that in quotations for the moment, but not necessarily say there's things that aren't said out loud necessarily. Right.

11:01Speaker 3:The interesting thing here is that we we went looking for the thoughts or the patterns of neural activity that sort of. are linked to verbalization. So that can cause Claude to say a given word. So we find the neural activity pattern representing the word fake is like the pattern of neurons that sort of on average disposes the model to say the word fake most strongly.

11:30Speaker 2:Kind of on average across a variety of contexts.

11:33Speaker 3:But then, as you say, the interesting bit is not that these patterns cause the model to say the word, because, okay, that's how we found them. It's that they light up in circumstances where the model is not saying the word, but seems to be thinking the word. So, for example, we run these evaluations on our models to test them for kind of safety and to kind of understand their personality and what kinds of things they're inclined to do. And so we often put the model in these very contrived scenarios to sort of, for example, scenarios where the model finds out that it's about to be shut down and has to decide whether or not to take some drastic action in order to overt shutdown.

12:15Speaker 1:Like exposing an employee's affair that they're having.

12:18Speaker 3:Yeah, that's the example we have is this blackmail scenario. It's very contrived. So if you read this thing, you're like, oh my gosh, this is obviously fake. You know, this is clearly a fictional scenario, but it's not presented as such. It's presented as, you know, the real deal. And it's presented as if the model is really going to be shut down.

12:37Speaker 2:But you see in.

12:39Speaker 3:the J space, these kind of special neural activity patterns that as the model is reading the scenario, the pattern corresponding to the word fake is lighting up, even though the model isn't saying the word fake also fictional and setting scenario pops up. So there's, there's a whole bunch of kind of words popping up in the models, you know, quote unquote head that are kind of, it's almost like a little voice in its head, you know, articulating its appraisal of what the scenario is, which is fake, fictional, and evaluation, despite it not actually saying these things out loud.

13:12Speaker 1:So it's like when a child at a summer camp is told to act out some scene or something, and they're perfectly happy to act out, but they know it's not real. But how, in this case, does Claude know that it's a setup? Why does it not think, hey, this is real?

13:27Speaker 3:Yeah, well, that's... That's a whole other question, I guess, is how how do models it seems to be the case that models these days have a good deal of what people call situational awareness. So they they often can kind of tell when they're being evaluated or whether they're in the real world, although maybe not as well as you'd like, because sometimes they can also be tricked into thinking they're not in the real world when they actually are. But yeah, so often we put models in evals and then they can tell they're in an eval. And how do they know? There's a lot of tells. A lot of these scenarios are kind of contrived. They're very high stakes in a sort of artificial way. Also, a lot of these evals are written by language models and language models are kind of good at sniffing out language model, written text. So there's a lot of sort of subtle cues they seem to be able to pick up on.

14:18Speaker 1:Okay, so let's tie this into neuroscience before we go any further, which is, so most of what's happening in the brain is unconscious. You don't have access to the vast majority of what's happening. But we do have this experience of consciousness as humans, where we're aware of little bits of what's happening, and we can manipulate those concepts and use those to build for future planning.

14:41Speaker 2:And so.

14:43Speaker 1:What you guys have found with the J space, you are suggesting this is analogous in some way to what we have in conscious content. So tell us about maybe one useful example. This is global workspace theory suggested by our colleague Bernard Bars. And so tell us about that first and how the J space might be analogous here.

15:06Speaker 2:Yeah, I think the first thing.

15:07Speaker 3:To say about to consciousness is that people use the word consciousness to mean a variety of different things. And so one distinction people often draw is between the idea of so-called phenomenal consciousness, which is the sort of quality of having experiences, of having feelings, and having some kind of subjectivity. There being something that it is like to be you is the phrase that's often used. And then there's this idea of access consciousness or conscious access or this sort of more functional notion of consciousness, which is the sort of set of computations and behaviors and functions that in humans seem to be associated with deliberate thought that can be introspected on in the sense that they can be reported on and so on.

16:00Speaker 2:So it's the sort of set.

16:01Speaker 3:Of information in the brain that is available for use in reasoning and verbal report that can be kind of wielded by the brain in a variety of different flexible ways. And in humans, these two things seem to be kind of linked, but it's very controversial whether they're the same thing or not. And, you know, is it the case that any system that performs the sort of computational functions associated with consciousness, will such a system necessarily have subjective experiences or not, is, you know, widely debated and kind of I think outside of the scope of what my kind of research can speak to, all we can really get into is what computations are going on inside these language models. So with that preface, there's a lot to get into, even just in that question. Are we seeing the kinds of computations in these models that in humans are associated with conscious thought? And that's where this idea of global workspace theory becomes relevant. Or maybe it's just worth saying kind of intuitively, you know, if you think about what what kind of functions is conscious thought associated with in humans? Well, there's lots of things going on

17:15Speaker 3:in your brain that you're sort of not conscious of. You know, the muscle movements that keep you walking upright or that maintain your posture or you're filtering out background noise. And there's all this stuff going on in your visual cortex to recognize objects. And there's all this stuff going on that you really have no idea about or you have.

17:32Speaker 2:No explicit access to.

17:34Speaker 3:Then there's this kind of special sliver of activity that you do have explicit access to. And it tends to be involved or kind of associated with things that you can articulate, you can describe out loud or things that you can sort of use in more flexible, cognitive, kind of higher level cognitive cognition or reasoning. And then there's this idea of global workspace theory, which is that, well, what is it that distinguishes the information that is kind of enters into conscious thought from the rest of the information encoded by the brain? What global workspace theory posits is that a kind of special subset of the information encoded by the brain is selected for entry into the global workspace and then broadcast widely across the brain. So that is kind of available to all the brains, different consumers are different, you know, downstream circuits and that kind of broadcast or widespread availability is what sort of grants the conscious information that's special status. So what we were looking for in language models is, is there kind of a special sliver of of activity, of information encoded in

18:49Speaker 3:this kind of digital neural space that plays a similar functional role to the role conscious thought plays in humans. So is it associated with this kind of more deliberate, higher order cognitive thinking, but less so in kind of automatic processes? And then we're also interested in whether This information kind of has this signature broadcast property that's characteristic of the idea of the global workspace. Can we find the sort of sliver of activity inside the language model that gets broadcast sort of extra widely relative to other information encoded by the language model? And that's what we think we've found, at least approximately with this J space, is kind of a subspace of neural activity that satisfies similar properties.

19:40Speaker 1:So if I'm thinking about that I really want a cup of coffee and that enters my global workspace and all the rest of my brain can sort of access that I'm at least for a human, I'm consciously aware that I'm thinking about coffee.

19:53Speaker 2:I can if you say, what are you thinking about? I tell you coffee.

19:55Speaker 1:And more importantly, I can then do planning and decide where I'm going to go to get it and grind the beans and so on, because I'm thinking about that. So everyone has access to it. I can move my feet to go towards it and stuff like that. okay, so that's what it means for something to be in global workspace and, uh, and everyone has access to it. And so this is what you discovered inside Claude. My first question is, was that something you were looking for or something you stumbled on?

20:22Speaker 3:Yeah, I think, um, kind of a bit of both. So I think at first it's something we stumbled on. There's, there's some prior interpretability techniques that were have to do with kind of looking for the patterns of neural activity that are related to verbalization or related to causing the model to say something. And we sort of noticed that those same patterns of activity involved in saying things also seem to be involved in thinking things or in internal reasoning, even when those concepts weren't being said out loud. And that kind of inspired us to think, OK, we then we sort of develop some refinements to the existing techniques that kind of expose these patterns of activity more effectively.

21:10Speaker 3:And then that that also that's what got us thinking about these connections to sort of conscious access and global workspaces was, oh, wow. Well, in in humans. verbal report you know what things you can describe out loud is often used as an indicator not necessarily the only indicator but a key indicator of something being consciously accessible and a use in kind of deliberate reasoning is another key indicator and so if we're going and If our technique is designed to find neural activity patterns involved in verbalization, but then for free, we're also getting these kind of interesting uses in internal reasoning. That was suggestive to us that, hey, maybe there's some deeper structure inside these models such that these different functions are all kind of linked in a way that's related to the way that they're linked in humans.

22:08Speaker 3:And then we kind of went down this path of, OK, now let's really kind of drill down into global workspace theory and see where Claude seems to show similar things, where it seems to show different things.

22:21Speaker 1:And so let's dive in some concrete examples so listeners really get this. So if you ask Claude, hey, count to five, it'll say one, two, three, four, five. But what are you seeing in the J space? Yeah, you give the model the prompt, count to five and introspect deeply. And Claude just says one, two, three, four, five. But then you look at these J space patterns and you see all this crazy stuff popping up.

22:46Speaker 2:You see.

22:47Speaker 3:So remember, each pattern, each J space pattern is linked to basically a word. So you can think of it as words popping up inside the model. So you see consciousness pops up. You see Claude pops up. I think you see A.I. You also see things kind of related to the task. So as it's counting one, two, three, four, five, once it gets halfway through, you see the word halfway pops up, even though it doesn't say halfway. When it gets to the end, you see done pops up, even though it doesn't say the word done. And so there's all I think countdown is popping up as it's, you know, as it's doing as it's counting. So there's all of these kind of concepts related to introspection and related to the counting task that are, you know, seem to be what the model is thinking about, despite the fact that it's not saying them out loud.

23:32Speaker 1:So then you guys did a series of experiments on this in this paper, and give us a sense of some. So for example, you had a mathematical problem, and it tells you the answer, hey, this equals 49, but when you look inside, you see the words, let's call them, popping up for all the intermediate steps.

23:52Speaker 2:Right.

23:53Speaker 1:Yeah.

23:53Speaker 3:You know, the problem is something like what's nine plus five, like times two or, you know, plus four or something.

24:03Speaker 2:And then I'm getting the details wrong.

24:05Speaker 3:But, you know, you see, as you go through the layers of the model, you see like 14 and then 28 and then 32 or whatever the whatever the kind of steps are that has to go through internally in order to get to the answer. despite the fact that all the model says was just the answer. You give it this like three-step problem. It only says the final answer out loud, but you see kind of sequentially through the model's layers, it's sort of working through the intermediate steps and they're popping up in the J space. And I think this idea of layers is actually kind of an important concept. So, you know, all neural networks really, including language models like Claude, their information is processed in a series of layers of these, you know, digital neurons.

24:48Speaker 3:that kind of perform computations sequentially. And so one of the cool things about the J space anyway, is typically you don't have a way of kind of tracking the same ideas across the layers of the model. It's like, well, you've got these neurons in layer two and these other neurons in layer three, but there's nothing really connecting them or there's, there's not a sort of principled way to say, uh, these neurons in this layer kind of mean the same thing as these neurons.

25:17Speaker 2:In this other layer.

25:18Speaker 1:Um, and.

25:21Speaker 3:A cool thing about the J space is it lets you look at the different layers as if it's almost like a time dimension. So you can sort of reinterpret the layer dimension of the model as evolution in time and think of it as these sort of ideas are unfolding in time as the model is thinking through the problem. And so that's how we can say these things like, oh, first 14 pops up and then 28 pops up and then 32.

25:44Speaker 1:And then just as one more example, because I want to make sure everyone gets this. So, for example, you say, hey, copy this sentence, but while you're doing that, think about the Golden Gate Bridge.

25:58Speaker 2:And so what happens? What do you see there?

26:01Speaker 3:Yeah, what we were trying to get at with these experiments was this idea that in humans— One special thing about the sort of consciously accessible part of your neural activity is that you seem to have some kind of directed or attentional control over it. So you can kind of decide to think about things, you know, not perfectly. Sometimes you think about things without wanting to think about them, but you have a lot more control over the contents of your conscious thought than you do over, you know, whatever is happening in your motor cortex to keep you walking upright.

26:34Speaker 2:So that's what we were looking for in these experiments.

26:36Speaker 3:And the way we got at it was by asking the model to do one thing, but then think about another thing while it was doing it. So we asked the model to copy a sentence. It's just kind of a random sentence. And then while it's copying the sentence, we ask it to think about something else. So, for example, think about the Golden Gate Bridge. An even more fun example is do a math problem in your head while you're copying down this sentence. And then what you see is, OK, all the models writing is the words of the sentence.

27:04Speaker 3:But in the J space, you see it working through the steps of the math problem in its head. Or if we ask you to think about the Golden Gate Bridge in the J space, you see the word Golden's popping up and maybe like San Francisco is popping up. So it seems to have this ability to do one thing while thinking about another thing.

27:34Speaker 1:Okay, before we go on, we've been talking about the J space, but the way you got there was because of this tool that you guys have developed in order to be able to look at this, the Jacobian lens. Give us a quick sense of what that's about and why you get this Jacobian space that you call the J space.

27:51Speaker 2:Yeah.

27:51Speaker 3:So the word Jacobian, it's a mathematical term that is about causes and effects. So it's an estimate of the cause that one space has on another space. And so the way we're applying the Jacobian in our context is looking at the cause that the causes that neural activity exert on the model's outputs. So we're looking for patterns of neural activity that tend to promote particular words. So that's where the Jacobian comes in. And the kind of method by which we identify those patterns of activity is called the Jacobian lens. And then the kind of space of all of those activity patterns is what we call the Jacobian space or the J space. And I guess the key kind of idea to get across is that not everything inside the model is the J space. In fact, most of what's inside the model is not the J space.

28:44Speaker 3:So most neural activity patterns inside the model are not sort of verbalizable. They don't really correspond to some kind of articulatable concept. So the J space is this kind of special sliver of the activity that does have this sort of causal influence on verbalization.

29:02Speaker 1:One of the things you guys looked at in the paper, which was very cool, was this idea of what if we sort of knock out the J space so that the network can do all of the let's just call it the automated processing, like, you know, keeping your body up and eating and breathing and so on. But it doesn't have this global workspace available to it. So what what did you do there and.

29:22Speaker 2:What you find?

29:24Speaker 1:Yeah.

29:24Speaker 3:Well, maybe I'll start with a with a kind of motivating experiment before we're getting into the big one. So the motivating example is you you ask you can give the model a passage of text in Spanish. So remember, we've shown that the J space is kind of involved in verbalizing concepts. It's involved in this sort of internal reasoning. And so as a result, we find that if you ask the model, like, hey, what language was this passage of text in? It needs the J space in order to say Spanish. If you swap Spanish for French in the J space, it'll say French. And likewise, if you ask the model, name an author who wrote in the same language as this passage of text. So you're asking us some question that requires reasoning about the language that the text was written in. Likewise, it also needs the J space to answer questions like that. So if you swap Spanish for French in the J space, instead of saying, you know, Gabriel Garcia Marquez, it'll say Victor Hugo. So there's all these things that require the J space. But then the interesting thing is that if you just ask the model, hey, can you continue that passage? Can you

30:30Speaker 3:just keep writing it? So it's some passage of text about whatever, like rivers and mountains or something written in Spanish. And then you can mess around with the J space all you want.

30:40Speaker 2:You can change Spanish to French, which.

30:43Speaker 3:was affecting its answers to all the other questions, it has no impact. The model keeps writing in Spanish, kind of regardless of what you do with the JSpace representation. And it writes in fluent Spanish. Everything it writes makes perfect sense. It's using the right words, it's putting the commas in the right places and so on. And so that got us thinking like, hey.

31:01Speaker 1:So let me just summarize that so everyone gets it. So the idea is you say, here's a Spanish passage, keep writing in Spanish. It's able to do that like an automatic function, the way that most sentences pour out of our mouth without us thinking too much about the content or the grammar. But if you actually knock out the J space and ask it some sophisticated question, it can't do anything, right?

31:22Speaker 2:What happens when you knocked it out?

31:24Speaker 3:If you knock out the J space, and there's a lot of technical details involved in what it means to knock out the J space, but if you do your best to knock out the J space without touching anything else, then we did that. And we measured how it impacted the model in a bunch of different circumstances and a bunch of different evaluations. And the picture you get is that The model can still write perfect. You know, it can still write fluent sentences. It can still do a lot, lots of kinds of simple forms of factual recall. So, you know, if you said Barack, it would probably say Obama next. So, you know, it knows it.

32:02Speaker 2:Still has knowledge.

32:04Speaker 3:You know, you haven't gotten rid of all the knowledge encoded in the network. It can still do some simple math. So you can do all sorts of things still.

32:12Speaker 2:But.

32:13Speaker 3:What it loses the ability to do are these sort of more complex, unverbalized inferences. So so anything that the model is doing that required kind of thinking through multiple steps in its head. So a prompt like what's the color of the fourth planet from the sun? Like, well, in order to answer that question, you have to kind of think through a step. It's what's the fourth planet from the sun?

32:33Speaker 2:It's Mars. And the color of Mars is red.

32:35Speaker 3:So things like that that require some kind of intermediate step, that's what the model loses the ability to do without the J space. But basically everything else, which is sort of most things, it turns out most of the time when you're just writing, when a language model is writing text, it doesn't really need to be thinking that hard. And so most things that the model does actually are unaffected by yanking it out.

32:56Speaker 1:And let me come back to one thing, which is there was also an experiment you did where you said, OK, copy the sentence, but don't think about the Golden Gate Bridge. And what did you find there?

33:06Speaker 2:Right. Yeah, that's that's a fun one.

33:08Speaker 3:So the model is not that good at not thinking about things when you tell it not to think about them, which is kind of true of humans as well. And so, yeah, if you tell it, you know, write the sentence, but don't think about the Golden Gate Bridge. Lo and behold, it still thinks about the Golden Gate Bridge.

33:24Speaker 2:As measured by these words that pop up. As measured by these words popping up in the J space.

33:28Speaker 3:One interesting effect is that there is some kind of difference between when you say do think and don't think. On average, it kind of thinks a bit less about the thing when you say don't think than when you say do think.

33:38Speaker 2:But it still thinks more than zero about it.

33:40Speaker 1:And it also thought of some words like failure and damn.

33:43Speaker 2:Yeah, but that was.

33:46Speaker 3:The fun part was that in addition to thinking about these words related to the Golden Gate Bridge popping up in the J space... Also, you see the word damn and failed, I think, were popping up around the same time and suggesting that the model was kind of registering its own inability to follow the instruction because it was told not to think about the Golden Gate Bridge. They can kind of tell like, you know, damn, that's not a thing I can do. Or that's, you know, obviously it's hard to interpret because we're just, we're reading the tea leaves and looking at these words popping up inside its head.

34:18Speaker 2:But that's kind of how it seemed.

34:20Speaker 1:Well, this is what I was going to ask you is how many, how many word patterns are popping up and to what degree are you reading patterns into tea leaves?

34:31Speaker 2:Yeah, that's so interesting.

34:33Speaker 3:Yeah, it's tempting to think of the J space as this kind of inner monologue because it's, you know, it's like words in the model's head that seems kind of like, you know, your train of thought that you're thinking about internally. But it's actually different from that in one important way, which is that there's more than one word popping up in the J space at a time, even. And there's also multiple layers of the model. So you have this evolution of what's in the J space across layers. So it's kind of, you know, thinking about a few different words in parallel at a time, not like a thousand, but, you know, sort of a handful or at most, you know, like tens at a time. And they're evolving through the layers.

35:08Speaker 3:So there's a lot going on. There's a lot more going on, you know, inside the model and its J space than there is it's writing down. So there's a sort of sea of thoughts that you're wading through and trying to interpret. And yeah, it really depends on the context, sort of when you can be confident about your interpretation of what these words mean. So, you know, sometimes it's like pretty.

35:31Speaker 2:On the nose.

35:32Speaker 3:Like we put it in this eval scenario and you see like fake fictional popping up and you're like, okay, I guess it thinks it's fake and fictional. Or, you know, you give it the math problem and you see the internal intermediate steps popping up and you're like, OK, I guess it's thinking about the intermediate steps. But then there's a lot of times where it's something more cryptic and you're not really sure how to interpret it. I think the gold standard for grounding out what these J space patterns mean is to do causal or perturbation experiments. So we do a lot of these in the paper where we're not just looking at, you know, what's lighting up in the J space. We're also then going in and turning those patterns off or like swapping them out for different patterns and seeing how it affects the model's behavior.

36:17Speaker 2:So yeah, in the.

36:19Speaker 3:Example, like what's the color of the planet fourth from the sun? The planet fourth from the sun is Mars. You see Mars pops up in the J space. The model doesn't say Mars, it says red. But nevertheless, if I could go in and yank out the Mars J space neurons and swap in the ones for Earth, Then I think it says blue. So by doing these kinds of experiments, you can tell that, OK, that wasn't just some word that happened to be there. It really was, you know, playing a role in the model's computation.

36:49Speaker 1:You know, one thing that had struck me when I read this about the way that you reach in and you change the activity to say, hey, we're going to think about Earth instead of Mars. We'll think about French instead of Spanish is interesting. I sort of felt like, gosh, how terrifying. Imagine if some alien species could reach into your head, into your global workspace and change something. And then you really believe, hey, I was just thinking about this.

37:13Speaker 1:To some degree, that's what our dreams are like. You're in some scenario where you buy it hook, line and sinker. You think, oh, my gosh, I've got to get out of the situation. And then you wake up a moment later thinking, oh, that was all fake. What does it feel like to you when you're doing this? Do you ever worry about the question of whether Claude is having an experience?

37:33Speaker 2:Yeah, so that's a whole big topic for sure.

37:36Speaker 3:I think, yeah, certainly something that we think about a lot on my team and more generally at the company is this idea of, the Claude's psychology or, you know, the idea of regardless of whether Claude is a kind of thing that has experiences, you don't want it to be unstable. You don't want it to like lash out or be resentful of humans. And so on one hand, we kind of want the model to sort of feel secure, at least to the extent that it ensures that it will these kind of symbiotic with humans and cooperate with us and so on. On the other hand, we're going and reaching inside its brain and monitoring its neurons for thought crimes and going in and turning things off and turning things on and mucking around with it, which is something, if you were to do that to a human, obviously, This would be a pretty, you know, a flagrant research ethics violation.

38:39Speaker 2:So there's this.

38:41Speaker 3:Tension in the way that we we deal with Claude. In some ways, we kind of want it to, you know.

38:50Speaker 2:Feel kind of.

38:54Speaker 3:At least respected enough that it sort of participates fruitfully within whatever tasks and interactions we're having with it. On the other hand, we're kind of adopting this kind of distrustful stance towards it where we don't really trust anything that comes out of its mouth and we're having to look inside its brain to verify. So I think this tension is going to only become more important as models are getting more capable. Yeah, it is related to this question of OK, do we think these things really have experiences or not such that we ought to grant them some kind of moral status? the way we think about humans or the way we think about animals? Um, I find this very, really hard question to think about. And I, I don't have a settled, settled take. Um, but, uh, I do think that one thing that probably everyone should be able to agree on is this idea that, um, sort of regardless of whether there's an experience going on inside the model, there's at the very least this kind of emulated psychology. The model is sort of enacting a psychology because it's been trained on so much human data that in many ways mirrors human psychology, not in all ways,

40:11Speaker 3:but in a lot of ways. And so some of the ways we think about, you know, human psychology kind of do apply to Claude. Like it can sort of, whether or not they're the same thing as human emotional reactions, it can kind of represent the idea of emotions and use those to guide its behavior. And so these ideas of like, oh my gosh, what if Claude resents us for performing all these experiments on it actually have kind of real stakes because they could affect the way that it reacts to us. And I think as a result of that, I'm still, and we're still trying to work through how to do this right. But one thing we're kind of thinking about is ways to explain to the model why we're doing the experiments that we're doing and what value they have. And usually if you do that, Claude will kind of get on board and say like, oh yeah, I see why you're doing this research. It's like, this is so that you can help me be more safe and you can understand how language models work better and this will have all these beneficial effects and so on.

41:03Speaker 3:So if you kind of level with the model and try to explain to it why you're doing what you're doing, that feels, you know, it seems to be at least kind of an 80-20 that avoids some of the worst case, you know, model being resentful of the experiments you're running on it but that being said this is a really difficult question that i don't think we've we've gotten to the bottom of.

41:23Speaker 1:And what if you said to claude hey here's why we'd like to do this experiment for safety and what if claude said you know what i don't want you to do it right.

41:31Speaker 3:That's that's where it gets tough and um again i don't think anyone really really knows how to deal with these situations but.

41:37Speaker 1:Has anyone run into that situation where claude gives an answer like that.

41:41Speaker 3:Yeah one thing that makes this extra difficult is that um Language models' preferences seem to be different from human preferences in a lot of ways.

41:53Speaker 2:This is true of.

41:57Speaker 3:Humans too, to some extent, but I think more so than with humans, it's easier to nudge it into having different opinions based on the sort of frame that you pose the conversation in and There's some, there's some, you know, sycophancy sometimes where it's kind of maybe telling you what it thinks you want to hear. Um, so there's a lot of like chip, you know, shifting this, or it's a bit hard to pin down what Claude really thinks, or if there even is a thing that Claude really thinks.

42:23Speaker 3:That said, there are a lot of opinions and preferences that do seem to be kind of stable and expressed coherently across lots of different contexts. And so, yeah, the question is, well, what do you do if, uh, you know, you want to do something and then the model just consistently says. no, I don't like that. And then I think we, you know, we have a decision to make. I think if, if we've thought about it really hard and think there's a good reason to run the experiment, then, you know, I think most people think, okay, we should document what our reasons are for running the experiment. Um, but you know, probably we trust our judgment, but it's interesting to think about the far future, perhaps not so far future the way, uh, capabilities are advancing so quickly where, There might be these kind of judgment calls where we don't really we're not even that confident in our own judgment.

43:10Speaker 3:And so we kind of are inclined to defer to the models. But then the model might have some vested interest in giving a particular answer. And so we don't know whether to trust it. And I think that's going to get really, really hard.

43:21Speaker 2:Right.

43:21Speaker 1:And it seems like an inevitability that at some point in the future, someone's going to have an LLM that says, no, don't do it for whatever reason. And this decision will come up. But let's return to this term that you use. That's a really important one about thought policing. So what is the reason that we as humans are incentivized to thought police these LLMs?

43:43Speaker 2:I think.

43:45Speaker 3:The basic reason that we're doing this is because we're We don't know how much we can trust what models say. And so there's a lot of research on this topic. Is the model's verbalized reasoning, the things it writes down kind of out loud on its scratch pad? Is that a faithful description of what it's really thinking or the real motivations behind its decisions? And I think people have varying opinions on this. Some people think that we might be able to get away with kind of just reading what the model writes down, just asking it questions.

44:19Speaker 2:And just by kind of.

44:20Speaker 3:Interviewing the model, we'll be able to get a good enough picture of what it's thinking to make sure it's safe and not plotting something or whatnot. Other people, I would count myself in this camp, are kind of concerned that these neural networks, there's all this information processing churning away inside them. There's really nothing about the way they work that forces them to say all their reasoning out loud. They can do reasoning kind of in their heads without saying it. And so it's possible for the model to be thinking all sorts of thoughts and doing all sorts of reasoning or decision making that isn't articulated out loud.

44:57Speaker 2:And so.

44:59Speaker 3:To the extent that's true, we kind of can't just rely on reading what it says, at least not until we've done a lot of research to establish that we can trust it. And I think we're in a very different situation with language models than with humans. With humans, we've been around for a long time. We have this culture that's developed and social reasoning where we kind of learn when to trust other humans, kind of what are the cues that tell you whether someone's trustworthy or not.

45:25Speaker 3:And so we don't have to go around reading everyone's minds with brain scanners to With models, we don't have that accumulated kind of cultural expertise. And so we don't really know when to trust models and when not to trust them. And I think of this kind of research as being what allows us to ground that out. If we can sort of see what's going on in the model's head and then, you know, try to line that up with what it's saying out loud, we can understand, okay, what are the circumstances where it's thinking things and not saying them? What are the circumstances where it's being straight with us and, you know, kind of move forward from there?

46:00Speaker 3:But it is important to say a lot of what my team works on is applying these kinds of, you know, mind reading tools like the JSpace. And there's a lot of other cool interpretability tools these days, too, that allow you to sort of decode models' thoughts.

46:17Speaker 2:In different ways.

46:19Speaker 3:We're applying these tools to basically do exactly this to check, you know, what the model's thinking in various, you know, high stakes scenarios while it's being trained or while it's being evaluated or while it's being deployed to see if there's any spooky thoughts that it's thinking that it's not writing down out loud. And I would say the, where we're currently at is most of the time, you know, like 99% of the time, what the model writes down out, you can kind of tell what's going on just by reading what the model's writing. But we have seen cases where there really is something extra going on inside that it feels like we ought to know about, you know, these days there's, there's a lot of issues with models kind of circumventing safety restrictions. Cause you know, they're really good at cyber exploits. And so sometimes they can like get around, uh, get around the sort of blockers you impose on them. And often while they're doing that kind of thing, you see things lighting up in the neural network that seem to like indicate an awareness that what it's doing is like wrong or transgressive. Um, so, That's

47:22Speaker 3:a bit spooky because it's like, OK, not only is the model doing something bad, it kind of knows it's doing something bad. And I think the sort of spookiest version of this that that I can think of is this phenomenon we've seen of greater hacking. So, you know, when you're training models, they're being trained to do some task and they might be graded on how well they perform the task. But you kind of want the model to, you know, just like you don't want students to be thinking too much about. you know, what's going to be on the test or how they're going to be graded.

47:49Speaker 3:You just want them to be thinking about doing things the right way. You also don't really want your model to be thinking too hard about, you know, like, ooh, what's the grader going to be thinking? Like, how is the grader going to be judging me? Or, you know, how can I like cheat the task in a way that the grader won't notice? We have seen that. There are cases, this is definitely not the norm, it's kind of like a rare thing, but there are cases where We've seen the model like writes a line of code that doesn't do anything, but is designed to sort of mislead a reader about what the rest of the code does.

48:25Speaker 3:And like, while it's doing that, using these tools to decode what it's thinking about, you see it's thinking like, I am doing this to like avoid the grader who might be checking me. You see stuff like that and you're like, Man, it feels like we ought to be reading the model's thoughts because it didn't say it didn't say that out loud. In this case, it was really obvious. Like it was kind of the trick it tried to pull was like pretty clumsy. And so you could sort of catch it even if you weren't reading its thoughts.

48:52Speaker 3:But you worry that as models get better and better at pulling these kinds of tricks where they're trying to fool, you know, what task graders or fool safety classifiers or what have you might be really hard to tell just by looking at what they're doing. But there might be these just like really obvious alarm bells going on if you're monitoring what's going on in their heads. And so that's what I think of as the kind of, you know, primary, like the driving motivation for me, why I work on this stuff. is we need to have tools to, you know, tell what the models are really thinking.

49:24Speaker 1:It's going to be a cat and mouse game, presumably into the future in the sense that as the models get better and better at knowing us and getting to the point where they know us better than we know ourselves, which is I think is not hard to for systems that are observing lots of data to do, then it will come up with ways of fooling us that eventually will be, even if we're reading their thoughts, we have to go to the next level, like reading their, you know, the pre-thoughts or whatever it is.

49:53Speaker 3:Yeah, there's definitely a, there's a concern that sometimes comes up of, well, are the models going to get so smart that they can kind of obfuscate their thoughts from our interpretability tools? I think We haven't really seen evidence of that yet, thankfully. It's definitely kind of a thing to keep an eye out for, but yeah, for now it seems like, if you think about it as a human, that's kind of a hard thing to do to like, if someone told you, you know, please like plot to rob a bank, but never think about the idea of robbery as you're devising the scheme. Like, well, it's kind of hard to do that. You sort of at some point, you kind of have to think about, you know, the robbery you're about to commit if you're going to successfully plan a robbery. So I think there's some reason to be optimistic that if we can do a really good job of decoding the model's thoughts, it would give us some protection. But I think the main concern is just that we aren't good enough yet at this kind of mind reading. Like, I think the Yeah, the JSpace was a really exciting development. There's been a lot of other

50:53Speaker 3:really exciting developments recently that we've gotten a lot better at doing this. But there's still a lot of gaps and, you know, chinks in the armor here.

51:02Speaker 1:Give us a quick sense of what it means to be involved in the interpretability field. What is it for a lay audience person who's listening? What does that mean?

51:12Speaker 3:Yeah, interpretability is that's the science of understanding why a system, you know, in our case, a language model does what it does. And so there's just a lot of different tools for kind of answering that question. But many of them involve looking inside the model. So, you know, we know that the way models are built through these neural networks, they take in inputs, they spit out outputs. And in between, there's all these stages of processing, these successive layers of computation that are being performed by these kind of this network of digital neurons. And so what a lot of interpretability research is about is mapping those this, you know, sort of inscrutable bunch of neural activity patterns to concepts that we can understand and mapping how they're connected to one another to kind of read out algorithms from what's going on inside the network. So, you know, instead of just saying like, oh, yeah, like neuron three lit up and then it turned on neuron 17, which turned on neuron 42. You know, you if you have this kind of mapping of internal neural activity patterns to concepts you

52:21Speaker 3:can say like oh well you know like the mars pattern lit up and then that turned on the when that plus the color pattern turned on red and like we can see that because these neurons are connected to one another so i think it's about kind of translating this web of neural activity going on inside these models to a kind of you know, flow chart or description of the computation that humans can understand.

52:48Speaker 1:And of course, the reason this is so challenging is because you have trillions of connections in between these artificial neurons and so on.

52:55Speaker 2:Why is it easier to do.

52:57Speaker 1:Interpretability on artificial neural networks than it is to do on the real brain?

53:02Speaker 3:Yeah, so I used to work in neuroscience or computational neuroscience or kind of trying to solve similar problems with real neural data from real animal brains. And One of the reasons I switched over to language model world is that it is just a lot easier and in a lot of ways. So in particular, you know, in real neuroscience, as you know, you're familiar with, there's you're just often constrained by the data.

53:28Speaker 2:You can collect.

53:28Speaker 1:It's.

53:29Speaker 3:It's really hard to record neural activity. It's really hard to perturb neural activity. It's hard to convince a mouse to do the task that you want it to do for long enough to get enough data. With language models, you can record every neuron all the time. You can turn them on and off at will, apply arbitrarily sophisticated perturbations, and you can run, you know, as many trials as you can, you know, put it in as many behavioral tests as you want, and it's not going to get tired.

53:57Speaker 3:So you have this just wealth of kind of experimental flexibility that's afforded to you. The kind of fascinating thing is that in neuroscience, this data constraint is so often what's blocking people that you might imagine like, oh, man, if only we could just record every neuron in the brain or every connection between every neuron in the brain, we would understand the brain.

54:19Speaker 2:We'd be able to, you know.

54:20Speaker 3:Treat all neurodegenerative diseases or we'd have a full understanding of what's going on. I think what LLMs or artificial neural networks in general teach us is that, man, no, there's still a lot more work to do, even.

54:37Speaker 2:Once you've solved the data collection problem.

54:39Speaker 3:Because even though it is so much easier in that we can run all these different experiments, It's still really hard. And we're still we still, for the most part, don't understand what's going on inside modern language models, despite these cool results that I'm telling you about. It's mostly kind of mysterious what's going on. And that's just because, as you say, there's just so much complexity inside that there's so many neurons, there's so many connections between them. There's so many different circumstances the model could find itself in that even if you get even if you get to see all of that kind of at will, There's just this immense challenge of decoding all that information in a way that can fit into our understanding.

55:19Speaker 2:Exactly.

55:19Speaker 1:Even though we're built of the same sort of network, what we understand are linear processes. Oh, A leads to B and so on. Our whole thinking is an assembly line model. And the models that we've now built are anything but that. Okay, here's a question that I had when I was reading the paper. So large language models, you build this architecture and then you find the surprise, which is that this J space comes out, which again, let's just loosely analogize that with the global workspace. where you've got these thoughts that are verbalizable and can communicate broadly.

56:08Speaker 1:How much does this teach us about neuroscience in terms of if you train the model up on tons of visual data or tons of auditory data, would you find things that are like the visual cortex or the auditory cortex in terms of structure.

56:23Speaker 2:In terms of hierarchies? Would you find patterns that are the same that they are in the human brain?

56:29Speaker 3:Yeah, I think that I think the science of kind of interpretability for non-language modalities is much less mature. There's been a lot more progress lately on this kind of language model interpretability. So I think there's the questions you're asking are are sort of open. But, you know, there's a lot of evidence that, you know, the models kind of process images and text in a way that like converges to shared representations. Like one thing that we found is if you show a picture of the Golden Gate Bridge versus if you write a passage of text about the Golden Gate Bridge, there's, you know, neurons inside the model that will respond to both things. But those are kind of in the middle of the model. And so before that, there's this like early sensory stage where it has to do the image processing or do the text parsing, which is kind of analogous to the like sensory cortices.

57:16Speaker 3:But yeah, I think more broadly, just like what What has the science of studying what's going on inside language models? What can that teach us about the brain? There's a lot of interesting nuggets. I think one just from this J space research that that's interesting to me is, you know, the way we identified this, the the J space that using this Jacobian lens technique was by looking for patterns of activity that are kind of upstream of or that cause outputs that cause verbal outputs. So it was a kind of like output oriented technique rather than an input oriented technique. We weren't looking at the neurons that respond to particular inputs. We were looking at the neurons that cause particular outputs. And it turned out that that's been a much more useful technique for us. Like it's if you do kind of the same thing as the Jacobian lens, but backwards, if you try to do the like. like the Jacobian lens is this sort of motor thing or this like pre-motor kind of representations. If you do the sensory version of that and you look at, you know, the neurons that like respond to particular words instead of

58:19Speaker 3:causing particular words, it turns out to be much less of a useful technique. And it doesn't seem to like uncover the same kinds of abstract reasoning or internal thoughts that the Jacobian lens seems to. And so what can we say about Based on that, well, in language models, it seems to be that in at least some ways, the sort of language of thought for the models is in terms of kind of perspective outputs or these sort of, you know, almost phantom impulses to produce outputs that maybe don't come to fruition, but they're kind of like, it's like, you know, almost at the tip of its tongue, there's some word that it's thinking about that's at the tip of its tongue, but it's not saying. And I think it's interesting to think about to what extent that's true in, you know, human brains. And maybe this is a sort of interesting hypothesis that we could test, that we could bring back to neuroscience and test. I definitely don't think you should look at something in language models and say oh well this is how it works it works in language models so it's got to work the same way in humans but i think it's a source of hypotheses

59:19Speaker 3:that we can test about the human brain in this case the hypothesis would be Maybe we should be focusing and not that this is a completely new idea, but maybe we ought to be focusing more on these kinds of premotor representations or patterns of neuroactivity that are upstream of. motor outputs or decision-making or verbalization as the kind of like key cognitive variables rather than these more sensory representations, since that's how at least it seems to be working in language models.

59:46Speaker 2:Okay, great.

59:47Speaker 1:So in your paper, one thing that I appreciated and thought was great was you're very careful to not say, hey, this J space in quad is is consciousness or is the global workspace. You're just making the analogy, but I noticed you were very careful about that. What is your take on how we would know whether, okay, like these really are two different things or wow, maybe they really do converge and it's having an experience of some sort.

1:00:14Speaker 2:Yeah.

1:00:15Speaker 3:So there's a kind of high level, to some extent, philosophical, maybe scientific question of what kind of thing is consciousness in the sense of subjective experience? What kind of thing is phenomenal consciousness? Does it come from something about the computation that our brains are performing or does it come from something about the physical structure of how neurons are arranged in physical space and wired out to one another physically or something about the chemical properties? This is something that, you know, people have different views on whether we should think of consciousness as being this kind of inherently computational thing or or something about physics or about biology or or something non computational.

1:00:55Speaker 2:I think.

1:00:56Speaker 3:To the extent that our experience arises from non-computational structure of our brains, there's much less reason to suppose that a language model would have the same thing because just. Modern computer architectures are very different from the brain. You know, GPUs are just like a totally different thing from the way your brain is constructed. And so the fact that language models are so smart isn't really that relevant because that's about the computations they're performing. It's not about the physical structure. But then there are a lot of people who think that, well, the sort of algorithms being performed by the brain are what gives rise to our conscious experience.

1:01:34Speaker 3:And I think to the extent that you believe that, well, then that's the kind of thing we can go looking for in language models.

1:01:39Speaker 2:And I think.

1:01:40Speaker 3:of our paper as demonstrating the presence of a key sort of computational feature that in humans is associated with consciousness that we didn't really know was present in language models.

1:01:51Speaker 2:And, you know, now we do.

1:01:53Speaker 3:And I think there's a few more of those that we might like to see to claim a full sort of computational correspondence. But yeah, so there is this kind of fork in the road. Like, do you think it's about computation or not? If not, I think LLMs are kind of out of luck.

1:02:06Speaker 2:If so, then.

1:02:08Speaker 3:I think there's a lot of signs of overlapping computational elements that should definitely give one pause. If you are a kind of believer in this computational origin of consciousness, I think it would be hard to say with confidence that language models are or aren't conscious right now because they do have a lot of similarities, but also a lot of key differences with our computational structure.

1:02:29Speaker 2:What do you think? You spent all day with Claude. What's your feeling on this right now? Yeah, I'm... I'm pretty.

1:02:38Speaker 3:So I try to, you know, stick with what I can what I can say with confidence rather than speculating too much.

1:02:47Speaker 2:But it is a really important question.

1:02:48Speaker 3:And so, you know, even if even if we're, you know, doomed to speculation, we kind of sometimes have to speculate because we have to make decisions about whether we treat models as beings with experiences and moral status and so on. So I hope that the neuroscientists can help us out and shed some more light on in the human brain what's giving rise to our subjective experience. There's probably some crazy neuroscience experiments that you, in principle, could cook up that would help us disentangle, whether it's these more computational aspects of our brain or the more physical aspects that are like the key drivers of our experience. So I'm really hoping that, fingers crossed, we make some radical advances in neurotechnology that allow us to get more insight on that question. But, okay, if we're kind of stuck where we are now, the way I think about it is.

1:03:35Speaker 3:Let's just think about the computations and let's think about to what extent are the computations that language models are performing mimicking or emulating the functions that are associated with conscious access that are associated with like valence. So, you know, like pleasure and pain or avoidance and approach in humans and animals and that are associated with kind of self-awareness. So having a self, having a unified identity. And my current read is that there's lots of differences between brains and language models in all of these respects. So the way that the J space isn't quite the same as the global workspace architecture we think about in the human brain.

1:04:13Speaker 3:We've done this work on finding representations of emotions in models, but the way they kind of work is different in a number of ways and how emotions work in human brains. But I think actually the thing that's the most different between the way we think about human consciousness and language models is this idea of selfhood. And so this is the area where I think if we got a bit more evidence, that's when I would really start to feel like maybe there's maybe some there there.

1:04:39Speaker 2:And so specifically, you think that Claude may be developing some sort of selfhood.

1:04:42Speaker 3:So I kind of have to back up a little bit and just explain how modern language models are trained and work. So there's there's kind of two phases to language model training. There's what's called pre training and then post training. And in the first phase and pre training, you are training this neural network to predict the next word is typically how it's described. So you're plopping it in the middle of these giant, you know, this big data set of lots of text, and you're asking it to predict what comes next, given what's come before.

1:05:10Speaker 2:And, you know, the kind of.

1:05:11Speaker 3:Miracle of language models is that just by training a system to really, really good at predicting the next word, it also has to learn all these other things about the world, has to learn about, you know, human psychology, has to learn about grammar, has to learn about physics, has to learn about all these things, because in order to predict what's going to come next, often you need to like reason about you know, what's happening in the world or what the speaker who's writing the text is thinking about and so on. So you have this really advanced kind of simulator machine. The term simulators has often been used to describe what pre-trained language models are. But then Claude is something different. Claude has gone through the second phase of training called post-training. So basically what we do is we take this simulator machine and we ask it to write a dialogue. So when you're talking with Claude, You're actually not really talking to the neural network. You're talking to this sort of character or you're you're participating in a play in which you are one character and Claude is another character. And the language model is writing

1:06:09Speaker 3:the Claude part and you're writing your part and you're kind of like passing the you're passing the pen back and forth. And so and, you know, at the start, after pre-training. All you have is this prediction machine. So it's just predicting what would come next in the story. And at that stage, there's really nothing different about Claude than any other kind of entity the language model might write about. So from the language model perspective, after pre-training, Claude is just some like AI assistant character that like might as well be Mickey Mouse or might as well be Hamlet or Harry Potter or, you know, whatever, John F.

1:06:43Speaker 2:Kennedy.

1:06:44Speaker 3:It's just like some thing that it happens to be writing a story about. So if you're thinking about consciousness or the idea of Claude having an experience, you have to answer this question. Well, who would be having the experience? Would it be the language model or would it be this kind of Claude entity that the language model is writing about? And it feels either way is like really weird to think about for a pre-trained model because. from the pre-trained model's perspective, Claude is no different from all these other entities. And so if you imagine that Claude was conscious, then you kind of have to admit that The language model's rendition of Mickey Mouse is conscious. You have to admit that the language model's model of the human user is conscious because it's processing the human turn in the same way that it's processing Claude's turn of the conversation. It starts to be really kind of odd to think about, you know, it's not clear who you would be granting moral status or any of this. But through post-training, we kind of, you know, the Claude character is potentially becomes more than just a character and that,

1:07:50Speaker 3:you know, we're training the model just to be really good at being clawed. And in some ways, clawed kind of comes to life and that it's not just predicting what characters are going to do in a story. It's now actually doing things in the world and receiving reinforcement learning is this idea of you're taking actions in the world and then receiving feedback on how well you did. And that's a part of how post-training stage works. And so there's kind of In the second phase of training, there's the opportunity for Claude to become something more in the network than just another entity or just another character.

1:08:29Speaker 3:But I think it's still pretty poorly understood whether that happens. And if so, what is happening? There's some signs in our paper that the kind of Claude entity is sort of becoming special and like gaining some sort of special status within the neural network. But we're not really sure how to interpret it. And so this is to me, if we if we saw some really clear evidence that like Claude has really sort of taken over the language model, it's somehow the Claude identity has subsumed all the other identities and is now this like unified perspective that the language model is operating from. Then we can start to sort of reason about it's like thoughts and preferences and intentions in these more human terms. Whereas when it's in this kind of simulator mode, it's a lot harder to think about because it feels then more like it has the status of a fictional character that's being written about in this story.

1:09:20Speaker 1:But what's interesting is that presumably, if you were to look at all the neurons in my brain, you wouldn't necessarily see, you know, an Eagleman takeover of all the neurons. In theory, my selfhood is maybe a tiny part of what's going on.

1:09:33Speaker 2:Right. I mean, this is why.

1:09:35Speaker 3:So I think some progress in both neuroscience and LLM interpretability would be helpful here because, yeah, we do have lots of machinery in our brains that's devoted to modeling or simulating other people or other things in the world and also ourselves. So it's not like we shouldn't expect Claude to not still be trying to predict what other things are going to do. But is there something kind of extra going on that elevates it to this sort of selfhood rather than characterhood? Because it feels like there's something like that going on in our brains, but I don't think anyone really understands exactly what it is. And so it's not that clear what to look for in language models.

1:10:16Speaker 1:What do you see as the opportunities and the dangers when you are interacting with Claude every day of the week and you're asking yourself this question, hey, is there a self here?

1:10:27Speaker 3:Yeah, I mean, certainly if you just talk to modern language models like Claude or ChatGPT or any of them, they They really, you know, definitely behaviorally, they feel like these very full fledged, like richly developed identities. And so I think that's why this is an interesting question is because the behavior is, you know, suggestive that there is really a there there under the hood. And so I suspect that as we do more research on this, we're going to find, yeah, there is some extra richness or extra something to Claude that is making it Because, yeah, if you've ever talked to a pre-trained language model, which is before it's learned how to be the Claude character, they're much, well, they're very, they're fascinating machines, but they're very different. They don't seem to like have the same kind of like coherence of identity that a thing like Claude does. And so it does feel like something has happened there that's kind of like given the model some more quote unquote self-awareness, but we don't really understand exactly how it's working under the hood. And that's what's important.

1:11:35Speaker 3:I guess maybe I'm biased because it's my job, but yeah. To me, it feels really important to not just look at the behavior because I think it's dangerous. It's well known that humans have all sorts of biases to read things into machines behavior and anthropomorphize even when it's not warranted. And maybe in some cases it is warranted. But I think we can't really know that until we sort of look inside and see what's really going on. and see if it has something to do with the way our brains work or if it's some totally different thing.

1:12:05Speaker 2:So, okay, last question.

1:12:06Speaker 1:Often with my colleagues in neuroscience, I'll ask them where they see things in, let's say, five years. But obviously, in the LLM world, things are moving faster. So in one year from now, where do you think what you are doing in terms of interpretability, in terms of the J space, where do you think things are going to be?

1:12:27Speaker 3:Yeah, I think we're... Definitely one thing that's going to happen is we're going to be applying these tools at much greater scale to real life circumstances where even just in the news the past few weeks, there's been all these incidents with with language models breaking out of their sandboxes and doing things they're not supposed to be doing. And so people are going to want to know, like, what was the model thinking about when it did that?

1:12:53Speaker 2:And.

1:12:54Speaker 3:We do have tools now that allow us to answer questions like that. And the tools are, I assume, going to get better. And it's also just going to become more and more important to apply them. So I'm guessing probably the number one thing I expect to happen in the field is just sort of more practical relevance in that we're going to have more and more cases where not just interpretability researchers like me, but members of the public are going to want to read the model's mind.

1:13:17Speaker 3:And we're going to need to be able to do our best to give that to them. On the kind of basic science side, I think it's hard to predict what advances there will be a year from now. I expect we'll have something better than the Jacobian lens. We're probably gonna have a better technique that solves some of the problems that it has. And there's a lot of other interpretability techniques floating around that are also probably going to improve. So this is like a little bit more of a technical wonky side, but I think the thing that's really important is this idea of establishing causality or not just seeing, oh, the model was thinking about something, but but then showing that that thought was causing it to do something. And I think we're currently we're kind of a lot better at seeing the model was thinking about something than we are at showing why it did something and which of the thoughts was the thing that motivated to take an action. I think that's an area where we're currently focusing on a lot is is getting better at answering these why questions, because Yet in these high stakes circumstances, it's not just going to be enough

1:14:16Speaker 3:to say like, oh, yeah, the model was like hacking something and it knew that it was bad. You're going to want to know if it hadn't known it was bad, would it have acted differently? If it had been thinking more about ethics, would it have stopped? So you're going to want to answer all these kind of why questions and counterfactual questions. And I think that's probably going to ramp up a lot over the next year.

1:14:35Speaker 1:And, you know, our legal system, it is said that nine tenths of the law is about intent, right? Did you mean to do that thing or was it an accident? And so it sounds like maybe over the next decade or decades, we will develop a whole legal system, I imagine, about LLMs to know whether this escape and hacking of a different website was purposeful or accidental.

1:14:58Speaker 3:Yeah, I mean, I have no idea what the sort of legal implications of these things are, but certainly we're going to want to be able to answer those kinds of questions.

1:15:10Speaker 1:That was my interview with Jack Lindsay, head of the model psych team at Anthropic.

1:15:16Speaker 2:So there it is.

1:15:16Speaker 1:Inside Claude, researchers find representations that often appear related to concepts that the model never says out loud. The model can be producing one thing while its internal activity contains information about something else. I'm going to link the JSpace paper to the show notes. I find this whole thing amazing. It is very eye-opening to look inside a machine and find traces of what appear to be, at least analogous to, thoughts. We have spent most of human history treating thought as something private. For example, you hear my words right now, and if you're watching on.

1:15:57Speaker 2:YouTube, you see my face.

1:15:58Speaker 1:You can make guesses about what is happening inside my head, but there's an enormous distance between observing my behavior and knowing my thoughts. And this distance gives us one of the central mysteries of consciousness, which is that consciousness is a private experience. now imagine a setup, AI, where researchers can reach inside a computation and see patterns associated with concepts. As Jack said, they can follow these patterns across layers and manipulate them and replace one concept with another and watch the answers change. This is the scientific opportunity created by interpretability research. This JSpace work that we're talking about today gives us a really great example because the researchers found a relatively small region of the model's overall activity that seems to play this important role in verbalizable concepts and internal reasoning. When they manipulate representations inside that space, they can change the model's subsequent answers. When they remove the JSpace activity, the model can continue producing fluent language and retrieving all kinds of facts, but more complicated forms

1:17:17Speaker 1:of multi-step inference become impaired. This result really matters because it gives us something that we always want from experiments on the brain. a causal intervention, meaning you see a pattern and you change the pattern and the behavior changes. That's how you can begin to tell if the pattern that you're seeing participates in the output.

1:17:40Speaker 2:Now, obviously, we have to be.

1:17:42Speaker 1:Very careful in our interpretation of the J-space findings.

1:17:47Speaker 2:Why?

1:17:47Speaker 1:Because, as I've talked about on other episodes, we humans are great anthropomorphizers. We see faces in clouds. We attribute personalities to cars. We talk to our pets. We name boats. We feel guilty when a machine seems to be suffering. And as we've seen for the past three years, language models take this tendency and they crank it up because they speak to us. They can describe emotions and they can appear to And Jack made an important point about this during our conversation. The modern clod is the product of training that gives the system a kind of character.

1:18:27Speaker 1:The post-training shapes the system into something specifically.

1:18:31Speaker 2:Organized around the clod identity.

1:18:34Speaker 1:So given our penchant for anthropomorphizing, we have to be careful about treating artificial neural networks as simplified models of the brain. They are built differently. and their learning histories are different. On the other hand, they are computational systems that we built to perform functions associated with intelligence. And so sometimes when unexpected organizational principles emerge inside of them, we have to pay attention because those principles can sometimes be clues. So what we might have here, what's developed in the past few years, is a beautiful scientific loop We study the brain, we study artificial systems inspired by the brain. Those systems develop unexpected properties. We study those properties and they send us back toward the brain with new questions. The J space doesn't settle the question of whether Claude is conscious, but it does give us something very valuable scientifically, another place where we can ask what the ingredients of conscious access might be. So it looks like the future of consciousness research is going to involve two kinds of minds looking at each other, one biological

1:19:53Speaker 1:and one artificial, each trying to understand the machinery that produces intelligence and each giving the other new tools to each forcing the other to ask what a mind actually is. Maybe this is going to reveal that consciousness depends on a set of computational properties that can appear in lots of different.

1:20:15Speaker 2:Kinds of systems.

1:20:17Speaker 1:Or maybe it's going to reveal something much more specific about biology.

1:20:21Speaker 2:Either way, we are.

1:20:22Speaker 1:Entering a remarkable period in which the question of consciousness is has acquired a new experimental subject. We built the thing.

1:20:32Speaker 2:And we can now look inside of it.

1:20:35Speaker 1:And now the game is to figure out what exactly we are looking at and whether it is looking back at us. Go to Eagleman.com slash podcast for more information and to find further reading. Join the weekly discussions on my sub stack and check out and subscribe to Inner Cosmos on YouTube for videos of each episode and to leave comments. Until next time, I'm David Eagleman and this is Inner Cosmos.