Episode 304: The Agentic Era & Books with Mark Saroufim

0:00Mark Saroufim:Well, like, I think my take on events these days is just like the agentic era has made open-ended judging a bit cooked because it's like, how do you tell whether something is amazing versus not? If it's just like if Claude one-shot it. But if someone did a hill climb on a task that we can all collectively agree on is hard and made like a breakthrough, that's better than what I would do myself as like an advanced user. Like, if they can just do that during the hackathon, like I'm very impressed.

0:29Mark Saroufim:And so, like, it just doesn't matter how they did it, it would matter is the output. So, I think that I'm very open to, and I think it's gonna become a lot more interesting doing it with open models this year, because like the open models are good finally at these tests. I think it's it looks like they've been RL'd on kernel data. Yeah, and so I think it's gonna be a lot more exciting than it was like a year ago.

0:57Conor:Welcome to ADSP the podcast, episode 304, recorded on July 20th, 2026. My name is Connor, and today with my co-host Bryce, we finish part three of our three-part conversation with Mark Seraphim.

1:08Bryce:In this episode, we continue our conversation about GPU mode, auto research, Mark's favorite books, and more.7, which feels like a long time ago, was like auto research on Cubs a histogram. And that auto research, I didn't give it any steering, and it basically that auto research popped out six different relatively novel algorithms. Two of them were like kind of existing, a twist on existing techniques, but the rest were like really kind of I think novel.

1:48Bryce:They did build on like existing things from literature. This is the SMIM cache that Connor that we talked about in the last episode. Like, certainly like caches, hash tables are like a thing that are in literature. And I think they're you know, histograms are very similar to to to to to a hash table in some way. So, you know, maybe there is actually some literature out there on this particular technique, but it really did feel like it was it was a novel, it wasn't a completely novel idea, but it was a novel combination of ideas, and there was no chance I would have ever come up with that idea on my own.

2:23Bryce:Yeah.

2:24Mark Saroufim:I I I guess Bryce, I'm curious to ask you on the specific example, like what was what was it like trying to merge it back into Cub? I don't know because I don't know if you did or something. I'm in the process of explaining that right now.

2:35Bryce:So so the the interesting thing with that, with the cub histogram, was the key like auto research run was done in like I think the last auto research run I did for that was like end of May. And so I I came out of that with like the the the candidate, you know, code in kernels. And what I've had to do since then is I was running auto research with Cub's existing benchmark, which handles which has basically like two different distributions. And you know, I knew I needed to do a much wider input characterization.

3:08Bryce:So I constructed a whole lot more inputs, I constructed adversarial inputs. Me and the agent learned that in fact the way that Cobb generated its histogram inputs for the benchmark have always been broken since the beginning of time. And the the the full like input characterization r takes like 30 hours on a B200 to like really get good data. And with Cobb, there's the constraint that we really can't ship six different algorithms, one for this particular dynamic shape, one for that particular dynamic shape.

3:40Bryce:And and so I've done more like manual work. How come, by the way?

3:44Mark Saroufim:Like, was it because it's sort of uh binary size?

3:49Bryce:Yeah. So so I needed to really to pick like I was gonna have to do go with two algorithms, one for the small bin case and one for the large bin case. And there are a couple of like additional post-auto research optimizations that me and the agent did. And the the those things were basically they weren't new algorithms, it was just like the right combination of things, like combining the SMEM cache with the existing privatized, you know, histograms and global memory.

4:19Bryce:And then like at one point we we turned off warp coalescing before you hit the SMEM cache was like a big loss, but doing coalescing on spills from the cache was a big win. And like I just got it sort of nailed down, and I I've looked at the code a few times throughout this process. I think there's two things I can try. The first thing is I can try to run my simplify scale, and the simplify scale is basically minimize the bytes of code difference, of non-common code difference from upstream.

4:51Bryce:The alternative I think would be to identify like a series of commits of like, okay, this is like the here's like four PRs. We could break this down into four PRs, and let's just let's just do the four PRs as like from fresh. Like, don't try to take the existing code and like shape reshape it into a PR, but just like using the existing code as a reference point, generate the PRs. And I think those PRs are gonna go up uh sometime this month. And my guess is gonna be that it'll take two to two months or so to merge.

5:26Bryce:I think it's in a shape now where it will be acceptable. The yeah, I mean the current the current code is it's a little bit of a mess. Like the biggest problem, the bit the single biggest problem was not following conventions in the code base. So like there was a convention for how tuning got done and it wasn't followed. And the a good part of it was things that the model never could have known. For example, like, oh, you know, this is the way that we used to do the tuning in this part of code, but you know, there's a new there's a new policy that's not documented anywhere for how we would do this, but the PR probably won't be accepted if it doesn't follow this new convention.

6:02Bryce:Or like there's you know, there's this convention that we if you want to have an algorithm that dynamically dispatches to two, if you want to have a cub routine that can dynamically dispatches to two algorithms, like you need we need to get a special sign-off from someone. And that's not documented in the code. So how could the model have ever known that?

6:22Mark Saroufim:This is this is interesting. Like basically, the this sort of makes sense if you sort of assume, like this is my guess, by the way, that like the cub team has been working on it for a while, that the original team hasn't disbanded much, and so a lot of these sort of like unwritten rules are actually in people's heads, and they all have internalized them, and there's no reason to write docs because maybe the team doesn't grow very quickly. But then, like in an age where you're forcing them to verbalize everything, they're like, Oh, goddammit, like this is so much work.

6:47Mark Saroufim:I have to write docs now and I have to write tests. Yeah, yeah, I see interesting.

6:51Bryce:Yeah, but I think it will be very interesting to see whether we can get the get this landed because Cobb in particular is a really high barrier to land because you need to be good across a really wide range of inputs, and you know, you you you you can't like ignore edge cases. And so it'll be I I I I'm hopeful because I do think it's a it's like a pretty big win uh for the for the cases that it like now I've gotten it to the point where essentially there's there's no adversarial inputs that are really bad for it.

7:25Bryce:There may be one left. I gotta try that. Yeah. Have you have you have you done any auto research work that has landed upstream PRs in like big projects that have a high bar for quality?

7:38Mark Saroufim:Personally, no. Like I think like around the time I left Meta, there was like I think an R like a fused RMS norm, which was like the first AI generated kernel that was merged, but that was like not my work. That was like Laura Laura's work on the kernel agent team. I do remember though, the main experience we had at the time was that like the kernel was quite long, and so I think it was important to deslopify it. I think there was sort of sometimes a bit of an ideological debate between like what's the point of deslopifying it, like just don't read it, to which like the PyTorch core maintainers were like, What are you talking about?

8:10Mark Saroufim:Because this is maintenance debt, I need to understand every single line. I'm not sure which of those two stances is correct because like some of those PyTorch people felt very strongly about these things, are just like no longer working on the team, and so like I actually don't know. Like, like, but kind of like one thing that stands out to me at least was like this one. Um I think like this is sort of a paraphrase from a talk by Terence Stowl, so it's not exactly what he said, but it's like what I came away with, which is that like he was sort of talking about like I mean, let me just not just talk about kernels.

8:39Mark Saroufim:Like, basically, like my my hypothesis is that typically by the time you're done writing a complicated kernel, you already have simplified it multiple times because as you're writing something, you're like evolving it, you're fixing it, and you've also like understood it. Like by the end of it, if you sort of like artisanally crafted something, basically uh generating, understanding, simplifying is like in some sense the same act. But like if we're in an age where people are only doing generation, it becomes like I think a bit of a problem because like let's say your kernel introduces a race condition, but it's like non-deterministic.

9:15Mark Saroufim:Well, you know, that really sucks for a model training run. But maybe it's forgivable for like a tiny little inference service that like runs some sort of speculative optimization. So there's like a risk-reward thing where like maybe there's some areas where it need to be really paranoid and this is not acceptable. But generally, like when I broadly look at this, I just don't buy the people don't care because you know, as we talked at the beginning of this episode on like if people are freaked out about non-determinism, like boy are they gonna be freaked out about like straight up incorrectness.

9:42Mark Saroufim:Um bugs that are like non-deterministic as well. Like, I think this is bound to cause conspiracy theories, uh, especially at scale, right? Like, if let's say 1% of your users only see something. Yeah, yeah. So, my broad take is like I think we should solve the hard problem. I don't think the pro the solution is people are reading the code. Like that that to me feels like uh like no, like, because it's let's say the QR problem. You know, at first I was like, hey, I'm gonna sort of synthesize all the kernels by all 170 users, and then I'm gonna try to come up with like a beautiful QR kernel.

10:14Mark Saroufim:Uh, you know, like I've I've yet to do this, it feels onerous every time I think about like sitting down and doing this exercise. But at the same time, if I just ask it like Godex and I'm like, hey, Godex, go cook, like, look at all these kernels and tell me what comes up. In some sense, I don't even know. Did it read everything? Like, did it like really understand like everything? Did it capture the key ideas? Like, maybe there's a slow kernel that had more elegant ideas, will it really capture them or will it really just prioritize the first like dozen kernels that are fast?

10:40Mark Saroufim:Like, I'm not sure. Like, in some sense, like I don't know what the model is doing, like I can't inspect it. Like, so it does feel like it's an alien artifact. I tell it go cook. But my hope is that like we can sort of figure out like how to come up with beautiful kernels from a large sum of fast but slop kernels. I just think like it's maybe in the beginning there's a bit more manual than we'd like, but I'm I'm fairly confident we can figure out like good tooling for for problems like this.

11:07Bryce:So, first of all, I I gotta say, I I f I find I find it very interesting the proposition that a like a uh a race condition, slightly buggy kernel that's like good 99% of the time but and really fast, maybe it's good enough in some cases. I hadn't considered that before. Maybe maybe it is. Um for inferences. So my goal in auto research here is to think about what's the process that I as a human would do. And I think that both for humans and agents, it's very hard to optimize for multiple things at once.

11:41Bryce:And every time that I've had to myself do like deep like kernel performance work, it's almost always where I have a phase where I'm trying to reach speed of light. And during that phase, you know, I'm adding restrict places, I'm adding in assumes, I'm adding in this and that, I'm rebenchmarking, I'm trying things, I'm if-defing stuff out, you know, I'm I'm I've got some dead code branches, etc. And then like, and then I reach performance. And I'm like, ah, great.

12:10Bryce:This is the right thing. And then I've got this mess, and then what I do is I go back and I'm like, okay, okay, I'm gonna delete all the if-defs, I'm gonna, I'm gonna, you know, take out those like if I take out this assume, is it fine? Okay, and you know, and then I've got I've always got like four versions of this code. And it's like, okay, well, like I can get I can delete this variant that I don't need anymore, and I can delete that variant. And it's like I always do this process where I've first spent a bunch of time trying to find speed of light, and then once you've found speed of light, then like you have an Oracle answer.

12:43Bryce:And then I can try to write better, cleaner versions. And so I think it's it's to me, it is like a natural process to do this V-shaped thing, or maybe not V or upside down V, where like you're you're climbing up to find the fast thing, and then like you're climbing down to make it not so ugly. And I think one of the, you know, maybe maybe I'm wrong about this idea that you or I said that like you can't always optimize for two things at once. Like it's harder to optimize for multiple things, harder to to optimize for multiple things.

13:16Bryce:And I think that's true, but I think part of that is just that we don't have the right reward metrics and safeguards in place. That like the to me, if if with my auto research process always generating these like 10,000 line, you know, slop things that's got all this dead code over, that to me means I I haven't put the right guardrails in place. I haven't put the right linters and the right things to like for code quality. And if I did more of that, then like I could probably generate a better kernels.

13:46Bryce:And also I think that would help with the stalls. And yeah, not not everything can be like deterministically checked or machine checked. The other thing is like I so right now I have in my framework I I uh you know the model can go off, I I let it it can do a macro optimization, a micro optimization, and then a combine. And I do have a fourth type called simplify. And usually I don't let it run simplify during the main auto research loop. But maybe I should.

14:14Bryce:Maybe it should be simplifying as it goes. I just the reason I set it up this way is that I've I guess I've always done some of the simplification as I go, but usually I make a mess of things until I find speed of light.

14:25Mark Saroufim:Yeah, it's an like your harness is like reflecting like your own process. Yeah. I'm not sure. Like I think it's hard for me to say this is right or wrong. Like it's this seems like much more of a like personal choice. So I can see argument good arguments being made either way.

14:39Bryce:Yeah. So yeah, it's I I I find I find the the the the mission that that you guys have at core automation is very compelling to me because like it seems it seems very evident that like if you can make the model really good at automating the development of the model, then like you can like then it can all it can do all the rest, right? You can just sit back and sip tea.

15:05Mark Saroufim:I mean what's cool about it too is that like I uh again, because like for me this is all new, but like by virtue of focusing on like on a few problems, I think it simplifies like a lot of things. Like uh, you know, we don't have to have like a model that's like say necessarily generally capable of all sorts of things, but it it primarily needs to be good at things that are tasks in the lab. So like you can think of every individual person in the lab as like an eval in some sense.

15:29Bryce:Yeah.

15:29Mark Saroufim:And I think this is like very helpful for me. And you know, when you look at like the kinds of tasks people do in the lab, like it's just it becomes like very self-evident, like how you should do the rest. That like, let's say for kernels at least, like none of the evals I know of open source, and I, you know, I know this because I've also created a couple of them, are like sufficient for what we want. And then it's like very cool thinking through like basically like how do you like basically what would you know the way I explain this to my colleagues is like what evals would help us like aura farm really hard?

15:57Mark Saroufim:You know, and then these tend to be like capabilities that you yourself think are really impressive. Yeah. Like like what do I think is at the limits of my capabilities? And if I can get like our next release to do those things, you know, that then I think I'd be really, really proud of the work we did.

16:10Bryce:Yeah. You know, and the the another the last point I'll touch on before I let everybody go and I'll go to sleep. You know, Connor, you mentioned earlier that like like the stuff you were doing, you know, like I stopped it because it it wasn't valuable. And I I just keep coming across places in which the usage data, the like the experimentation, like the the history of what has been tried is just so valuable. It's valuable for like one learning like an you can analyze it, you can improve things, you can analyze how particular tools or frameworks were used, and like once you have data at scale, like you can learn a lot from that.

16:53Bryce:It it's it's out it's valuable for like simulating how you know how these long horizon tasks work, which you know can tell us a lot about how various like frameworks and hardware might perform. And then it can be valuable for training models for for doing post-training and RL. And it can be valuable for like extracting, like, oh, like maybe this is something that would be a good, you know, a good benchmark. And I I just I think there's so much value in all this data.

17:24Bryce:And like that's one of the things I think is great about GPU mode is it's just it's this, you know, it's this farm that's producing all of this super high value data, like open data that we can all benefit from.

17:39Mark Saroufim:I mean, Bryce, honestly, to me, it seems like the the more the more you're talking, like the more I get excited about this idea of like, why don't we just make it easier for people to compete in our competitions using open model tokens? We can subsidize the tokens by picking the problems to be the kernels that show up in the open models. And we get this like nice recursive self-improvement loop with people doing search over the best prompts collectively, because I think that's like highly non-obvious.

18:06Bryce:For this, for these sprints over the weekend, like I and the Connor's definitely gonna have to cut this part out. Because it was like advertised as like, come, like we're gonna do agentic engineering on this hard problem, like they didn't have to learn anything about CUDA first. What they did was they had to learn how to set up auto research. And then, like on the second day, once they'd produced some auto research results, then like we like sat down and I talked to them about like the problem and taught them a bit about GPU architecture, and like they were able to learn more because they'd already already accomplished something.

18:43Bryce:They they didn't know what they'd accomplished yet, because the agent had accomplished something.

18:47Mark Saroufim:Yeah, but but but but they got like that sort of first like dopamine rush, right? Like I made something faster.

18:52Bryce:Definitely like one thing I I I have this vision for like in the future of like doing, I guess the way I I know you said you weren't such a fan of like in-person events, but I'm imagining like GTC Auto Research Academy, where like you just like you have two hours every day where you come you give everybody unlimited inference for the week, you come in for two hours every day, and it's like a study group because you know you're like in between the study group meetings, people are going and running stuff and producing results, and then you come back and you you check in.

19:21Bryce:And yeah, I think it'd be I I think it'd be very valuable to have open tokens, and I think people position open tokens as a public good as like this idea of like, oh, it's like it's goodwill, but for me it seems like it's insanely valuable to NVIDIA to do this. Yeah, yeah.

19:40Mark Saroufim:Like, but basically, I just to sort of be specific on the like I think my take on events these days is just like the agentic era has made open-ended judging a bit cooked because it's like how do you tell whether something is amazing versus not if it's just like if Claude one-shot it? But if someone did a hill climb on a task that we can all collectively agree on is hard and made like a breakthrough, that's better than what I would do myself as like an advanced user.

20:09Mark Saroufim:Like, if they can just do that during the hackathon, like I'm very impressed. And so, like, it just doesn't matter how they did it, it matters is the the output. So I think that I'm very open to, and I think it's gonna become a lot more interesting doing it with open models this year, because like the open models are good finally at these tests. I think it's it looks like they've been RL'd on kernel data. Yeah. And so I think it's gonna be a lot more exciting than it was like a year ago.

20:30Bryce:I want to pitch you on another idea I had, which is you know, you talked about the idea of having like a package, like here's how to deploy a you know, like like an inference service to do this like at your company. I've been thinking about this idea of, well, you know, what we want to do is we we need two we need GPUs for two things. We need GPUs one to run the tests and the benchmarks, and two, we need GPUs to run the inference. And you know, there's like web search is something that like runs like inference, like, you know, on the inference provider side.

21:02Bryce:Like, why couldn't we have like a you know inference provider side like tool call that just like like like that's like modal that just like runs the serverless thing there? And then like if you're a company and you have like you know an 88x B200 machine, it can both run your inference and run your serverless, you know, evals. Because a lot of people seem gated on the GPU access too.

21:24Mark Saroufim:Yeah, I I think of these as like very related problems. It's just that like I would imagine more people would know what to do with tokens versus a GPU. So it's like I would imagine like broader market appeal. But yeah, I mean like ultimately, I think the way modal set up how you acquire GPU, I think this is the correct way to do it because it's very incentive aligned with your customers to charge for use. Like it's an honest business, basically the way I would frame it.

21:47Mark Saroufim:Uh and I think and I think paying for access is dishonest because it it's uh it I I just yeah, like because whether you use it or not, it's like it's not great. I don't like it.

21:56Bryce:Yeah, yeah, it's like if I could like my biggest thing with this. Nvidia right now is like I need like I have this stream of like free tokens from Brev. I just they if they could just make a serverless product, even if it's like a worse version of modal, then like I would not need to ask for as many tokens and like it could be so much more token efficient. By the way, I refreshed the uh I refreshed the tokens for the the profiling service again too.

22:21Mark Saroufim:Thank you, sir.

22:22Bryce:Yeah. Um so I think there's enough in there to run through to run through the end of July. But so when do how long does the optimizer contest run?

22:32Mark Saroufim:The optimizer con we we don't know yet, but we're thinking of making it run for a month. Okay.

22:37Bryce:I'm just gonna keep refreshing the tokens for the profiling service until somebody tells me to stop.

22:42Mark Saroufim:Okay, that's that's wonderful.

22:43Bryce:Yeah.

22:45Mark Saroufim:I think I think we didn't get the chance to talk much about the optimizer competition, but I think it's it's pretty it's pretty sick, actually. I'll tell you more about it next time.

22:52Bryce:Uh yeah, I'm very excited. I I saw I started looking into it. Yeah, it looks like it's gonna be very cool.

22:56Mark Saroufim:Do you mind if I just tell you about it for a bit? Yeah, sure. I think just okay. So so the optimizer competition, like like the sort of main idea was that we we've been very interested in picking tasks where transformers obviously fall flat on their face. And so the task we picked is one that involves depth. It's basically like repeated squaring and modulo. You try to make any like the largest transformers run well on this task, it's gonna run flat on, it's gonna fall flat on his face, and it's gonna try and cheat the task.

23:23Mark Saroufim:And intrinsically, the only models that have like the right inductive biases to solve these problems are those that include like basically some form of looping or adaptive depth, or like basically and they basically involve like scaling training time compute to sort of like solve the task. And these are very exciting because if you sort of if you maybe have a brief physics background, like a lot of these, like anything that involves looping becomes like basically it turns into mush, like basically it just goes either to zeros or it goes to nan.

23:53Mark Saroufim:Like the system has become very unstable. And so what we're very excited about here is we want people to come up with interesting like model architecture ideas to solve this very like simply simply formulated but very nasty problem. But because these architectures are quite unstable, they will also need to come up with sophisticated optimizers to solve this problem. And the sort of user experience we borrowed is the exact same as kernelbot. It's like you submit a single Python file and we run all of these evaluations for you.

24:22Mark Saroufim:Right now we're basically going through like a trial period where people are reward hacking us to death, which is okay. It helps us like have the rules be more robust. We're gonna launch for real this Thursday. And the competition will be open for about a month, and we're kind of excited to sort of see who comes out of the woodworks here. What's been interesting is that like there's this like you know, there's like it's sort of this interesting intersection between people that do complexity theory and model architecture, where they will write very specific proofs as to things like, hey, like a diffusion model can't actually learn long sequence like video generation.

24:54Mark Saroufim:And I I find this like to be really, really compelling, and I just can't wait to see what people do. Generally, for me, I think GPU mode really becomes healthier if people are aware of like frontier level problems, and this is like a frontier level problem that I think many, many people are interested in. So I think we made it very agent-friendly. The GPUs are all free. You get an hour of training time basically a day. So, you know, please have fun.

25:16Mark Saroufim:The the website is like one layer deeper.ai, and I think it's gonna be like one of our most interesting challenges yet, and I'm very excited about it.

25:23Bryce:Yeah, no, I think it's I mean, I not be not being not being an ML person. I uh I I like being more a kernel person than an ML person, it's a little out of my depth, but from what I've read up on it, it it looks like it's a very, a very cool problem. And um, I'm very excited to see what people will uh will will be able to do with it. And it also seems like it's something that is probably not in the training set and is like a frontier problem that we'll really be able to see what people can do with auto research for this.

25:53Bryce:Yeah.

25:54Conor:Alright, one one last one last question, and it hopefully will be a short one, because I know Bryce, you need to go to sleep because you're over in the Europe's. One of my it's not really a hobby, but something I've been doing over the last several months is I I Well, the details don't matter, but I listened to a lot of No no no, tell us the details. The details? Alright, very quickly. I mean, I have a a vibe-coded podcast player called Pod God, which uh it was gonna get released in the next couple weeks.

26:23Conor:But I've been using it since the beginning of February, and it's phenomenal. One of the things it allows you to do is it allows you to follow personalities. So there's something called the personality meta podcast, which is just a curated feed of every single time that person goes on a different podcast. Because there are some people that I care about and I want to listen to any time they're being interviewed, but I don't want to have to go and listen to example.

26:44Conor:The person that I'm about to mention is Boris Cherney. Like, but the thing is is one No idea who that is. What? The creator of Claude Code?

26:51Bryce:No, no idea.

26:52Conor:That's very sad. He's like, he's honestly would be my number one pick to have as like a future guest because every every time he is interviewed, I think he has just like amazing takes. But anyway, so like there's a bunch of podcasts that one I don't want to subscribe to, like there's like Sequoia has this like kind of there's a bunch of these like VC money podcasts, but every once in a while they have someone interesting on. So like I don't want to have to subscribe to this and then like basically just like download the thing and then delete it.

27:21Conor:I just want the interviews. Anyways, that's I that's not long story short, that's long story long. Whenever you listen to Boris Cherney, like one out of every two podcasts, he gets like book recommendations or he gives book recommendations, and they've been phenomenal. The first one that he recommended, he actually mentioned in the Peterman podcast, which is like a former, I think, Instagram or Facebook guy, and he's just been it's it's great interviews.

27:46Conor:He interviewed uh Boris Cherney, and then he mentions that during the interview for Anthropic, he ended up talking about some book by Greg Egan. You know, this is back in like Claude, I don't know, 3.7, 4.0 days. I immediately go to like whatever ChatGPT ask, like, what book is he referring to? And it ended up being Permutation City, and that is like part of a trilogy that has Diaspora and Shield's Child. He on a more recent podcast mentioned Werner Vinge's A Fire Upon the Deep, which is in the Zones of Thought trilogy.

28:15Conor:I'm like one down, two to go. I bring all of this up, where's the question, to ask you? So you wrote the Robot Overlord manual, and you're the first person actually that we've interviewed on this podcast, which is a little bit surprising, that is as high on AI as Bryce and I. We we've interviewed some folks over the last little bit, but not like not people that are sipping the Kool-Aid as much as we are. And so uh the the answer to this question may be I don't have any recommendations.

28:44Conor:My question is, do you have any book recommendations in general? And on top of that, specifically, I would love to know if you have like any recommendations in the space of you know future AI, like good these kinds of good reads that like people like Boris Cherney recommend recommending. So over to you.

29:02Mark Saroufim:Yeah, like I mean, I I love reading.

29:04Conor:Like this, these are all my books. The listener can't see, but there's also like a bookshelf in the background with what looks like at least a couple hundred books there.

29:11Mark Saroufim:So the the there's like a couple of other hundred that I've given away over the years, but yeah, I mean, I I like I actually like I I do I I had a tweet about this, which is like my ten favorite books. Let me see. So I would say on the AI, like, okay, so these are these are the books that were most influential to me. There's ten of them. So let me see. Maybe I can pick two for you.

29:33Conor:Well, we'll include we'll we'll include uh the link to the tweet so you can get all ten. You can and feel free to just read all ten off if you want to.

29:39Mark Saroufim:Okay, like it's a more interesting question. Okay, the all ten. So my like the book that I think was like the most fun to me was a book called The Journey to the Ants. So it's it's it's it's a book about like ant-based intelligence. And the reason why I found it very interesting is because like the the kinds of shit ants can do is pretty wild. Like, for instance, like you can have like the mother ant, the queen, like control the gender ratio of a population, control like people's tasks.

30:05Mark Saroufim:Like you can be a worker versus a warrior elastically based on the demands of the colony, and they all sort of spread this message between each other to sort of dictate. This is very contrary to how like human societies operate, where we're like, oh, I'm a colonel guy, and I will never not do kernels, like we're very inelastic. So it's like pretty remarkable for me to see an example of um of societies that were not this way. And I think this book is wonderful.

30:27Mark Saroufim:Another book I really love is visual group theory. So visual group theory, and uh, maybe I would add Burns Euclid here. What I found really lovely about these books is that like any anything that's algebraic tends to be like very dry and like kind of unintuitive to read because you just see look at these long formulas. Both of these books are pretty like beautiful, they almost feel like pieces of art where people are sort of like going through like these arguments like very rigorously.

30:50Mark Saroufim:And so for me, they really made me appreciate like the idea of a beautiful explanation, which is like typically when I think of like my job, I don't necessarily think of it as like producing like beautiful code or fast code. I think of it as being able to produce a good explanation for why the code is fast. Like that's kind of almost like my love language. Another book I've adored is The Theory of Fun. So the Theory of Fun is a book on why people like playing games.

31:14Mark Saroufim:It's sort of like the sort of the main thesis of the book is that playing is basically the way we learn. So I've carried that a lot into my projects where I've tried to make a lot of the things we've done with GPU mode fun. And as a result, people end up learning more and might find it more valuable. So a lot of what I learned with GPU mode was like basically early days of playing World of Warcraft and Warcraft 3. So like a lot of the lessons, funnel enough, do apply.

31:36Mark Saroufim:I would say two technical books that I've I think I'm gonna skip over a couple of books, but two technical books that I think I highly recommend to everyone are uh game programming patterns and working with legacy code. Working with legacy code often surprises people as one of my favorite books of all time. But I found it really wonderful when I'm just like plopped into like a monstrous code base. How do you sort of like quickly make sense of it?

31:58Mark Saroufim:And like what are good design patterns to make codes like simpler and easier to reason about? I think a lot of like my contributions in the in past projects has been like going into a complicated code base and like understanding it and then figuring out how to develop features more quickly on top. I think a lot of these ideas will make a comeback if we're like basically in mountains of slop. Maybe yet another book that I've really enjoyed, maybe okay, maybe I'll give you two more.

32:22Mark Saroufim:Like Game Theoretic Foundations for Probability and Finance. It's not a finance book, even though it's in the title. It's a book about rewriting all the axiom of probability theory with game theory in a way that makes them a lot more interpretable. And so I've also really enjoyed this kind of book because I've always enjoyed like more wacky math. It just can help you build intuition. And if you didn't like one notation, maybe you'll appreciate another.

32:43Mark Saroufim:The last one is the structure and interpretation of classical mechanics. I'm sure many of you might have read the structure interpretation of like computer programs. That's what I thought I was gonna say. I I think this book is better because it basically goes over like it's writes short programs for physics written in Lisp, and you basically end up with nature as like a short series of like self-modifying Lisp programs. It's one of those like books that like blow your mind the first time you read it, but like again, like as we get into the age of auto research and architecture research, I suspect a lot of these ideas are are making a big comeback.

33:17Mark Saroufim:So these are things that I think are are you know there's a lot more, but like maybe you know, berserk and necronomicon, maybe with some pieces of fiction that I think are just like absolutely gorgeous. I I've always been a sucker for like there's some piece of knowledge that if you acquire it, I don't know, you'll blow up because you'll be rendered mad. Like I don't know why, but it's like one of those things that I've always like I've been like a sucker for this kind of like fantasy, fantasy work.

33:43Mark Saroufim:But yeah, all these books are phenomenal. I'll share the full list if you like with you, like in a few minutes. But yeah, I I think these are sufficiently out of distribution to give you better ideas.

33:52Conor:I mean, that was I feel bad we haven't asked our other past guests because that was that was uh amazing. Even better than not that I you know had expectations for the quality of your answer, but uh even better than I expected. Go ahead, Bryce.

34:03Bryce:I I have a I have a related question, which is it sound like it sounds like you have been a person who's played a lot of video games in the past. And I'm wondering if you have recommendations for video games.

34:14Mark Saroufim:Uh let me see. Um so what are the greatest games I've played? So so the one of the issues with like like I think the greatest game I've ever played is probably Dota 2, but I would not recommend you play it because it just will take over your life. So I think it's like an optimizer's dream in some sense, and it teaches you how to never get mad at anything because like it's very formative to be insulted at in like five different languages and still manage how to win a game.

34:40Mark Saroufim:So it's very formative. But but I would say, like, I mean, if I'm thinking like games that like left like a deep impression on me, I would say recently, I I would probably say it was like Expedition 33 as like sort of a story-based game, like really left like a deep impression on me, and I was like a big sucker for like role-playing games like growing up, you know, all the way ranging from like Pokemon to like Chrono Trigger. As a result, I've also really enjoyed Final Fantasy XVI recently.

35:03Mark Saroufim:I've been playing Final Fantasy 7 as well, which I really enjoy. But yeah, otherwise, like these days, like I mostly spent a lot of time playing Slay the Spire, but I also feel like I spent too much time doing it. So if you give me an optimization puzzle, usually I can get obsessed with it. If you give me something that's fast and reflex-based, like Fury or Katana Zero, I'm also like a big sucker for those. Like I enjoy sort of this like flow feeling.

35:27Mark Saroufim:And I think these are all games that will give you like a major, major uh sense of flow. So they're all treats if you haven't played them.

35:33Bryce:So my my actually my my oldest friend, the person that I've had the longest contact with, is somebody I've never actually met in person. And he and I, I used to play a lot of MUDs, text-based games when I was a kid, and he and I played MUDs together. And in particular, my favorite type of MUD was a roleplay-intensive mud. So that's where everybody has to be in character at all times. I primarily played either Star Wars or Lord of the Rings themed MUDs, and these were all like community-run multiplayer games, and nobody was really concerned about the fact that like Star Wars and Lord of the Rings were IP that somebody else owned.

36:09Bryce:But, anyways, him and I were having this conversation the other day where it's a it's a text-based game. Like, it is you could just like hook up, I'm sure, like a frontier model to it, and you know, the problem that these games have today is like, you know, the NPCs are not particularly good and there's just not a lot of players. And him and I were talking about like, well, we could just make as many players as we want. You know, you could just like you wouldn't even need to code them into the game.

36:33Bryce:You could just connect an existing agent to an existing mud and just have it interact the game as it would any other player character. And I just like I I can't haven't been able to get this idea out of my head. And if I had if I had more infinite inference, I would go build my like AI mud game, and I'm sure I'd have a blast with it.

36:52Mark Saroufim:I I I think we should try it out because I I think one hybrid idea I'd be very optimistic about would be something like uh like an in-person Dungeons and Dragons game. Oh yes. Where like the like where the DM can basically choose to like basically where there's a DM who's constantly prompting to try and anticipate where like things might go. That I'm I'm really optimistic about. But but I think a lot of the I think issues I foresee with current alums is like post-compaction, maybe they'll lose everything, like they'll just have no idea what the fuck's going on with the game.

37:22Bryce:We should do a whole we we should we should talk some other time about my thoughts on compaction. I think sometimes compaction helps and sometimes it hurts. In the short. Yeah. The thing that the thing that's most valuable that I get that happens at compaction is most harnesses on compaction will reread skills. And that can sometimes help because then it's like a reminder of like, were you following, were you following like the prompts and skills that you were originally given?

37:47Bryce:Anyways, I should go, it's very it's getting very late here, and we should let Mark get going.

37:52Mark Saroufim:Good night, Bryce. And then thank you for inviting me, Bryce.

37:55Conor:It was a pleasure to chat with you. Be sure to check these show notes either in your podcast app or at adspthepodcast.com for links to anything we mentioned in today's episode, as well as a link to a GitHub discussion where you can leave thoughts, comments, and questions. Thanks for listening. We hope you enjoyed and have a great day.

38:10Bryce:Low quality, high quality, that is the tagline of our podcast.

38:15Conor:That's not the tagline. Our tagline is chaos with sprinkles of information.