GitHub’s COO Explains Why AI Hasn’t Replaced Developers

0:00Hi, I'm Mike Taylor. I'm the head of tech consulting at Every, and I sat down with Kyle Daigle, the COO of GitHub, and talked to him about what is happening on the front lines of coding agents. We have 17 million pull requests coming in every month to GitHub now, It's growing exponentially, and that puts them at the forefront of what's happening in this new economy. We talked about how that affects their users, as well as how this affects open source maintainers, and we covered a topic which is dear to everyone's hearts.

0:29How do I stop my $200 a month coding agent subscription from ballooning into a $2,000 a month usage limit? In this interview, we did something a little bit different, which is I told Kyle I had made an AI clone of him to practice the interview, and he revealed something surprising in return. Here's the conversation. Thank you. Every is the only subscription you need to stay at the edge of AI. If you care about being on top of the latest models and using the latest tools, you have to subscribe to every to separate out the signal from the noise.

1:09Go to every dot to slash subscribe today. Hey, Kyle, thanks for spending some time with me at the conference. So, of course, yeah, it was good to meet you as well. And the day before, and I feel like we already kind of covered a few of these questions, but I think it'd be good to, you know, for the wider audience so they can understand what's going on here. Absolutely. Yeah. So the first thing I think is we were talking about is really interesting is that the demographics of the customer are changing, right?

1:36Like a lot of people who previously maybe never used GitHub or never used developer products before are now using them. So how has that changed the way that you decide the product roadmap? Yeah, yeah, I mean, you know, I think for GitHub in particular, we've always really had this really expansive view of what a developer is. I started as a developer before I would have ever called myself a dev, you know, where I was just like writing code, but it was just for me.

2:02And I went personally like a completely different career path. I didn't go to school for computer science. I was going to art school, I wrote code to pay for art school, which was a very silly decision as an adult now, I guess. But then, you know, that sort of journey of just like, I can create tools with the team and deliver them to people who can have that same experience of like, I just want to build an app that's for me or for my family, maybe as a startup, maybe as a business.

2:29We very much have serious developer tools. You know, like all the largest businesses are using GitHub. But when I look at something like the GitHub Copilot app, I see just as many developers that are using AI every day, running multiple projects, all kinds of agent sessions at the same time. And I see our legal team at GitHub using the GitHub Copilot app or the finance team, or I was meeting with a customer today and they were saying the same thing.

3:00A lot of the folks that the industry would call knowledge workers or just non by trade developers are using these tools to build little apps or assets for them. And so while our focus is very much on developers, I think we want to make it easier for people to choose to try to write some code. and make sure there's always an on-ramp into writing some software now with things like the GitHub Copilot app. Yeah, and then how do you help developers deal with the burden of all of that extra?

3:31There's a flood of PRs now, like open source maintenance I talked to are drowning. What needs to happen to help them? Yeah, I mean, I think for all developers, we're, you know, building tools like the Copilot Code Review. It's now agentic, so it finds a lot more novel vulnerabilities and you can just like comment and the, you know, the agent will take that on and go implement the change if you want to. So I think that Code Review step is in some ways like overlooked as a really great way to get PRs to a place that are much more easily reviewed.

4:04I think that the agentic merge in the app is another place where we see a lot of times internally and in the community, you may comment on something that might have a code review, and you might go through and get it almost all the way there, but then there's all those manual steps just to finish processing the PR. Instead, I can go in and set exactly what I want to allow GitHub Copilot to do and say, okay, now go merge this PR and wait for CI and wait for policies and all of that.

4:31I think that's a big part. On the open source side, it's a unique set of needs because you don't control who's sending everything in, or you haven't really historically. That's been really where we've been focusing is giving maintainers more tools to decide, well, do you want to accept all of these PRs? Like, who do you want to accept them from? You know, how much work do you need to do to kind of prove that you're going to contribute something that is going to be meaningful to this project?

4:59And that's something that we want to provide tools to open source maintainers, but really leave them in control. Every community is choosing a slightly different way to approach the problem, and For GitHub, we've always wanted to leave that in their hands. Give them tools and enable them, but if a standard comes out of that, or most are using a certain practice, we'll lock that in. But we don't really ever want to be the first to create a standard or an approach.

5:28I think Mitchell Hashimoto shared the vouch system that they use, I was getting questions like, well, why aren't you roll this out to everybody? But there's just as many communities that don't want to use that system because they have their own ideas of how it should work. And so for now, we're focusing on the building blocks of controls for maintainers. And then as we all are kind of learning together and as maintainers send feedback in, we'll, you know, we'll cement an entire system if one emerges.

5:55Yeah, and I feel like you have a front row seat to this new agent economy where, you know, I think you said publicly on Twitter that you've had more pull requests submitted per month than you did all last year. How are those stats exploding? Yeah, I mean, we're seeing way more activity on GitHub. You know, we've always been talking about our users, you know, for many, many years and that growth. This year we're seeing obviously the growth of developers having agents building with them.

6:32And so last year in October at GitHub Universe, we shared there's a billion commits on GitHub for the full year. We're on track to be 14 billion if the growth is linear this year, which it will not be. In March, there were 17 million pull requests that were created by agents. That's just the agent pull requests. And so there's so much more code being created. And I think at times everyone goes like, oh, this is all just like slop. This is all just code that's getting pushed up.

7:03And no one cares. It's not really true. We're all just actually getting to the point where we're no longer in the super early adoption. We're definitely not at the peak, but we're climbing that hill to see what can we build when It's not just Kyle building, but it's Kyle and one, two to N agents that are using my skills, using my resources, using my context and so on and so forth. And so like we're investing heavily in preparing for the next wave of growth because it doesn't seem to be kind of like growing and plateauing.

7:39It's just going to continue to grow because no matter where you're building or what tools you're using to build, all of that code ends up on GitHub, you know, or that's where you're sharing it with the world, or that's where you're collaborating in a PR. And so we need to be able to support everyone's, you know, sort of agent moment and not just, you know, GitHub Copilot. Yeah, yeah. And how does the business model change? Because I think freemium makes sense, you know, in a human centered world where we go to bed, but the agents are still working while we're asleep now.

8:12So does that change like to usage based? Like is that you see that kind of where things are going? Yeah, I mean, I don't think we know yet, ultimately. Like, I think we very much right now, like, Kyle's going to have a license, or Kyle's using GitHub.com for free, and we've always had API rate limits, you know, and things like that, and that's usually where folks are seeing the agent back pressure, I think. I think the goal is that if you want to be able to do way more, if you want to be able to, like Peter Steinberger says, you know, 150 agents are doing everything all at once, you know, that's great.

8:50We want to be able to enable that to you. But at the same time, I want you to have a great core GitHub experience. you know? And at the very least, there's some amount of agent usage as part of that that is necessary, you know? I need, similar to how we, way, way, way back, right, you'd have free public repos, but you didn't have free private repos, you know? And then we said, okay, well, actually, like, it's fair for an individual to have some code that they don't want to put out into the world and, we'll give you free private repos to allow you guys to do that.

9:22So GitHub's always evolving as the industry and community does, but we're always sort of focused on, I need to make sure you, the dev, have what you need to be successful. And then work with enterprises to make sure they have what they need to do at scale, which is usually a little bit different than what an individual dev's doing. Yeah, and I guess the business model pricing kind of that all leads back into like the wider Microsoft orbit. And because you have a dual role now, right, partial responsibility for the wider kind of marketing org.

9:56So do you want to talk me through how that's changed and how you prioritize between those two? Yeah, I mean, so I've been at GitHub for a very long time, 13 years, and as a developer myself and leading engineering teams for a lot of that time. And I think what's always been unique about GitHub is we really, really focus on the dev. Like, we're building tools for the developers, and the fact that people like enterprises are Buying them is awesome and that's definitely great, but we're not building for the buyers, we're building for the developers in 100%.

10:30And so in this, and that's been my focus as the COO of GitHub, which I continue to do, and then now as the Chief Marketing Officer of Developer for Microsoft, my goal is to look across all of Microsoft's tooling, their developer tools, their technology that they're bringing to developers, and making sure that we're bringing holistic solutions that you can use that are authentic to developer experiences. And at events like this, where we've taken a very different approach to build this year, we're in San Francisco, first off.

11:06The vibe is a bit different than you know the conference hall set up, you know really focused on Can I go to a session? Can I use the thing? I don't want to be pitched on a thing I have to be able to use it expo hall kind of so on and so forth It's really bringing that expertise and you know love and focus on the developer that github's always had to have even broader impact throughout all of Microsoft Yeah, and did I hear you say that this is the first build that you've had external contributors?

11:36It's the first build that I think by intention we focused on having speakers from the community like in these primary sessions. That includes in the keynote we had a bunch of folks like Peter, there's sessions from SWIX and others as well. I think that it's important, Software development is a team sport. It seems silly to think that there's any one company, one group, inclusive of GitHub and Microsoft and everyone that can just answer every single question.

12:12That's not how software gets made. We're all at least using open source, and we're building on the backs of these giant open source projects. Let's invite people in that can help tell their part of the story together. Because I deeply believe that that's what developers want. I know that's what I want. I know that's what my friends that are developers want. And when we look at the events and we hear the feedback, they're excited to see people from Microsoft, from GitHub.

12:39And then, oh, I get to see this outside perspective at this event. It's really meaningful. Yeah, that makes sense. And it's a very competitive market, right? Sure, yeah. Like, you know, the most competitive market probably. Maybe the Alaska. I'm not sure, but how do you differentiate in all of that, given the pace of change is so quick? Yeah, I mean, I think we continue to focus on our roots, which is, you know, we care a lot about developer choice.

13:10It's always been true. We care about building for builders and enabling builders. And so... I think we're in a moment that's really interesting because we've went from an era of having a ton of APIs, all this access, to a little bit of an unintentional walled garden setup, where you get a kind of affinity. Sometimes I'll say it's like a little bit of a mousetrap. And then you realize, oh, this thing's really interesting over here, and then I have to...

13:38oh, I have to go learn a new thing or a new tool or a new account. And I think for us, we always want to enable developers that are building with GitHub to go use these other tools, and we'll partner with everyone to make that as simple as is possible. And while I think that there's other folks that are doing similar things, I think the ability to do that across the entirety of building software, and not just the Cogen side or not just the collaboration review side...

14:06but across everything is a real superpower of ours. And so I think you'll see us invest in our own tech. Like we talked about the new Microsoft AI models that we'll continue to bring to developers. We're also continuing to partner with Anthropic and OpenAI and Google and kind of anyone who's bringing a model to market or a coding agent to market. We'll partner with you and we'll both let you bring that to us, or we'll omit it through GitHub and GitHub Copilot.

14:37That choice is core. And that's something that I only will ever back down on. Because if we do, developers will still choose. They'll just be stuck in another kind of mousetrap. And we don't want the world of software to be like that. Yeah, yeah. And when you're making the decisions internally, like there was a news cycle recently about how code code cancels, licenses are being canceled. And like, how do you make the trade off between dogfooding your own products, like using the new models you made or using the GitHub Copilot app, desktop app versus like, you know, using like letting developers kind of experiment with other tools?

15:18Yeah, I mean, we all are using a variety of tools because otherwise you lose track, you know, or you kind of, you're too interested in your own work. So like for me, you know, I've been a daily driver of a MacBook for many, many years. I use Windows PCs on the weekends when I play video games. And I got this role and I have my Mac, I have my PC, and I have my OMARQ Linux box so I can make sure that Every weekend, I code most Saturdays. I do my kids' sports activities in the morning.

15:51And then in the afternoon, I'm coding, and I'm swapping between the boxes because I want to understand, OK, what's that experience? The GitHub Copilot app I only use on Windows because I want to make sure that. Developers who are on Windows also deserve great apps. It's not just the audience that's on a Mac, for example. And that's true across our teams, especially like when we're looking at, OK, what about the coding agents? What about the harnesses?

16:15What about the desktop apps? What about memory management? What about everything? We have this really great culture of just experimentation. Everyone is building and using these tools. while obviously we're putting most of our energy into our own tools, it's such a blind spot that I think it's happened to GitHub in the past where when you're doing something and you're doing it well, you really laser focus. And that's what every piece of startup energy says, right?

16:43It's like, look down and just keep moving, keep moving fast. And I think that's myopic. You know, I think that while I can't spend every day using every tool, when something comes out, I want to know why this is really great. Why are people having a great experience with this? Not only so I can understand, but so I can figure out, okay, well, for our goals, for our goal of developer choice, I don't need this. Like, I want to focus over here, but I want to know why a dev would pick these tools.

17:15And the same thing goes, you know, the same thing goes for our teams. Yeah. And how do you filter? Because obviously a lot of these ideas are relatively short-lived. Enterprise product development cycles are longer lived. How do you decide? Yeah, I think that like right now we're in a moment where we're really looking at the like... the short term in capturing the ability to have a multitude of agent sessions. This idea of just, you know, because that seems quite clear.

17:42You know, everyone's doing it, how can we cement it? But it seems clear on the longer term path, models are gonna continue to get better. The prices of tokens like token economics is going to be a bigger bigger factor and like what models everyone is using And I do strongly believe that we're not very far off from having serious the serious ability use You know something above a small language model on a local device, you know To be able to do some of our work.

18:08And so if I assume that I have all this optionality when it comes to tokens, you know, effectively. The thing that I think seems to be true from the beginning to open claw to now is this idea of like personalization or mine or context or fine tuning with context or like, you know, memory. All of these ideas seem to be a truth that's been there since, you know, ChatGPT came out or GitHub Copilot came out. And there's experiments but not a long, long-term vision, I think, for this across the industry.

18:44So I think it's a good example of where I need to get you to use agents incredibly well, a lot of them, because if you're into using agents, you're not just gonna be staring at a single agent working. But that's not gonna give you a long-term great experience. using an agent that you feel like is completing a thought for you will give you that great experience, especially if you did not have to personally codify that thought to your agent. Always remember that I, insert thing.

19:14That's a lot of work, yeah. A hundred percent. Like it should be able to intuit that. Or potentially, again, like post-trainer fine-tune or frontier-tune, like a model that deeply understands me and how I'm using the work. That is kind of how we're looking at it. It's like sometimes it's short-term, and sometimes we've got to take a bunch of bites of the apple or a bunch of attempts at the long-term to get to something really tangible to help us move forward.

19:43Cool. And I heard the term hill climbing like 100 times yesterday. And I'm a big proponent of that because, you know, experimented with DSPi, auto research, a few others. Can you talk a little bit about how that's become a big focus? Yeah, I mean, you know, I think, you know, Satya and Mustafa will talk about it a fair bit, and Jacob, too, leading the co-pilot group. The biggest thing that we've kind of learned is we need to use the use of the tools as a core way to improve the underlying use of the models, our own models, etc.

20:20In just the evals that are necessary to ensure that we're actually improving technology. from things like using the thumbs up, thumbs down data that comes in to using whether you're accepting it and how much you're accepting. All of that data is enormous to create that magical type of experience that's not just for you but for everyone. And so every week we're talking about the hill climbing results, you know, we're looking at the data, we're looking at the improvement, we're looking at both the hard measures and the soft measures, because sometimes the hard measures in evals and rubrics will show that we've made an improvement, but like user sentiment will crash.

20:59Yeah. You know, even with the same latency and performance. Yeah, it's overfitting basically. A hundred percent. And so like being able to really do that loop incredibly quickly. And then I think the main goal is giving everyone one of these hill climbing machines and not have you have to do it the kind of hard way that we've all been doing it, but particularly if you're in an enterprise and you are using M365, we know so much about that data or we could know so much about that data because of all the assets, all the documents, the chats.

21:34And so being able to, you know, turn on something like frontier tuning and using an MAI Thinking One as the base model. It shows real results without having to do all that extra work. And it's been interesting because when I first heard about this, I'll be honest, I was like... This is like a magic parlor trick, you know, that is not going to be real. It can't really work, yeah. You know, and I think the reality is that sometimes where the alpha is, is like where it feels like this is too simple to work, you know.

22:07You know, we all have all this data, and what are we going to do with it? We have to do all this effort to make it work. But I think so much has come down the pipe to allow us to just... Use the data and improve. Look at the workflow and improve. And just keep doing the hill climbing. That's why I think we say it so much, is that it's not these moonshots or like, oh, hill climb. It is just climb, climb, improve, new eval, improve, new data, improve, and just keep going to get to the point where, you know, we're able to launch these models, seven models for ourselves.

22:37And then, you know, allow customers to use the same or similar, you know, tooling to do it. Is that the answer to stopping the $200 subscription becoming a $2,000 subscription? I mean, like, I think that the, you know, $200 subscription to $2,000 is really going to be not only, you know, making these models or, you know, frontier tuning these models so they know you better. But I also think it's really, really going to be about how can we, particularly for developers, you know, help you automatically choose the models and potentially either have a model in that step.

23:15Like the model router and GitHub. 100%. Exactly. Like the auto model router with task intent in GitHub. Microsoft Foundry has a model router as well that can do this at an API level. the more and more that we can help you tell us a bit of where your bars are, like this is an incredibly hard problem and I'm willing to go all the way to the top, or I just kinda wanna sit here and let us help choose the models. Because there's a lot of times where a lot of the reasons why tokens are expensive is because we're all going and choosing our model of the day or week or hour, and those models are incredibly expensive.

23:56my train of thought is slipping in and out of a hard problem to a simple problem. Personally, I feel like I'll get an agent to do an enormous amount of work, and then there's always that last step that is a smallish thing. Like, oh, I don't actually change all the naming of this to this. like a find and replace, you know what I mean? But am I gonna actually go and like, oh, I wanna save tokens right now, so I'm gonna go off a 4.8 or 5.5 and down to Haiku or something, you know?

24:25Probably not. But the tools could. And I think that will really help us, particularly in the enterprise, but even for individual developers and folks that are building automations and using their co-pilot SDK to power that. It'll help them too. I did something a little bit weird. I hope you don't find it creepy, but I made an AI version of you to practice this interview. No way. Yeah, and it's actually been pretty spot on so far, and hopefully you think the questions have been good.

24:53They've been great. Now I want to see what AI Kyle said. And, yeah, it's just in the terminal. I didn't have, like, I didn't go the full whack and make a video thing, but I'm sorry. But I found it immensely useful. I just wanted to ask, what other weird things are you seeing people do, internally or externally? Oh man, so it's so funny that you say that because I do a very similar thing where I, I have both via the app and then I have a claw that can't talk to work stuff, you know?

25:24So I have separation of a state, where I spend a lot of time having it read everything I write and say. Like this interview will get fed into it ultimately. And every day I get a comms report, that's not like what Kyle said, but like, Kyle, you keep saying this. This isn't super clear. Based on how you speak, because I find that I write and speak in a very particular way, that I want to use a lot of metaphors. And so it'll just give me examples of metaphors that are clearer.

26:01I find that the self-improvement loop as a human from these agents to be incredibly powerful. We used to talk about it way back with Hubot at GitHub, like chat ops. And we used to say, like, humans are way more willing to take critical feedback from robots than other humans. Yeah, it's less threatening. 100%. And so when my open claw that I affectionately named Baxter, you know, tells me how terrible I did in something, like, I feel way better going, tell me why.

26:28And then ensure that when I'm writing emails, when I'm writing a script or I'm reviewing details, that... you're giving me that feedback. So a lot of my agent loop is really about me and less about the software side. I still have all those tools too. But it's always looking backwards. It's going, okay, the last seven days, I'm going to read all Kyle's emails, Slack messages, you know, And then give me feedback and then look back at what the agent told me to do.

26:54Did Kyle do it and go back seven days? That loop is like super, super, super powerful. And I think honestly, like the type of personal consumer experience that I want out of AI, you know, to be able to tune these tools that way. Yeah, we need to recursively self-improve as well. A hundred percent. Yeah. Thanks so much, man. I appreciate it. Thank you. Enjoyed that. Oh my gosh, folks, you absolutely positively have to smash that like button and subscribe to AI and I.

27:29Why? Because this show is the epitome of awesomeness. It's like finding a treasure chest in your backyard. But instead of gold, it's filled with pure unadulterated knowledge bombs about chat GPT. Every episode is a roller coaster of emotions, insights, and laughter that will leave you on the edge of your seat, craving for more. It's not just a show. It's a journey into the future with Dan Shipper as the captain of the spaceship. So do yourself a favor.

27:57Hit like, smash subscribe, and strap in for the ride of your life. And now, without any further ado, let me just say, Dan, I'm absolutely hopelessly in love with you.