Specialized AI for Data Engineers: Inside Astronomer’s Otto

0:11Tobias Macey:Hello, and welcome to the Data Engineering Podcast, the show about modern data management. Your host is Tobias Macey, and today I'm interviewing Yetunde Dada about Otto, astronomer's expert airflow agent. So, Yatunde, can you start by introducing yourself?

0:25Yetunde Dada:Sure. Thank you so much for having me as well. So my name is Yetunde. I'm a senior director of product management at Astronemer. You'll find me across the Astronomer ecosystem because I oversee things. We're going to talk about auto today. But I also lead other initiatives like Kosmos, which is our way of converting DBT projects into Airflow decks. My background before this is many, many years in the open source space. I think there's been a previous version of your show where you actually covered Kedra, which was like a data engineering and data science framework that I worked on like many, many years ago. So it's good to see you again and be back.

1:00Tobias Macey:Absolutely. Yeah. No. I I definitely remember that one. Yeah. I've been doing this show for probably too long. Not long enough. Not long enough. There keeps being enough interesting things to keep it going. So, yeah, I guess, can you just give a bit of an overview about how you get started working in data and what keeps you here?

1:16Yetunde Dada:Sure. So I guess my origins go way, way, way back. I started off as a data product manager, so more of a consumer of data in a bank in South Africa. It was there that I started to I think this was in the beginnings of Hadoop and Spark way back in the day when we were trying to obviously create these big data pipelines, combine all of our customer data together in the corporate investment banking setting. My journey beyond that was actually more thinking about the data engineering teams, the data analyst teams that were actually trying to work with the data and building tools to support them. So that was the journey with Kedro when I was at Quantum Black.

1:54I got to work quite closely with Pete on the astronomer side because there was a natural entry point where we said, Kedro, what it does is it's kind of like Django or React for data science and data engineering projects. And there's a natural entry point where you say, Okay, you've got this production ready data product, and you want to put it into production, and you want it orchestrated, so just do it with Airflow. And that was how I got to know Pete. And yeah, now you obviously see me on the astronomer side. So always been passionate about data and I guess how it actually helps people when it's it's correct, when it's timely. So, yeah. Now you're in the vanguard of the net new and yet to be explored area of the combination

2:36Tobias Macey:of data engineering and agentic capabilities. So I'm wondering if you can give a bit of overview about what auto is and some of the story behind how it came to be and why you're building it.

2:46Yetunde Dada:So I guess for this, I mean, like, we've obviously seen how agents and, like, their use specifically in code and on the software engineering side has been, quite prolific. These tools have become incredible for the work that we need to do when we talk about, like, generic software engineering. But last year, we actually ran a survey. We're quite active participants in the state of Apache Airflow survey. And we did find, while a lot of data engineers are using a lot more of the generic coding tools and being productive with them, only 9% of them could actually speak about the quality of the code that was being produced by the coding tools. They often spoke about the missing context that it needed to be excellent in what it did. So there's obviously time spent where you have to create the context for it yourself or spend time editing the code that it was producing or doing additional steps to troubleshoot, for instance, if you're trying to use it in that use case. So it became an exam question for us where we said, Okay, cool. What does it look like if we actually help provide that context and we provide that layer

3:49so that we could make any agent really productive? And this spurned auto and the work that we've done to really just cement it as like astronomers perspective and way that we celebrate airflow and make sure that our data engineers are productive when working on the myriad of things that you'd be doing in your day to day.

4:09Tobias Macey:For people who are looking at auto and deciding whether and how to use it, what are some of the core problems that you're really tackling with the auto product and who is the target audience to sort of the characteristics of who is best served by it?

4:25Yetunde Dada:So for this one, I'll start with a persona because, you know, product loves this question, but it's data engineers and the range of things that you do. When we speak about like auto's capabilities, it'll help you all the way from things like DAG authoring and making sure that it's encoded with Airflow based practices when writing that code, all the way through to doing upgrades. That's a major capability that auto has. The skill that's present in auto doesn't only do major version bumps of airflow. So we know there's obviously a big conversation around moving from Airflow two to three, because Airflow two will be reaching end of life. But it also helps with minor version updates as well. So 3.1 to 3.2, as well as keeping your provider ecosystem up to date as well, because we obviously know that there'll be new versions released.

5:11Auto also helps with things like doing investigations. So we obviously know, as a data engineer, your pipelines will inevitably fail because nothing is perfect and errors will pop up. But we have specific capabilities built into auto, which really help with troubleshooting and really cutting down that mean time to resolution as well. Additionally as well, auto also comes prepackaged with capabilities as well for migrating code. Some of our customers have spoken about their journeys of moving from legacy schedulers, Control M and Autosys, to Airflow because they really want to standardize on Airflow as the way that they do things going forward because it's open source and it can be more widely supported across the organization.

5:53So it really does come packaged with some of those capabilities that we expose to some of our customers when they opt in for it. There's obviously a host of ways that we've also found our customers still leverage and use it. One of my favorites I was hearing about recently was how our customer will use it to do health checks on data platform and really just find things that they need to be resolving on a day to day. So it really does try to be a partner for you across the many things that you'll be working on.

6:21Tobias Macey:And as far as the design and rollout of auto, obviously, you can't boil the ocean and get everything done all at once. So I'm wondering what was your process for figuring out what are the initial set of capabilities that we're going to target and focus on and refine, particularly given that agentic capabilities are still being discovered as far as how best to architect and implement them. And, also, there is the probabilistic elements. So you want to make sure that whatever you're doing, it has a high success rate and a low error rate, particularly for the target audience that you're focusing on?

6:58Yetunde Dada:I guess it speaks about, like, the journey that we've actually taken with auto. So last year, we released a product called the Astra IDE, which is an in browser web based interface for authoring DAGs or modifying them. And we also have a workflow for testing DAGs against ephemeral deployments so that you can easily spin up the DAG, quickly test it, and then obviously commit the changes so that you can push them downstream and do whatever you need to there. What we started to see with this interface that we had for the Astra IDE was the type of questions that people were obviously going to be asking as they interacted with it. Obviously, the first use cases that Auto, in the end, sold for initially was the DAG altering and DAG modification use cases. But we also saw as well that a lot of folks would be using it for troubleshooting. So the case for building out specific capabilities for investigations became the next big angle that we looked at. We also did see as well that a lot of folks would also use it for upgrading their code at the many different levels and helping using the agent to make those code changes, especially when there's breaking changes involved with actually changing the structure of your code. So that was another use case that we decided to double down on. I think going forward as well, this does speak to where to next in terms of some of the capabilities that we want to support with auto.

8:18There's been some questions around how do we, for instance, right size our deployment on Astra. And we're busy looking at different ways to make those use cases come to life so that auto, in a way, can assist our customers there. But mean, we will also call out as well that there were high acceptance rates for the code even in the space. I think we speak about an 80% to 90% acceptance rate, straight acceptance rate for our customers, being able to choose and commit the code that Auto is producing for them as well. So yeah, that was kind of how we thought about the initial capabilities.

8:57I will also speak about some of the work that we've also done building out additional use cases as well. So this also speaks to the way that auto is constructed. Auto has, in terms of skill set and skills that are available to it as markdown files, we'll have a component that is present in the open source community as well. So we have on our astronomers organization on GitHub, we have a repo called, Agents. And what it does is it has 25 skills that we've seen across the data engineering ecosystem as well that do a host of things, whether it's the DAG authoring case to working with DBT, working with Cosmos, as I mentioned as well, our framework for converting DBT projects into Airflow DAGs. So there's always additional use cases that we see that will be cold.

9:47So yeah.

9:49Tobias Macey:In terms of the skills in particular, that is definitely one of the higher leverage capabilities for anybody who's investing in agentic engineering. It's definitely something that I and my team have been spending a lot of time on. And one of the challenges is making sure that it is useful enough without being overly prescriptive because then it guides the agent on too narrow of a path and also constrains the set of technologies that it's able to reason about. And so I'm curious what your process is for developing and designing and evaluating those skills, particularly as models and harnesses and tool chains evolve over time.

10:32Yetunde Dada:You basically run into one of our active work streams. We're working quite closely with the university. I won't reveal just who yet because we need to figure out how we do the price embargo stuff. But you will see stuff coming out of the work that we're doing here. What we've basically designed is a test bench, which basically encompasses a range of data engineering tasks. We've also assessed as well the complexity and the hardness of those tasks as well when developing our bench. And the tasks are designed to look across use case or different activity that you would be doing with the agent and being productive with it. We're running basically auto versus other generic agents through the gauntlet of those engineering tasks, and also assessing as well the cost that it takes, the range of accuracy or completeness that it does when trying to complete the task, and obviously developing the framework there. We will be open sourcing the framework as well, the Data Engineering Benchmark, to make sure that going forward, when people look at how they develop new agents to also tackle things across the data engineering stack,

11:39that they can use this as an additional way of assessing its capabilities and also make things better. It's one of the ways I think that we get to celebrate in the open source community as well, building better agents so that our data engineers can be more productive. But it also makes it easier for us to also show you why auto's capabilities rank higher than most other agents because of the way that it's built.

12:01Tobias Macey:Digging now a bit more into auto and its particular relationship to Astronomer in terms of the target, it makes sense because the orchestrator is intended to be the core element of any data platform. It has the most visibility. It has the highest potential leverage for impacting the entire data estate of an organization, presuming that it's actually being properly utilized. And so I'm wondering if you can talk to some of the architectural aspects of airflow in particular that lend it to be something that a system like auto can work well against and just some of the peculiarities and particularities of airflow that you've had to either reconsider and just some of the ways that the interplay between the orchestration framework and the agent that you're developing forces changes to your overall design and approach across that boundary.

13:02Yetunde Dada:And, you know, if I actually just build on your question, I guess it depends on how you're using Airflow as well as to whether or not, like, Airflow is truly in the the interception path behind having all the context that it needs to be able to answer that question. I guess this actually also does suggest as well, one of the ways that we have decided to build out auto is, especially when you're using it in the CLI, because maybe this is another thing. Auto is available via the Astra IDE, which is our in browser workflow there. It's also available via our CLI, which is the Astra CLI. So you can just get started and use it there.

13:37When we talk about auto and the CLI and its uses there, the fact that it's able to call in additional MCPs to complete the picture is one of the ways that we find that our customers have been, in particular, successful with it. Because to your point around the way that Airflow works, Airflow is the supreme orchestrator. We know this, and it does its job fantastically well. But one of its design considerations is that because it's so broad and so general, unless you're very specific about telling it what each individual task does and where all of the context around that task is located, you might not necessarily have the full picture.

14:17So to give you an example of what this would have meant with our investigation capabilities in auto, when it was initially designed, we had situations with our customers where they would say things like, hey, the investigation agent has told me, yes, this task has failed. But it's basically told me the error is somewhere in Databricks and kind of go figure it out. And for that, when we want to talk about our vision towards self healing pipelines, for it to stop, for the journey to stop around troubleshooting at that boundary is not enough for us, which is why, like, Autumn or broadly, especially in the CLI, has obviously a way for you to set up your MCP so that it can actually go forth and go and help you troubleshoot in the platform and you complete the answer there. So yeah, it does depend, obviously, on exactly how you're using Airflow.

15:07I will say as well, it does I also maybe I can call out why, for instance, you'd maybe use Cosmos, for instance. When we talk about making sure that Airflow is complete and has that full picture, tools like Cosmos, which obviously stretch into your DBT project and really help unfold it into it being a full Airflow DAG, is much better than you using, for instance, like a Bash operator or KPO to be orchestrating and calling the DBT CLI to complete that picture there. So the more information you give to Airflow and the more central it is in your pathway, the better time actually you have with using these agents to have a one stop answer, especially when things are not going as expected.

15:49Tobias Macey:And for people who are developing these pipelines, they're bringing auto into the mix. I'm sure a lot of them have already started experimenting with some of the more generic general purpose agents such as Cloud Code or Codex or OpenCode or Copilot, what have you. What are some of the distinctions, and what are some of the reasons that they might reach for auto specifically instead of their day to day driver coding agent? And what are some of the specific workflows and capabilities that auto not necessarily enforces but encourages that you can't easily replicate by just pulling in a set of skills into whatever your day to day agent harness might be.

16:35Yetunde Dada:So for this one, like, obviously, we are the Airflow company. And we spend a lot of time curating proprietary knowledge that is not necessarily available in the open source community. In particular, I'll call out three branches of it. One might be exactly on how to use the Astra platform. And we make that obviously available to Otto as a knowledge base. Another one to call out is the capabilities around troubleshooting and investigating DAG and task failures, because we have extensive knowledge on how to resolve those issues.

17:08So we've built that into Auto2. And the last one to call out is that the knowledge base around upgrading Airflow DAGs, especially moving from major versions, minor versions, and then equally, knowledge around upgrading your provider ecosystem as well is extensive. And that's also made available to auto there. So what you eventually see when our customers actually talk about whether or not they're going to be using Cloud Code or Codecs in their workflow. They'll often reach for auto when working with Airflow because it is the best tool in and above it being able to load the public documentation and the astronomer documentation into its interface. They'll use that. And they actually talk about it just being extremely good at airflow in and above those generic tools because it has that we'd call it the public knowledge that any of these generic tools could use. It obviously has the Astra proprietary knowledge also built into it. And then it also has additionally, we have ways to obviously store memories and remember things about its interactions when you're doing things like perhaps it would be as simple as, this tag always fails, just retry it. And being able to store it,

18:19those pieces of knowledge context for auto to be using going forward. So it does get smarter over time.

18:26Tobias Macey:To that point of getting smarter and self improvement in an agent context, that also brings up some of the challenges and complexities when you're building any data engineering oriented tool is the concept of data access and data ownership and also the potential for lock in where data engineers are allergic to any potential for vendor lock in. We want our data to be easily transferred and migrated wherever we want it to as the wins and wins change.

18:56And so I'm wondering if you can talk to some of the ways that you've thought about that aspect of the design of auto to make sure that it is fully visible and in control of the person who's using it and avoids any potential vendor lock and then also the security and data access sensitivity element of it to make sure that there are very clear and well understood boundaries about what data is accessed when and where it might be applied so that there's no risk of training on somebody's corporate data.

19:29Yetunde Dada:So I'll first speak to, like, how, you know, we obviously save additional context about you, especially as you move through your journey of, like, using auto. We do obviously adopt a lot of the open source patterns around being able to save memories to a markdown file for you to access and make it so that you can edit it as well. So in terms of that whole thing of like, Okay, I've been heavily reliant on auto. And for whatever reason, I want to be able to use this context in other situations.

19:55Let's just save to your Git repo so that you are productive already in those situations. And you can obviously point a new agent to use it there. We have fought extensively about security, data privacy, and also letting our customers know exactly what forms of data are consumed by auto, how we make sure that we're protecting their privacy as we use it. The way that auto is designed, at least so auto has, obviously, its skills based. Part of it is proprietary, and the other part of it is open source. And then all of this is connected by our LLM gateway, which allows you to choose which models you'd like to use through our model provider.

20:36We have thought extensively about how we allow our customers to access the different models. And there is heavy separation around their data that is used, because obviously we'd be accessing things like your code. We would see your prompts through the LLM gateway as well. That all of those things are air gapped between our customers as well. So there's no cross sharing at all. It's just you and the way that you interact with auto as opposed to it being broadly available. So the way that you find that information, if you were very curious for whatever reason, is we do have a very good trust website which actually talks about how we think about data privacy and security for our customers to make sure that really they can trust that we have their best interests at heart when they're using auto and these agents.

21:22Tobias Macey:And the other interesting aspect of any agent engineering is the model choice factor, where somebody maybe has an existing enterprise license with one vendor or another, maybe they focus on open weight models that they run themselves, or maybe they even have their own in house model, and that can dramatically change the capabilities and behaviors of the agent within a given harness, within a given set of skills. And so I'm wondering what are some of the challenges that you're dealing with in terms of the design and implementation to have that adaptability and element of choice for people.

21:57Yetunde Dada:This actually does suggest what are some of the things we've learned about our customers as well as we've been on this journey. Because for some of our customers, to your point, they might just have a preference and have standardized on a single tool that they use in their organization. Or it might even be as simple as, look, it's hectic. Let's say, hectic to get AI tools approved internally in their organization. And they're worried about how do they get another one approved, especially when this one is a standard choice.

22:25With this one, we've kind of made two steps to actually solve it. And they're going be shipped over the next month or two. The first one is an ability to bring your own model. So within the LLM Gateway, we're making it possible to kind of bring your own API key to your model so that you can plug it into the spaces like where auto fits across the Astra platform. So whether it's through the CLI or through our IDE or even on the Astra workspace, and we would be able to leverage your model, especially where you have these requirements.

22:54And then the second thing as well is we're just about to re release our Astra MCP as well. And obviously, the scope of the MCP was largely around providing a wrapper, you could say, around the Astro API and some of the functions that you can do there. We will also make as well in that package a way for you to access auto skills as well as part of the package that you get with the Astro MCP. And in this world, it means, obviously, that if you'd standardize on a specific agent, you could still actually leverage at least auto skills in order to be productive with the workflow, especially in cases where you do have a preference around using a specific agent and want to do that.

23:38Tobias Macey:And so digging now into auto itself, can you describe some of the design and architecture of that utility and some of the ways that the scope and implementation have evolved from when you first started building it? So

23:55Yetunde Dada:in terms of the way auto is constructed, there are two layers of basically proprietary and open source knowledge. The open source knowledge comes from the agents repo, which I spoke about that you should go have a look at. And then obviously, we have a proprietary knowledge base that is also part of the equation for auto. It is all behind an LLM gateway, which basically allows our customers to choose which model they want to use through our model providers. So they have the flexibility to do that. To your point around what kind of architectural decisions and choices have we made and I guess I can also speak to auto V2, which is also one of the things that we're working on now I think in tune with all the work that we've done on the benchmarks and really assessing auto against specific use cases and how well it stacks on being able to complete data engineering tasks in a timely way, using less tokens and making sure that, obviously, our customers spend less.

24:57Auto is built on top of the PIE framework as well. We've taken a real stab at really looking at what it means for which context to be applied to auto. These models have become very good at what they do in terms of how they know about airflow and have a wide understanding of it. So not all context, as we had it in the previous version, in the current version of auto, is necessary for it to be productive because the agents already know. So you do see us making this exercise of we're applying context, running that version of auto through the gauntlet of data engineering tasks, assessing if that context made auto significantly better because that's the delta, And then if not, removing that context as well. So you are going to see as well a slimmer version of auto, which only really has the stuff that really makes it exceptional against the other agents. And that's what will be packaged and released. So, yeah, those are some of the changes that we've made.

25:53Tobias Macey:Also, because you are bringing this into the workflow of an organization and a team, those teams have their own dynamics. They have their own preferences. They want to make sure that auto is well aligned to their stylistic guidelines, their data quality controls. What are the different axes along which they can personalize and extend auto to fit their particular situation?

26:19Yetunde Dada:Cool. So with this one, auto does respect if you're using, like, Cloud MD or Skills MD, Skills markdown file, like, Cloud does respect those things. So we always think of that as, the bucket of, like, team context that you might be applying to auto to make sure that it actually is following best practices and guidelines with the way that it works. In particular, though, I will call out, when we shipped the initial version of auto in the Astra IDE, the additional thing that it used to do was obviously refer to the whole code base as the standard for how it should be mimicking and writing code as well. So it would also consider that when it was doing code generation.

26:55I think as much as possible, even though we don't know how the initial code base was developed and whether or not that code base is best practice, it is the one it is the code base that is most familiar to your end users and to the way that you write. So auto does try and respect some of those things as well, but it it might make suggestions on how to improve things if it does seem bad practices being done. So yeah.

27:18Tobias Macey:As people are starting to bring auto into the mix, what are some of the ways that it changes their overall approach or workflow or just some of the adoption curve of people, particularly if they haven't already been heavy users of agents and they just say, my Airflow system is a mess. I need to get it into better shape. I'm gonna bring in auto because it's just magic. It'll do everything I want it to do. I don't even have to think about it. Just some of the, I guess, misconceptions or proper approaches for how to introduce it to a team, especially if they haven't already built up that muscle of working heavily with agents for other areas of their engineering work.

28:01Yetunde Dada:To be fair, with the way that agents are so prolific, there's agents everywhere. Most of our customers have had experience with some other tool when using it. And obviously, for them, when they try auto, the real differentiating factor is its knowledge on ASTRO and its knowledge on Airflow, which really does change the game for them. I can maybe speak to two examples of customers that have used auto for different purposes and really used it to make themselves productive in situations. So the first one is Janus Henderson Investors. They're one of the world's largest asset managers.

28:38And really, them, the use case for them was around, how do we make sure that we can reduce the mean time to resolution for our DAG failures? When a DAG goes down for them at 3AM in the morning, that's money lost for them as a company. And they used to have tons of on call engineers that their job was just to be maintaining these pipelines. So they built Auto into their Lighthouse product. And basically, focus of Auto in this situation is, how can we help with investigating the failure at the moment that it happens, creating the PR, especially if a code fix is required.

29:17And then also, if the answer is just retriggering the task to see if that actually does it, Auto should do that for them. And it does it via the Astralem API. And then if it's anything that it can't really fix, then it should hand it over to the team to come and look at it. They talk about a 95% reduction in the meantime to resolve open bugs and things that they have related to their DAGs as really one of the ways that they talk about success of this initiative.

29:48Another case might be as well, to your point, because you did bring it up when you spoke about some teams that have not really adopted maybe agents in their day to day. I can also give a case about a customer that needed to adopt Airflow because they were trying to do what we call a legacy tool migration. So they're one of The UK's largest consumer electronic companies. And they have many stores and then also have a logistics network that manages the sale of electronics.

30:18They spoke about wanting to sunset their jam scheduler. And they had about 5,000 jobs on this jam scheduler and needed to very quickly get up to speed with Airflow because that was the tool the data engineering team had selected that they were going to be moving to. Auto was essential in the team being able to get up to speed with Airflow knowledge and also migrate a host of jobs in a very short time scale. And when they started using Auto in this way, they had had experience with other tools like Copilot, but still found as well the quality of the code that was coming out of Auto, the fact that it was best suited to Airflow and ASTRO was actually best fit for them. So they continued to work with Auto on their journey of modernizing their data platform. And as they sunset more tools and move to Airflow, it really is an essential tool for them around the journeys of upskilling, along the journeys of DAG authoring in this case as well. And then also, they also use it as well for DAG improvements going forward. So yeah.

31:26Tobias Macey:And as people are investing more in using auto as part of their day to day work, one of the other challenges of generating new code is the challenge of validating it. And I know that that Auto has a PR review capability, and I'm wondering if you can talk to some of the work that went into ensuring that that was providing enough confidence for people to be able to actually move faster because it doesn't matter how much code you can generate if you never ship any of it and just some of the other elements of building useful and accurate validation loops into that overall use case of pushing more of the change through agents that are directed by human operators.

32:11Yetunde Dada:So I guess there's actually two layers to thinking through how you validate your code, because you could validate it at the time of development, and then you can validate it at the time the PR is created. I'll speak to both for auto as well because it does have tools that help you with both of those things. Auto, especially when you're working with it through the CLI, will actually call something we've called AF CLI, the Airflow CLI. And it's a trimmed down version of the ASTRO CLI.

32:39And it was built with the intent that it's more agent friendly. The ASTRO CLI is built for humans to be using, so it has more of an interactive feel as you're working through it. And we really did want an approach where an agent could interact with a CLI and be productive with it. Because Auto has access to this tool, it's actually able to validate your code for you. So it will spin up a local Airflow instance for you. It will trigger your DAGs for you. It will troubleshoot and resolve the errors for you as well before you're ready to create, obviously, your PR and work from there. Now, to your point of raising the fact that auto actually helps with code review, we do actually have skills that are focused on reviewing the code against Airflow based practices, against things like our Airflow upgrade knowledge base as well. So even the code that you're working against, it will constantly be checking to see if this code is up to scratch. And then make suggestions as well on what to improve as well for the person that is reviewing the code.

33:41Teams will use this in and above whatever because some of our customers do use another code review tool that has other best practices built in. But auto is basically the layer on top that is the airflow expert, airflow expert that will help guide the PRs and the changes that they need to do there.

34:00Tobias Macey:As you are bringing auto into the ecosystem, working with your customers, onboarding engineering teams, what are some of the most interesting or innovative or unexpected ways that you've seen it applied?

34:15Yetunde Dada:So I will speak to one of our customer use cases that's probably my favorite. They've decided to use auto as a way to do a health check across the ASTRO platform for them. What auto does in this situation is it generates a daily report of the performance of ASTRO across their deployments. So it gives a read on what is happening with their deployments, what is happening with their DAGs. And when they have this generated report, they basically triage specific issues that they need Auto to go resolve. And Auto is the functional partner with helping them manage their workload and address specific issues that they find.

34:53I think for this, it does speak to, obviously, one of the reasons why we developed this, which is that we want you to have an agent that is your partner alongside all of the work that you have to do. You will have so many competing priorities and things that you need to keep up to date. And if you can hand off some of the most boring pieces of your work to an agent to go and do, why not? So, yeah, I think probably using auto for health checks, I think, has been one of my favorite to learn.

35:20Tobias Macey:And as you are working in this space, building this system, helping to push the forefront of what agents can do, particularly in the challenging space of data engineering? What are some of the most interesting or unexpected or challenging lessons that you learned personally?

35:37Yetunde Dada:This maybe also speaks to the roadmap for auto going forward. So one of the things that we have learned is that and you did actually allude to it when you said something around people have their preferences about which agents they're using, which interfaces they're also accessing the agents from. And it really has become one of our goals that we're able to integrate auto exactly where you are. So you will see things over the next few months where auto comes to the Airflow UI. Because when you need to troubleshoot a DAG, you're not necessarily anywhere else. You're sometimes on Airflow and need to be able to quickly resolve that.

36:13To things like we are working towards releasing an Astro Versus Code extension as well, which also brings auto to your workspace because that's where most of our customers are writing code still to make sure that it's top and present. So yeah, we really speak about how do we make sure that we can put auto in the places that you're working so that you can be productive there go and use it, as opposed to making you leave different tools. But I think going forward, I am very excited to see how all of this agentic work really does impact the way that we view and see data engineering. My hope is that, obviously, with auto taking care of some of the more boring and operational tasks for your work, you're really able to really prioritize the value creation activities upfront, the new work that you have around making the data platform even better, to even things like, obviously, working through the massive backlog of new DAG requests that you have and being productive there.

37:17So I really do see an exciting future for data engineering just purely because we have these agents that have become really good for data engineering work. They know the code. They know the data. And they know exactly how to make you productive there. So yeah.

37:34Tobias Macey:And on that point of knowing the data, that brings me back around to something that we didn't dig into yet where the orchestrator knows what it knows, but it's not necessarily a complete view of the entire data estate. And maybe you have three different Airflow instances that are all focusing on different areas of the business or different use cases. And I'm wondering what are some of the elements of things like data catalogs that are maybe more cross cutting that give the complete view because maybe you don't want to have one Airflow instance running everything and just some of those other aspects of additional context engineering that you need to do to make sure that auto actually has the necessary detail to properly design and scope a particular change.

38:21Yetunde Dada:So this is where I actually alluded to the fact that, like, auto in the CLI as well, like, in particular, can access, like, an MCP connector. It has an MCP framework as well. This makes it obviously possible to bring in that additional context because to your point, depending on how deeply integrated you have Airflow in that system I think I gave the example of whether or not you choose to use, for instance, Cosmos, which will unpack the entire DBT project and have a very clear structure for it as opposed to using a Bash operator or Kapio to run the dbt CLI instead means that you're obviously able to bring in that additional context for where you need to complete the picture.

39:03Yeah, I do see it as the partnership between Airflow and the tool or the thing that it's supposed to orchestrate and how you bring in that additional layer to complete the picture. So yeah, think the only real way to do it is bring in the context where you need it. And auto can do just that.

39:20Tobias Macey:And for people who are working in the data engineering space, maybe they're already using Airflow, maybe they're using a different orchestrator or no orchestrator at all, or maybe they've already invested a lot into a different agent harness, what are the cases where auto is the wrong choice?

39:39Yetunde Dada:I guess if you're very, very no, let's actually break this one down because I think there's ways to go with it. When we talk about auto as an entire product, we're talking about its open source skills, its proprietary knowledge base, and the fact that it's connected to our LLM gateway. And it's available within, obviously, the Astronomer platform. And we provide all of the additional tools on top of it, like having access to memory and storing your team conventions and the like.

40:09In situations where you do have to use your own agent because maybe your organization has picked a specific tool that you're supposed to use, I would say that maybe auto doesn't fit for your use case then, because obviously, you'd have to go through ways of getting it approved or chosen to be used in as well. And maybe you don't have the authority to do that. But the reason I thought to highlight the fact that auto is broken down into those multiple components means that we can still make the context available to you in different ways.

40:38Because in those situations, maybe the better fit for you is to use the Astra MTP, which we're going to be releasing soon, which has obviously access to the auto skills. So you can pull them into your agent and use them from there. Or if you need to go a step further and just use widely available open source things, then I'd recommend that you'd use the agents repo, which we have, which has an available skill set of the open source skills at the least. Doesn't necessarily have the proprietary skills, which are the special sauce for auto.

41:08But you have something if you're going to be using it, especially in the context of working with Airflow as well. So there's different ways that we think to package up parts of auto so that you can use it, like, I guess, in the situations where you need it.

41:23Tobias Macey:And you've mentioned some of the forward looking plans that you have for auto, but I'm wondering if there are any other aspects of the road map or just particular

41:32Yetunde Dada:areas that you're excited to explore that you wanted to share with folks? I think maybe the I obviously spoke about, like, the surfaces that we're gonna make available for auto. So the Astro MCP, which will be shipping the Versus Code extension, and then also the fact that auto will be present on the Airflow UI as well, really as a way to help our data engineers be productive. Maybe the last one I'll talk about is the fact that we're moving towards this lofty vision of self healing pipelines.

41:59There's something about the Yanis Henderson investors example where we saw it was not just that Auto was able to diagnose the issue in their lighthouse design, but also able to act on behalf of the data engineers so that you do have this wonderful flow between there's something wrong, there is a fix for it, and the human just needs to approve whatever the fix is as a way to shorten time. And these are some of the things that we're working towards across the Astroid platform that we make this capability widely available for all of our customers.

42:30In the next quarter, we're kind of working on the resolution pathway for the different things that that Auto will diagnose as the issue, because it could be as simple as just retry the stack. It could be as complicated as there was a code change that needs to be made, and it's because of this data issue. Or maybe it's failed. We've realized that the data quality check has failed, and we need to try and diagnose these things. We really want to make sure that auto is really good at resolution before then extending it into ways where we could say, let's make it more automatic for our customers. And maybe say, for instance, that if there's an alert that goes off, that auto is immediately looking into resolving those things for our customers. So we're definitely on the journey towards building towards that self healing future with human approval, of course.

43:17And yeah, I'm excited. I'm really excited to see it brought to life.

43:21Tobias Macey:Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you and your team are doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling, technology, or human training that's available for building data systems today.

43:39Yetunde Dada:You've actually spoken about it quite extensively, which is that context layer which is missing and how complete that picture is. So I think really the journey for us is not just us as astronomer and airflow, is to make sure that we have all of the pieces to answer the question. Because for whatever your tool stack looks like internally in your organization, being able to connect those dots with your agents so that they have all the right pieces is actually where you get the most value in the end.

44:08So yeah, I think it's just like, if I could also advise as well, when you're thinking of constructing your data platform,

44:16Tobias Macey:really making sure that it is accessible to these agents as well so that you can be productive. All right. Well, thank you very much for taking the time today to join me and share the work that you and your team are doing on auto and just some of the overall challenges and learnings about bringing agents into the messy world of data. So I appreciate all the time and energy you're putting into that, and I hope you enjoy the rest of day. No. This has been fantastic, and thank you so much for having us. It's been great talking to you.

44:49Thank you for listening, and don't forget to check out our other shows. Podcast.net covers the Python language, its community, and the innovative ways it is being used. And the AI engineering podcast is your guide to the fast moving world of building AI systems. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@dataengineeringpodcast.com with your story. Just to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.