Building the Context Flywheel for AI Data Agents

0:11Tobias Macey:Hello, and welcome to the Data Engineering Podcast, the show about modern data management. Your host is Tobias Maci, and today I'm interviewing Prakulpa Sankar about strategies for building a context flywheel for your data agents. So, Prakulpa, welcome back. And for anybody who hasn't heard your previous appearances, if you can just give a quick introduction.

0:29Prukalpa Sankar:Yeah. My name is Prapepa. I'm one of the founders of ATLAN. At ATLAN, we are building a context layer for AI. I have been fascinated by the topic of how to build shared context for what I used to call the humans of data for now a better part of a decade. I started as a data practitioner myself, ran these large scale data projects, realized that data engineers, analysts, scientists, machine learning folks, business, what does business even mean? These people all have their own version of context, which made answering a simple question like number on a dashboard is broken, we don't know why, really complicated, like seven systems and five people were involved in answering that question.

1:14ATLAN was founded on the mission of building a shared context and collaboration layer for the humans of data, which we have now realized as far more sort of valuable in a world where AI agents are starting to take over some of those tasks, which largely, you know, humans used to be the glue for this context inside organizations. And so I've been on that journey for now, better part of, well, this decade with our customers.

1:40Tobias Macey:And do you remember how you first got started working in the data space?

1:44Prukalpa Sankar:Yeah. Well, I have been in the data space since I was 21 years old. I, at the time was doing work in finance and data. And I had done some work with nonprofits. I said these big problems like healthcare and education and infrastructure, they don't use data and it feels like they should. And let's go do something about that. So my co founder, Byron and I, we founded a company called SocialCops that was founded on the mission of bringing data science to real world critical problems.

2:16And that meant at the time going and partner with agencies that already had reach and scale. So it was large governments. It was the UN, the World Bank, the Gates Foundation. And we used to be their data team. We used to build data platforms for these problems at scale. And that is really where I learned everything that I learned about building and running data teams and how complex and chaotic it can get to actually make a data project come alive.

2:44Tobias Macey:And as you mentioned, you started ATLAN with this mission of collecting all of these disparate data systems and the metadata and semantics around that so that humans could make sense of all of the information that was available and how it was being used and how it can be used. Obviously, that has had a dramatic shift in terms of the scale and scope and workflow around it with the introduction of all of these agentic capabilities, especially since the beginning of this year.

3:14And I'm wondering if you can talk through some of the major notable changes that you've seen in terms of the roles and responsibilities for these data catalogs and metadata systems as agents become more of the actual operational substrate.

3:31Prukalpa Sankar:What's interesting about the moment we are in AI, on one hand, I live in San Francisco and I walk around the streets and everyone's like, AGI is here. And every day the models keep getting better. The benchmarks keep getting better. Two years ago, it couldn't pass the bar. Today, it's the top 1% of test scorers. In the last decade, intelligence has compounded 1000x. In the last six months, intelligence has compounded 2x. So by any parameter, exponential growth in every intelligence benchmark that you can have.

4:11On the other hand, I work in the real world. I work with real companies and real enterprises. And when I ask them, how are things going? Is AI useful? Almost always eyes drop. You know, somebody talks about this one pilot that they're making its way to production. The data on this could not be more stark. 56% of CEOs report zero financial benefit with AI. You know, one of five projects make it to production. And so there's a gap, whichever way you argue this, AI being valuable, we've probably proven that. AI being useful, we haven't really proven that.

4:51And so what's the gap? And this is where I like to always go back to understanding humans and what it takes to make humans work before you figure out what it takes to make AI systems work. And hidden there is in plain sight is the answer. Companies in the real world care about performance. Performance is about driving outcomes in the real world. What is performance a function of? Performance is a function of, I like to think, intelligence, but also context.

5:23If you think about human job performance, less than 10% of human job performance is explained by cognitive intelligence. Which makes sense, right? Like would you say your best employee is also the person that scored the highest on the SATs? Or would you say they're the person that, you know, takes feedback the fastest and learns and grows and, you know, just, you know, absorbs institutional knowledge and, you know, works really hard? Like, is that what the best person in your team looks like?

5:54And almost every time I say that people will say, yeah, of course, that's what the best person on my team looks like. For the last hundred years, we have built infrastructure that helps humans become great at their jobs. When they join a company, they are onboarded. Given a mentor. They go and attend and shadow other people in their team. They learn in these conversations. Their manager gives them feedback. They learn that way. They make mistakes on the job, and then they learn through that. These are all the things that it takes for humans to get really knowledge and skills and expertise that it takes to get really good at their job. And that's what's missing. Today, have AI agents that are basically the smartest intern we can ever hire, but there is no way to really onboard them onto our context.

6:43And this is what I think of as the next frontier. I think the next frontier is contextual intelligence. It's like, and I truly mean this. I don't think there is a concept of intelligence without it being contextual. It's like an academic construct. Just measuring cognitive intelligence, it's not really useful in a real world setting doesn't really mean anything. And the next frontier is going to be about how do we actually take and harness this intelligence by giving it the context that our companies have to make it useful.

7:14Tobias Macey:And to that point of onboarding and the idea that we've been doing this for a long time, I think that the fact that we have to do it at such an accelerated pace and over and over again, really stress tests the systems and protocols that we've had in place to do that onboarding where because we're working with humans who are able to do a lot of their own discovery and self learning and self direction or find the right person to go out and ask in physical space, we have been fairly lackadaisical in terms of the level of rigor and detail that we put into that onboarding setup. And so there has been a lot of delay and pain for newcomers to a team to be able to actually get up to speed and be effective where it's not unheard of for somebody to say that I don't actually expect anybody to really be fully up to speed for six months after they start, which when you really think about it is fairly insane. And so the fact that we have to do this over and over again and we're bootstrapping these agents from scratch every time forces us to be more deliberate in terms of ensuring that we actually have all of these details and

8:21protocols and information more explicitly codified for the agents to be able to bootstrap, which also helps the humans who are going through that same process.

8:29Prukalpa Sankar:Yeah, absolutely. In some ways, I think AI has shined the inefficiencies that exist in our human organizations that humans glue together. And now we realize that they're missing in a very explicit way. And that really, I do believe there will be a renaissance of organ like a new kind of learning organization, you know, like human like, what does it take to build a really successful company in the AI world will be dependent on how companies find a way to actually build new ways of structuring and building these and encoding these in the way the organizations itself run.

9:11Tobias Macey:And for the context of these metadata layers and the semantic details that we bake into them, I think another piece is that particularly for human analysts, they're able to do a lot of that information gathering, and there is that implicit knowledge, that institutional knowledge that gets built up over time where because they've had to run the same query 15 times or because they've had to ask the question of the sales team or the marketing team 15 different times to make sure that everybody's in agreement, they have the necessary context to make sure that the queries that they are producing are accurate for the organizational context where an LLM might be able to fetch some of that detail and maybe go through query logs of queries that were generated by humans to be able to glean that information.

10:01If you then persist that going forward, it makes everybody's job easier. And so I'm curious how you're seeing the nature and shape of these context layers and the metadata systems evolve to more explicitly incorporate and surface those types of details that maybe we wouldn't have bothered investing the resources into engineering into those catalog systems.

10:26Prukalpa Sankar:Yeah, absolutely. So let's unpack what a human analyst does today and what it takes for them to get really good at their jobs. Let's say you get a question and the question is something like, tell me what my top 10 new customers are. Sounds like a really simple question. Actually, a really hard question to answer because the first question is like, well, who's asking? What decision are they looking to make? If sales is asking, then it might be the top 10 customers by revenue. If it is marketing, it's probably the top 10 customers by brand and logo referenceability.

11:06If it's product, it might be by adoption of the product. And so even before we get into in the data world, talk a lot about semantics and how do we define a customer and how do we there's actually institutional knowledge that if you're an analyst and someone's asking you a question from the marketing team, you already have some sense of what this means. Then I use this word new in my conversation. What are the top 10 new customers? At ATLAN, we run our business on a quarterly basis. So if someone says new customer, it typically means a customer that signed in the last quarter. This is not documented anywhere.

11:42Nobody ever tells anybody this explicitly. You just learn it because you joined the town hall and you joined the all hands. Like, this just becomes how you think about the business. And then there is the, what does a customer mean? And how do we define customer? And how do we make that explicit? And it's going to be different in finance and it's going to be different in sales, depending on again, are you reporting to the board and or are you running like an acceleration program or a freemium program or so on? And then eventually we get down to the data itself. And how do I actually measure, like what tables do I use? What descriptions do I use? And so on.

12:20These are the four layers of first knowledge. This is the first piece that I think of. This is all knowledge. And there are different types of knowledge in here. There's institutional knowledge. There is semantic knowledge. There is process knowledge. There is user and org knowledge. This is the first step of knowledge that the human analyst brings together. Then let's make this a little bit more interesting. Typically, the analyst is not asked a question saying, what are my top 10 new customers? Typically, the analyst is asked a question that says, why did my revenue drop? Or why did my revenue go up? And this is where, for example, there is expertise that people learn. So for example, you learn that you know, July is a seasonal quarter, especially if you do business in Europe, because Europe tends to be, you know, on vacation in the summer. So the first thing that an analyst does when a revenue drops is that they first check Europe and then they say, okay, is this coming because of Europe? And is there a drop just in Europe? Is it seasonal? Is it if it's not if it's seasonal, then this is nothing to worry about. So then they check, okay, well, how are my other regions doing? Have I had a drop in pipeline in other regions? And then they come back with And this is expertise

13:32that humans learn over time in terms of the business and how the business operates. And that's the second layer of what people need, which is how do I do a what if analysis in my business? How do I think about understanding the root cause analysis of certain things that happen in my business? How do I think about seasonality trends and so on? And then eventually, then we have the tools. This is really where like, how do I run? How do I go run a query against my CRM? How do I bring a text to SQL together in the right way? This is what I think of as context. Context is a function of knowledge, skills, and tools that all need to come together.

14:09The interesting thing about this problem is that we are now at a point where AI itself can bootstrap a large part of the knowledge inside organizations. So one of the things we have seen a dramatic success in the last roughly, I'd say three and a half months is we have unleashed essentially what we call our context harness on the foundational data systems inside Novo. There's already a lot of information about the business that's hidden in the business systems.

14:45If you connect the Salesforce to the warehouse, to the BI and how people are running queries on your BI, You're able to bring that back. You have end to end lineage and then you can start reverse constructing. How does this business run? How does this business work? How do you think about all of these questions that I already talked about? How do you reverse construct a metrics tree? How do you reverse construct an ontology? Can you start thinking about seasonality trends in the right way? And so the first thing we've seen as an evolution, as we think about the context here is how do you bootstrap?

15:16And how do you bootstrap context from all the existing business systems that exist inside the organization? And the second thing that we see is how do we bring this into the development workflow of agents. And so we have now a construct we call a context repo. Think of it as a GitHub repo equivalent, but for enterprise context. It becomes the unit of portable context for a use case. And this is where essentially iteration loops, simulation loops, one of our core philosophies is AI itself should go do most of the work and ask humans for help.

15:56So can it read through for all of these use cases and say, hey, you define your revenue differently in sales and finance. For this use case, which of the two should I pick? Human, tell me. And the human approves or rejects. So how do you build those kinds of loops? And then turn this into a portable unit that then builds on the context over time.

16:21Tobias Macey:One of the major barriers to entry for adopting a metadata catalog or doing a master data management project or building a business glossary has been the activation energy required to get to the point where it's actually useful and worthwhile where for the first months or years, depending on how the project is managed, you're spending a lot of time and energy and potentially money in the process of populating all of this information. But until you hit a particular critical mass where the majority of the information is present and correct and trustworthy, nobody's going to bother looking at it. And a lot of times, you never get past that point where you've actually reached that threshold. And so it eventually will just quietly die off until the next time around that somebody says, oh, we really need this system, and then you rinse and repeat. And so I'm curious how you're seeing these agentic capabilities reduce the amount of investment and time and activation energy required to get past that threshold where the metadata system or the business glossary or the master data management golden records are to a level of completeness and quality that the business will actually rely on them.

17:36Prukalpa Sankar:Yeah, absolutely. This has been, I feel like a labor of love over the last many years for us at ATLAN. So 2022, in referring to early twenty twenty three, we were the first company to launch ATLAN AI in the category. And we kept iterating on it and it was a human assisted workflow to go generate some of this context. And we pushed the boundary as much as we could. Like, you know, honestly, were able to get about 75, 80% accuracy on this thing. It was the highest it could go. And that's when we realized that infrastructure had never been built for a world where AI was the primary producer of context. Metadata infrastructure itself was not ready for that world.

18:19And so what we did was we actually rebuilt the foundation of ATLAN completely. So now ATLAN is built on what we call a context lake house. It's an iceberg native file format. It allows us to leverage the same compute that we would leverage on our data systems. And so all metadata and traversal that an agent on Athlean can do is able to leverage this foundational Lakehouse infrastructure, which means that you can do compute level operations that you do on Snowflake and Databricks on your data, on your metadata.

18:51That was the first big architectural unlock that we needed to do to be able to get to them. And, of course, as you do that, like how do you think about graphs and how do you think about relationship traversal and how do you think about vectorization and how do you think about all of these things that it takes for AI to be able to truly work on your context at scale. That unlock was massive. Sometime middle of last year was when we announced the context lakehouse, and that was a massive unlock. The second thing that we did was we have done a very, very big investment into our own harness for context.

19:28And this is where we have realized context quality really compounds. So I'll give you an example. Honestly, you and I can today go take a table. We can go give it to ClotCode, and we can say, describe this table, give me descriptions. And Cloud Code does a kinda like okay job, like decent job, but not really accurate enough for me to push it into production. But imagine you're able to take the best AI that you have and intelligence you have, and you get it to go read your system of record and where your data was actually created in your database.

20:03Then you get it to read downstream how everybody is using it and how they're querying it and what is the usage business patterns against it. You connect all of this together, and then you have AI write a column description. A column description is extremely accurate. Today, are at a point where our customers tell us 89% of our customers say that it is as good or better than humans from an accuracy perspective. This was a tipping point. We got this tipping point this quarter with our context agents.

20:34Now, if I have high quality column descriptions, I can do a really good job with domain tagging. And I can start seeing, okay, these columns and tables belong to sales. These columns and tables belong to Crunch. Then I can read through the SQL, and then I can start reverse constructing metrics. And then the accuracy of that's pretty high. Once I have that and then I have semantics from my BI, I can start reverse constructing relationships. The accuracy of that's pretty high.

21:00And so what we have done over time is compound on the quality of this harness. We're now at a point where my customers are telling me to overwrite the human written context with AI generated context. I think we've really hit this tipping point where the quality of of what AI can do is at a point where it is it's truly production ready. And what I'm most excited by this is exactly what you said, Tobias. The biggest challenge with these programs has always been that it's really hard to get the activation energy for humans to go do this work. Everybody believes it's important work. Nobody has the time and the energy to go invest in this work. I should know, I wrote a book about it. Like last year, I wrote this whole book about how it takes to drive the people and the process that it takes to make this program successful. We studied all these implementations and we were like, this is what it takes to make this successful.

21:56And this year, I believe we're finally at this point where you don't need to do that anymore. You can actually get started, connect your systems, run AI, and within two weeks you can go live. And we're seeing this constantly across our ecosystem and customers today. And that's what I'm most excited by because context is a living compounding system. And if AI can do most of the work and only ask humans for help, it changes the dynamics of what's possible with it.

22:26Tobias Macey:Would that aspect of being production ready, the trustworthiness, obviously, LMs are notorious for making things up when they are left to their own devices. Having that context to give it grounding is necessary to avoid those situations. But then there's still that problem of bootstrapping where maybe you don't have enough context to give it the confidence. And I'm wondering how you are managing the bootstrapping of that context and managing the initial trust signals for the LLM to be able to build an appropriate flywheel so that it is building on trusted information rather than just going nuts, collecting everything it can find, making its own assumptions about details, and then putting that into the catalog that then feeds further hallucinations or disinformation as far as the ways that that data is supposed to be represented or just some of the ways that you maybe need to start with a narrower scope to build that flywheel and just the overall approach for organizations who don't already have an established set of data catalogs and semantics?

23:37Prukalpa Sankar:Yeah. Absolutely. So the way I like to define it to customers is there's two practices that you have to start in almost in parallel. The first is what I like to think of as left to right. And that's what I just described to you, right? Connect your data systems that AI do most of the work to reverse construct your business. Then there's a right to left. And I like to align this to your use cases that you want to go live with. So let's say I want to pick a seasonal revenue forecasting agent for my finance team.

24:15Now, what happens here? Why does it take really long to make an agent go live? The first, you alluded to this, it's really hard to bootstrap the context. Today, it's super easy to actually create an agent. It's really hard to give it the context that it takes to make it go live in production. I've had customers stuck in testing hell for five months where they are constantly iterating and it's a nondeterministic system. So how do you even test? They know that if they don't get to 80% accuracy, their business will typically abandon it. I had a customer the other day who got really excited. They launched about thousand Genie rooms. This is Databricks' AI agent.

24:56And within a month, 90% were abandoned because it didn't have the trust loop, the hallucination, all of these challenges that we've all seen. So the first problem is how do you bootstrap and then how do you get it to that accuracy threshold so that you can actually launch it to the business? Here the approach we've taken is take the AI first approach and use all your existing context to reverse construct and bootstrap. So we have a product that we call context engineering workspace.

25:29What it does is think of it as you just come in and define your spec. And you're typically doing this in a cursor or a cloud code or wherever you're building. And you say, this is the agent I'm looking to build. What it does is it traverses all the context that you already have and it bootstraps and says, okay, what is the top context that I need for this use case? So it already does that initial bootstrapping. The second thing that it does is it then goes and reads all the context about usage and use cases, and it generates simulation environments.

26:01So the AI itself is going and running simulations on, okay, well revenue forecasting, let me look at all the business dashboards that are already associated with this. Let me look at how these people are using it. What are the queries they're running? Based on this, let me simulate who are the personas that might be using this. Based on this, let me simulate all the questions they might possibly ask from this agent. Right? So again, this goes back to the context flywheel.

26:25Because you have that base context, you're able to do a much better job of generating these simulations. Then AI does the first job of basically saying, what do we already have context for? And what's the accuracy? It runs the Text to Sequel. What do we not have context for? So then what it does is it brings it to the humans and then says, hey, here's the ten, twelve context pieces we need to add, that we need to improve context for. And this is how essentially our teams improve and run the simulation context engineering workflow to get it to production.

26:57Once you launch into production, one of our core theses is is that this is a living, breathing thing. This is a discipline. That with AI, there is no such thing as GA. You are constantly improving your AI system. And so the last part of this, and we bring this into the context repo flow, is we pull traces and we pull memory. And then we have AI again, we have a set of curation agents that are sitting on top of traces and evals, and basically bringing it back and saying, okay, here's how we should improve context over time for these use cases. So this is really where we have customers who would say things like when we launched this agentic use case, we started with 50% autonomy.

27:44And now we're going to, you know, 90% autonomy. And the way we do that is by improving context over time for the system.

27:54Tobias Macey:With the fact that you have humans who are collaborating and agents who are collaborating, what do you see as the value in identifying the originator of a piece of information? Is implicitly more trustworthy if it's a human operator who is making a change, or do people typically see that because the agent is going to be more tireless and maybe have more attention to detail that I should trust what the agent puts in and not the humans? It's just some of the ways that you surface that level of detail as far as who provided this information, at what point, and what is the grounding for that level of detail, and then how you manage the the change process for adding new details or adding new fields or new metadata?

28:38Prukalpa Sankar:Yeah. So I think the base starts at just identity and tracing of identity and being able to say, okay, this is something that was generated through an AI. This was generated through a human. I am increasingly beginning to believe that we are actually going to go from human in the loop to human on the loop. And what AI is able to do, humans are just going to be the blocker to AI going into production and leveraging the scale. Like, again, you know, we have AI within a week doing context generation for millions of assets. Like there's no way a human is going to be able to like go approve, reject all of this. Right? That's just not possible anymore in the real world. So one of my mental models has become how do you have AI do all of the work and bring it to humans for governance and decision making. So for example, one of our most popular agents is an agent we call a metrics conflict resolution agent.

29:43And what it does is it basically goes, reads through all your code and your SQL and all this context that you have. And then it'll basically say, okay, hey, these are the metrics that you're literally defining differently in two parts of your organization. And that's a human decision. You know, like AI cannot make that decision for you. And some human in the company needs to decide that when we are reporting to the streets today, we decide that we are going to report this versus this, Right? And so that's a human decision and that needs human governance.

30:16But is this the most accurate description of this? Or is this how we're defining the metric? Is this the most accurate definition of it? I think AI can actually do a really good job in the right way. Again, like obviously not if you go and just throw this into Claude and expect it to give you a result. But if you build the right kind of harness and the right kind of guardrails and checkpoints and the right kind of AI systems to do it, I think AI can do it pretty well.

30:40The second thing is traceability is really important. So one of the reasons actually for building the lake house in many ways was actually that versioning, lifecycle management, all of these things that we've actually dealt with from a data world perspective will now start becoming very important from a context world perspective. So for example, agent made a decision, and it was a wrong decision. You will want to reverse state. Right? You will want to go back to, okay, which version of the context that did the agent use to make this decision.

31:11And so how do you build these same kinds of systems that we've had in the data management world, in the context management world, and how do you bring back a lot of those paradigms in the right way, I think will become even more critical. Again, it's a discipline. And how do we build the disciplines around who's allowed to approve what inside what organization becomes really critical. And we've been talking about data systems, but give you an example in, you know, a lot of our customers now use our Context Platform, not just for data agents, but also for operational agents.

31:46And so one common example that we'll hear as they manage knowledge and skills is brand voice. And the brand voice agent is so important in all these downstream agents like customer support agent, you know, the SDR agent that's on your website and so on. And typically, there's only one person in the company who's allowed to make a change or approve the Brandvoice skill and the changes that happen in that Brandvoice skill. Right? And so how do you think about these human governance loops inside organizations, ownership, maintenance, propagation?

32:20If something changes upstream, how does it propagate downstream? Those are all challenges that context management will solve over the next couple of years.

32:29Tobias Macey:And as I was preparing for this conversation, I was looking through some of the materials that you and your team have been publishing, and you make a differentiation between the idea of a context layer and the idea of a semantic layer. And I'm wondering if you could unpack some of the different gradations of what it means to go from, I have a metadata catalog to I have a semantic layer to I have a context layer. And especially when you're dealing with AI systems where the model itself needs to be something that is reflected in terms of the overall data catalog and also the fact that a lot of these AI systems maybe require specialized data stores, particularly in the form of vector indices and just how that layers on different technical elements of what needs to be present to have a fully blown context layer.

33:23Prukalpa Sankar:Absolutely. Yeah. So the way I like to layer this, a metadata catalog in its true form was about data context. So it was saying, okay, my data needs to have context associated with it. So I need to be able to describe my table. I need to be able to describe my columns. I need to have lineage that's associated with these data objects. And I need to be able to understand how my data itself or my data context ecosystem works. And AmeriData Catalog is the foundation for being able to do that really well.

34:00The layer on top of that is what we think of as the semantic layer. And the semantic layer is bringing one layer of meaning to the data itself. Right? And so this is where we say, okay, well, how do we define customer? How do we define new revenue? And that's both the conceptual definition. This is where, you know, you will hear people use words like business glossary or metrics catalog and so on. And second, it's the actual, how do I actually calculate this and encode this in semantically?

34:35And eventually, how do I execute this? So, you know, what is my execution query paradigm? This is where you'll see Snowflake talks about a concept that they call the Semantic View. You have some very specific query providers like Kube, for example, that will say, hey, run the query across five or six of my different tools, my BI tools, my warehouse and so on, right? And so you would see that as the semantic layer. On top of this, you need to actually add when you think about context.

35:05Context is you do need data context and you do need semantic context, but you also need knowledge context and you need procedural context. This is where, you know, your your institutional knowledge, your skills associated with it actually start layering on. And that's when you build a true context layer, where you bring data context, semantic context, knowledge context, procedural context together into that one core overall ecosystem from a context perspective.

35:38This is also where, and you alluded to this, the primary consumer of the context layer is an AI agent. And so technically, this means, well, how does AI traverse context inside my organization? This is where, you know, how do I build hybrid search capabilities? How do I ensure we have vectorization that's associated with it? How do I build the right kind of frameworks? At what point? And what's the right protocol, right? This is where protocols like MCP or agent to agent, A2A.

36:15At what point should AI just actually use SQL as an interface and etcetera? So how do you think about these interfaces, the protocols, the traversal, the graph relationships? All of these start becoming really critical from a context layer perspective. Does that help? I think it dubs the mental model of thinking through the, all the layers.

36:34Tobias Macey:Yeah, it absolutely does. And I think it's also worth exploring a little bit more some of the details of the maybe data modeling and semantic modeling that needs to be factored in when you do incorporate the LLMs and the ML models themselves into the overall catalog and the organizational context to say, okay. Well, I have this model that is being deployed in the context of this agent. This agent is using these data sources to produce these decisions and how that maybe changes some of the ways that the catalog itself needs to be able to store and represent the primitives of the information that we would typically have just relied on, what are the tables and how are they linked together.

37:19Prukalpa Sankar:Yeah. Absolutely. And, I mean, I don't know if this is an evolution of the catalog alog or Semantic Layer or if it's just the new, like, you know, I like to think of the context. Like sometimes I tell my customers, I'm like, if you're using a traditional Catalog or something like just, we can just connect it and we can bring it in into the context layer. And the context layer becomes your living compounding system for your AI agent. Your catalog just becomes something your humans are continuing to use for as long as humans continue to be the primary user for data analysis purposes, right?

37:51And so I don't know if the right question is how much of this needs to go back to the catalog. I think the right question that I like to think of now is how do you manage context as a living, breathing asset inside your organization? And there, I think these concepts of traces and people start using the word decision traces now are really critical to bring back into the observability loop. So like, I like to think of it as there's a context repo that is the center of it.

38:24That's the central point for this AI agent, which brings knowledge, skills, and tools that that AI agent needs to do its job really well. Outside of the context, all the agent needs is it's the model and the harness. These are the three things that make an agent. And then this context is a living compounding thing. So it has a maintainer. There are probably governance loops that are associated with this context. The context is changing both from the left because your business is not static. So your business is changing. And the context is also improving over time because with traces and observability, you're able to say, okay, how do I improve this context? So I improve the level of autonomy as I move towards an autonomous system.

39:11And that's really the unit of the discipline that needs to get built as humans go from being the people who do the job to people who context engineer these systems. So these systems can do this, the job as well as a human can maybe better.

39:30Tobias Macey:And as you have been working with your customers, and especially as you have been going through this evolution from the first version of the ATLAN product, we are going to focus on being the source of truth for all of the data that exists in an organizational context through this disruptive time of AI and agentic capabilities to being this context layer that also has all of that core data context and data semantics? What are some of the most interesting or innovative or unexpected ways that you're seeing teams manage the creation and application of the this agentic context and data context and to empower their businesses?

40:16Prukalpa Sankar:The best use cases we've seen are where companies are starting to think about their organizations as how do I build platforms for autonomous organizations or compounding organizations? So the way I like to think of it is, are agents in your company doing most of the work and asking humans for help? And how do you get there? And if you think about the paradigm to get there, organizations are at different slopes of majority, right? The starting point is humans are using AI for improving their productivity.

40:54And then you start having your first few truly agentic use cases go to production, right? And you have a customer support agent that's doing more work than the humans that are responding to customers or so on. And then you start moving to like a truly compounding system where, you know, you have, you know, tens and hundreds of agents that are doing work and then coming to humans for help. The most exciting customers that we have been working with have been really building the foundation of this.

41:31You can call it a frontier company, autonomous company. And almost all of them are building this primitive that I like to think of as the company brain. And what I'm most excited about this company brain is that it might allow us to move past the human inefficiencies that existed inside organizations. I was talking to a large telco customer of ours the other day and they said customer support, if I really want to solve my resolution time, I need to solve my network problem.

42:07And these are typically disconnected problems inside companies, right? Like typically you have resolution that's dealt with in contact center, you have network that's dealt with somewhere else. You have, you know, discounts handled somewhere else. And most of this is because of the way organizations are created. And organizations have these different silos and siloed ecosystems that are built. AI can actually allow us to break the cycle. If you learn something from the ground, from a customer perspective, AI can hop from the individual customer use case all the way up to the root cause of that problem and solve that problem.

42:49And, you know, do you even need do you even need somebody to make a call anymore to report a problem? Can you actually proactively fix the call? And there is a world in which you can see that world happening and not too far away. And I think that can just dramatically change how organizations function and learn. And so those have been the most exciting use cases where companies are really rethinking their foundational infrastructure how and they operate, and not just slapping on AI on top of existing processes and

43:21Tobias Macey:work that they're already doing. And in your own work of building and evolving the Atlin product and understanding the needs of the ecosystem as it continues to shift and grow and change directions, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?

43:43Prukalpa Sankar:The most interesting and useful one has been internally transforming into an AI native company and really pushing the boundary of the true problems that companies face as they work towards building that frontier organization. Our roots at Atlin has been we were a data team ourselves. We never meant to build a product for anybody else. We tried to buy a product quite honestly, and we couldn't find something that solved our own problems. So we eventually were kind of forced into building Athlean for ourselves.

44:18But that DNA is really important to us because we've always been very close to the problem. One of the reasons why I think we've, we've grown as fast as we have, and we're, you know, the fast growing company in our space and all these other things, there's these accolades that we get. They're all a function of just, we were closer to the problem and we cared very deeply about the problem. And so when AI happened, it was really important for us to go back to being really close to the problem. And thankfully, we are a company that is at scale. And so we have had the ability to have a front row seat in transforming how our company operates, right? You know, engineers at ATLAN don't code anymore.

44:56They only teach AI how to code. Marketers at ATLAN don't run campaigns anymore. They're only allowed to teach AI how to market. And so I think we've been very on the frontier with that. And that has helped us really, I'd say, grounded in what are real problems. How is context management? We started talking about this context thing now, well over a year and a half ago for how, you know, AI systems are going to need context management systems. And the reason we were able to get there ahead of the curve of where the market is, is largely just because we face the problems of the pain that it takes to productionalize these systems. So that has been, I'd say, coolest part of this journey. Also seeing how people can become superhumans, I would say, when people unlock themselves with AI systems. There's so many there's so many stories that we have in this world right now around AI, around, you know, like, what is it going to do? Is it going to take away human jobs? You know, blah, blah, blah. And I can't, you know, I can't prophesize on what will happen in the world in ten years. But what I do know now is that there are enough problems to be solved in the world that we have not found a cure for in humanity.

46:06And humans that allow find a way to turn themselves into superhumans with AI will be key to being able to do that. So just seeing humans unlock themselves has been beautiful to see. Seeing our customers have their own ClaudeCode moments when they turn on context agents and they tell us things like this is I had a customer last week who said, this is a 404000% increase to what we did last year. And so, you just see the possibilities of what can happen and you see people come alive. I think that has been really cool.

46:42And the last one, the most unexpected one has been that there's so much education to be done right now. In fact, technology can scale exponentially. They say organizations scale logarithmically. But humans, our slope is not the same. And right now there's so much market noise. There's so much hype. And so how do you make sure that people are educated, authentic, are able to trust like, you know, I honestly think it's I would hate to be a buyer of technology right now. Would it sucks, You know, like everybody is trying to market everything and it's very hard to like really tell the difference between, you know, reality and hype.

47:30And my favorite push to customers has been just like, are you building? Whoever you are, you could be like CTO, CIO, CDO, are you building? Did you build on the weekend? And that just changes. You don't need to believe anyone. You just know yourself on what it takes. And so seeing that also come alive and having a part to play in that has been just amazing.

47:57Tobias Macey:And I think too that while the pace of change and the pace of new products hitting the market has been massively accelerated because of the fact that agentic coding speeds your time to delivery. It also, I think, makes it more tenable to do that evaluation of technology selection. So for instance, I was recently looking at, do I want to use Apache Doris or StarRocks, which originated from the same code base but have diverged in terms of their overall project goals. And so you can just say, okay. Well, here, Copilot or Claude code. Here are the two code bases. Do an analysis of the actual code that's there, the GitHub issues, the pace of change, the types of projects and pull requests that are being opened, and give me a comparative analysis of what these two projects are doing since they have diverged and which one I should be focusing on for this particular use case. And two years ago, I would never even even bothered really doing that. I would just maybe read through some of the documentation and make a best guess and hope that I made the right choice.

49:03Prukalpa Sankar:Yeah. Absolutely. I mean, it's interesting how much agents might become the primary buyer of most things in the world. I think we're already seeing this, like if you look at the data from some database companies and you look at, I'd say, the percentage of revenue that is starting to get generated directly from just Cloud Code setting up and using databases. And I'd say that's the first place you would typically see this change is in the developer ecosystem, right?

49:34And I think it will start following in other parts of the world. And in some ways, I think it would be a it's a positive change, right? Because it allows you to the best product wins, the most useful thing to the consumer wins. And I am really excited about by all the new paradigms that that will create in just the way businesses are on.

49:58Tobias Macey:And so for people who are interested in figuring out how best to leverage the data that they have, figure out what data they have, what are the cases where you would argue against this very agent native context layer and instead focus on building out the maybe more traditional data catalog, either as a first step or exclusively maybe for a particular level of scale or complexity?

50:30Prukalpa Sankar:I would say if you are a company where you have or an organization where you have not rolled out an Ecopilot to your team and you're predominantly in sort of this era of humans are doing most of the work and at the best case I want to drive human productivity from from this, then I would say focus on maybe getting your foundational data catalog right and stay there. I would still say you should not be doing human data stewardship and you should be using an agent tech way of being able to build it, but maybe just the consumer being a human is maybe the way I would think about it. However, as I say this, I will also vehemently say, I do not believe that any organization should choose this strategy.

51:26Because foundationally, the bet of data has always been that you want to democratize data to help business. And this has been the dream, like for the last maybe twenty, thirty years, this has been the dream. And the challenge that you've always faced is that in the old world, you needed to teach business the language of data. And this was, you know, usually to these data literacy programs and all of these things. And for the first time with foundation models, you don't need to do that.

52:03You can actually teach data the language of business. And so if you think about yourself in the shoes of the business, what does the business care about? The business doesn't actually care about data. The business cares about running the business and doing that really well. And finally, from a technology perspective, we are there. And I believe that is the dream of any data practitioner as to why they do what they do. And we are finally at that frontier. And so if you're investing in any technology right now, there is no downside in just starting at and leapfrogging the generations. You don't need to go gen one, gen two, gen three. Think you can just leapfrog to gen three and miss the chaos and the trauma that gen one and gen two brought to you and just move directly to gen three and roll this out in a way where it will drive a true AI native kind of adoption inside the organization.

53:02Tobias Macey:And as you continue to build and iterate on your product, what are some of the predictions that you have for the next set of architectural shifts that we're going to be seeing in the data ecosystem that are driven by the pressures of AI powered systems?

53:18Prukalpa Sankar:The first prediction I have is there'll be a lot of change. And it'll be almost impossible to predict the change. Change. But change is inevitable and so organizations that build open non lock in ecosystems that allow them to evolve very quickly will be very valuable. Like just let's talk about skills. Skills as a paradigm did not exist six months ago. Six months ago. MCP as a paradigm did not exist a year ago, you know. And so we're just dealing with the rapid pace of change and evolution harnesses what I thought was a best in class AI system in October of last year versus what now is old school way of building AI systems. It's been six months.

54:19And while you could feel some cost bias and say, hey, you know what, like, I just spent all this time building this thing and now I have to rebuild the whole thing. Or you could say, change is the only reality and I'm going to keep rebuilding and changing along with times. And so the first prediction I have is that teams that change fast and build a culture that allows teams to change quickly and evolve quickly will be teams that win. And the second prediction I have is there will be a lot of heterogeneity that will get created in the agent tech stack.

55:02There will be, you know, dozens of agent tech platforms, hundreds of agents inside organizations, and it will be a new kind of heterogeneity than we've seen before. In that world, what I like to tell customers is context is your business IP. Keep it your own. Like don't lock it in into any individual agent. Don't lock it in into any individual system. Make sure that it's open, it's interoperable. We are starting to already see customers tell us about how they've context engineered these two different agentic platforms differently. Each of these systems have memory.

55:41So now they're all speaking a slightly different version of the truth. This was a classic data problem. People used to say, you ask sales and finance a revenue number, you'll get two different numbers. This is just happening at far larger scale for truly autonomous kinds of systems already. We're already seeing this happen. And so what you want to ensure is that you have an open context ecosystem where you're not locked into a single agentic platform or a single model ecosystem.

56:07Yeah, I'd say those are my two biggest predictions for the next, I don't know, six months. Is that is that a long enough horizon?

56:15Tobias Macey:I I think that's about as long as anyone can hope to look forward. And so are there any other aspects of the work that you're doing at ATLAN or this overall space of context layers and the shared substrate for agentic and human workloads that we didn't discuss yet that you'd like to cover before we close out the show?

56:35Prukalpa Sankar:Maybe the one last thing is that I am really excited about the future of the data practitioner. Data people are the only people inside organizations that have dealt with non determinism before. And I don't think I realized what that means until I truly productionalized so much of AI. And I realized that, you know, like software engineering, you think in binary, right? Like it's zero one. And, you know, if you've worked on a machine learning system, you've worked on a data engineering platform, you have dealt with non determinism.

57:10And there is a foundation where data practitioners have an extremely important role to play in the AI world. It will need change, it will need learning, It will need, you know, there's open questions like who owns AI platforms inside companies? Who owns context platforms inside companies? How does this play out? But I think there's a lot to learn from what we've just gone through in the last decade of data. And

57:39Tobias Macey:I'm really excited about just the role that data practitioners can play in this new world. All right. Well, anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology or, I guess, knowledge that's available for data management today.

58:02Prukalpa Sankar:I'd say the biggest gap is the confusion more than anything else. There is so much confusion right now in the space. And I mean, not to say that there are not technology gaps. Our technology will evolve pretty dramatically, you know, in the ecosystem over the next couple of years, I'm sure. But I think the biggest gap really is how do you get a trusted verified source of context about what to believe and what to not believe in the ecosystem. And how do you make the right decisions about tooling technology investments.

58:43So for example, one of the things we've started doing is we've actually been running a WTF is the context layer series, and we're trying to just demystify some of what you and I just did, right? Like, you know, what is the difference between a semantic layer and a knowledge graph and a graph database? And these are pretty deep concepts to go deeper into. And so my hope for the ecosystem is over the next couple of years, we end up seeing a lot more real life implementations that create trusted spaces for the community.

59:19Tobias Macey:All right. Well, thank you very much for taking the time today to join me and share your thoughts and experience on building these shared context layers for enabling agents and humans to collaborate on organizational data problems. It's definitely one of the perennial challenges and definitely one of the faster moving spaces that we have to deal with today. So appreciate all the time and energy that you're putting into making that more attractable, and I hope you enjoy the rest of your day. Thank you.

59:53Thank you for listening, and don't forget to check out our other shows. Podcast.net covers the Python language, its community, and the innovative ways it is being used. And the AI engineering podcast is your guide to the fast moving world of building AI systems. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@dataengineeringpodcast.com with your story. Just to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.