Episode 122: [Value Boost] 4 Questions Every Data Scientist Should Ask About AI Strategy

[00:00:00] Dr Genevieve Hayes: Hello, and welcome back to Value-Driven Data Science, where data professionals become strategic experts. I'm Dr. Genevieve Hayes, and I'm here again with Victor Coimbra, Global AI and Data Engineering Lead at AI consultancy Artifact. Last week, Victor and I discussed how the shift to hybrid human AI agentic operating models fundamentally changes what organizations need from their data professionals. [00:00:32] Today, in this Value Boost episode, you'll learn a four-layer framework you can use to diagnose where an organization's AI strategy is breaking down and discover where the real strategic decisions of the next two years will be made. Welcome back, Victor. [00:00:50] Victor Coimbra: Very glad to be here again [00:00:52] Dr Genevieve Hayes: Many organizations currently have AI strategies that read a lot like shopping lists. [00:00:58] To them, AI is just another software decision to be approved. Their biggest concerns are whether to choose Anthropic versus OpenAI, and how many Copilot seats they should add to their existing Microsoft bill. However, when it comes to rolling out agentic AI, the application layer is just the tip of the iceberg, and without considering what's below the surface, these organizations are setting themselves up for a world of pain. [00:01:27] Fortunately, this pain is something data scientists are well-placed to help their stakeholders avoid, but only if they know the right questions to ask. Victor, you recently wrote an article in which you outlined a four-layer framework that describes what organizations should really be considering when building their agentic AI capabilities. [00:01:49] Can you walk us through what each of those layers involves? [00:01:53] Victor Coimbra: For sure. So I like to think as a iceberg, as you said, doctor, the first layer is the application layer, and that's what everyone sees, when you look at the iceberg [00:02:03] so what tool I'm using. Do I am using Claude? Do I'm using CoWork? Do I'm using Copilot, and that's easy to see, but like in any iceberg the true nature of the work and the strategic impact is gonna be on the other layers that are below the sea. And the first layer below the sea is what we call the AI platform. [00:02:26] And in here lives three big things. The harness, the context, and the observability layer. That's your AI platform. Is where you're gonna decide how to manage the power of the LLMs with the harness. Is how you're gonna structure the context of your company, and context in here is super important. [00:02:45] Context is not the same thing as data. Data is maybe 10% of the context that you produce in your company. And finally, the observability layer, where I gonna understand the token lineage, where people actually spend the token in, the token out. And these three things help us to understand the full picture of what you are actually doing on your company. [00:03:09] So we are still on the very surface. This is still something that we deep dive a bit on the second layer. But there is two more below this that's become harder, that's your inference layer, and in here is how you control your inference token, is that you're gonna use open source model, you're gonna build your own models, you're gonna use something that's besides the LLM. [00:03:29] For example, visual models, word models. And then the fourth layer, and the most important one is the hardware layer, how do I control my hardware? And what I mean by hardware is how much do I have the power to actually produce intelligence, [00:03:45] and you're gonna see that most of the advanced big tech companies like SpaceX, Meta OpenAI, Microsoft, Google, they all invest a huge amount of money on data centers, and the reason for that is because they want to control this hardware layer, it's gonna be almost the electricity layer. [00:04:04] For some companies, might not be the goal,. But for a company that wants to be very tech-driven and AI-driven, at least have a strategy of how to handle these four layers, for some layers, you can outsource But other layers you need to at least outsource in a conscious way. [00:04:22] And today, I see that is most ignore and people don't take this as a conscious decision and only choose to see the tip of the iceberg and not actually deep dive on everything that's below the sea [00:04:36] Dr Genevieve Hayes: Why do you think that's the case that all these companies are focusing on the tip of the iceberg given that those bottom three layers are clearly critical from a strategic point of view? [00:04:47] Victor Coimbra: I think is where it shines the most, and I think we naturally are driven by short-term gains. And that's the truth of humanity, and I think [00:04:56] corporate tends to always look on the short term, and the true value is on these long-term gains. And technology prove us that, I have this phrase that I really like that's never bet against technology, because in the long term, technology always win, [00:05:13] and sometimes we like to bet against technology because we like short-term gains, but true transformation takes time, and true transformation takes effort for you to deep dive, open the black box, and actually see what's going on down there [00:05:30] Dr Genevieve Hayes: As I mentioned in the intro, if an organization builds their entire AI strategy at the application layer, then they're setting themselves up for failure. When that inevitable failure does occur, is there any way a data scientist can use this four-layer framework to diagnose where an organization's AI strategy is breaking down? [00:05:52] Victor Coimbra: The first thing is to identify when the company's failing. And I think there's three main symptoms that proves that probably you are take the wrong decision, the wrong layer. The first symptom is that people are not adopting your tool or your technology, like they are actually not using. [00:06:10] And sometimes we think it's because of the people that we have, but actually it's a totally different opposite way. Probably the tool that we got is not the best thing for the job. The second symptom is a governance problem, and we're gonna see a lot this in the very near future some leakage, some people upload something that should not, some decision that AI took that should not be revised by a human. [00:06:35] It's very important for us to remember that AI cannot get fired, and I think sometimes we forgot that, and I think that's the second symptom, when we start to have security and governance problem. And the third symptom, and usually is when people start to act, is the explosion of cost and tokens. [00:06:51] Uber, for example, spent in three or four months, the entire budget for the entire year in tokens, this is probably the first three signs of the apocalypse, the first three signs that probably the company's, is taking the wrong decisions. And when a data scientist identify those, the first thing is to understand, okay, on these four layers, where we don't have anything, and they're gonna figure out probably on the first layer below the sea, that's the AI platform, you probably don't have anything, [00:07:20] because nobody thought about building a AI platform in the first place. They just thought about buying license. And then they're gonna start to dig and th- gonna start to figure out that nobody thought about other models besides LLMs, and there is, there is fine-tuning. There is ability to open source models. [00:07:37] There is ability for to explore new things, and this is gonna figure out that this takes and needs computing, and I think that's where people go to the fourth layer. So I think that's super important, and I think the first symptom for leaders that are listening to us as well, is that when your team stop to work because Claude got out of service, you probably have a problem right now, [00:08:01] because your entire operation cannot be hostage of a thing or a solution that you do not control today. So you have to have at least a strategy of how to tackle that. [00:08:13] Dr Genevieve Hayes: In your article, you reduced each of the four layers down to a single strategic question. What are those four questions? [00:08:21] Victor Coimbra: The first question, the first application layer is for you to understand what people actually use AI. And I think on the previous episodes that we discussed, that's where the big question is. Identify the archetypes of usage that AI itself how do I use AI? [00:08:37] The second one, the platform layer for me, is do I have visibility where my token is being spent? And I bet 80% of the companies do not know, they buy tokens, but they, do not know where they are spending these tokens. [00:08:53] It's the same thing as you buy electricity or petroleum or oil, and you never know when to use it, it's the same rational, I don't buy electricity if I not know where exactly I'm gonna use. [00:09:03] And then the third one is, do I control my intelligence? Do I control what the models are actually producing? And you're gonna figure out that probably nobody actually control that. You are probably waiting for Claude to update their models to be better, and you're actually not thinking about a true intelligence strategy [00:09:22] it's not about use the most intelligent model, but about to use the best model to the best task. Sometimes it's Claude, sometimes it's an open source model, sometimes a model that some random person built in the academic world across the globe. Do you control your output of intelligence? [00:09:39] Probably not. And for me, the fourth one is, in the hardware layer, is do I control where my intelligence is created? And probably control part, that's when the intelligence created inside of your human employees, much more creative intelligence. But you probably do not control where your intelligence created when we're talking about agents and AI. [00:10:03] And here there's a big topic about sovereignty. If you do not control where your intelligence is created, tomorrow there is probably a regulation, or there's probably a new law that's very new that can block it, the, your entire operation from scratch, because we don't know where this actually is produced. [00:10:24] It's the same thing as electricity, you need to control, as a company, where do you produce probably the most important resource that you're gonna use that is intelligence, I think that's the main questions for each one of those layers. [00:10:36] Dr Genevieve Hayes: And if a data scientist only had time to ask their stakeholders one of those four questions which would you tell them to lead with and why? [00:10:43] Victor Coimbra: The first one, and the reason for that is because it's where they're gonna be most impact in terms of actual work. I think it gonna take some time for leaders and companies to understand the other layers. But today, if you need to ask one question, the question should be where and how should I use AI on the best way possible? [00:11:07] Because you're gonna see that different ways of using AI are gonna require different skill sets that you as a data scientist needs to develop. [00:11:15] Dr Genevieve Hayes: And that's it for today's conversation with Victor. If you haven't already, listen to our previous episode where we discussed what hybrid human AI agentic organizations mean for data science. Thanks for joining me again, Victor. [00:11:31] Victor Coimbra: Thank you very much. The pleasure is mine [00:11:33] Dr Genevieve Hayes: And for those in the audience, thanks for listening. I'm Dr. Genevieve Hayes, and this has been Value-Driven Data Science.