Are we creating AI solutions or just chasing problems?

Guido (00:01) Welcome to CRO.Cafe. I am Guido Jansen and this episode is part of a mini series I'm recording ahead of Space Academy Experimentation Leads event on practical AI experimentation, May 7th in London. And my guest today is Juliana Jackson. Juliana is cloud director of data science at Jellyfish. She teaches data storytelling at IHM Business School in Malmo.

she co-hosts the Standard Deviation podcast with Simo Ahava, one of the many tracks you can know Juliana of. Last year she also won with her team a bronze Lovie the People's Lovie and two BIMA bronzes for a Starbucks project. They deliberately built on a small language model.

which seems to be becoming a recurring theme of this podcast series. Very good.

Juliana (00:49) Remember when we were

talking about it? We didn't know back then this is gonna be so, so sexy. I didn't know either. Yeah.

Guido (00:58) Your LinkedIn tagline is "Known for saying things that others won't", which is exactly why I wanted to here on the show. And also, of course, why you are running the whole day at Space Academy as mission control host. Juliana, welcome to CRO Cafe.

Juliana (01:14) I'm super happy to be back. It's been a minute. It's been, it's been some years. Yeah. But always.

Guido (01:17) It's been a minute, it's been a minute. You've

been at Jellyfish now for about eight months, cloud director of data science. The company is in Copenhagen, The office, yes.

Juliana (01:26) Yes.

Yeah, so it's a global company. There's like

a lot of people everywhere. I'm hired by the Swedish office, but I go to Copenhagen because it's closer to Malmö than Stockholm.

Guido (01:42) Close the bar. Yeah. Yeah.

Momoko, I think it's a very easy trip with a train, right? Across the bridge.

Juliana (01:47) yeah i couldn't go

to Stockholm it would be six hours with the train

Guido (01:51) Yeah, exactly. Much longer trip. Hey, and so you're working with enterprises on Google Cloud and AI. What have you found yourself saying to clients more often than you expected to? What's the conversation that keeps coming up across multiple clients?

Juliana (01:54) Yeah.

That's a good... Let me formulate that answer. I think maybe one of the...

Guido (02:13) you

Juliana (02:16) Common teams would be obviously around AI because everybody is trying to implement AI as part of their workflows and as part of their systems. And we're always talk about governance and guardrails. Like how can we be in control of what is the output? How can we be in control with different things that we do? Because if you think about it, like, if you think about traditional data warehouse in general is built for reporting purposes. was never built for this autonomous decision maker.

reasoning type of tool that an LLM is and or an agent is. So now I guess the transformation is happening from going to a traditional data warehouse to a GNI ready data warehouse. So it's a lot of conversation about what are the layers that you need to be able to deploy AI, to be able to deploy different agents. So that comes a lot in the conversation.

you know, besides classic things of how do we build different audiences? How can we target Bitbearers on different platforms? But yeah, that's it.

Guido (03:20) Yeah. Yeah, I suppose, I

suppose also a lot of, a lot more like live retrieval of, of, of data than before. I mean, if you generate a report, the report can remain the same for like a day, but with live retrieval, you need much more, um, CPU power, I guess, behind it. Uh, and maybe data bandwidth, uh, to, to, uh, cope with that. Right.

Juliana (03:45) No, for sure. And you need these real-time systems to be able, if you want this agentic layer on top of your operations, you do need this live layer of data that agents can go and input so they can output across your systems. There is, of course, a lot of need for data. But again, this

the question of who's actioning it. So sure, we're going to say agents, but I think one problem that I still see after all these years that since we're talking and it's the same thing, now we have agents and we have AI, but we're trying to solve problems that have been already solved in the past by different things. So somehow I do feel that we are seeking problems for our solution instead of being at the other way around when you actually build

something because you want to innovate, because you want to solve something, now we're just taking the AI in our hands and we're just chasing problems in need of a solution. So yeah, it's complicated to be, I don't know, this is a very weird time, I guess, in our industry.

Guido (04:55) Yeah, I think in line with that, you wrote a piece earlier this year that opened with the line that the distance between what gets funded and what actually works has never been wider. Over $200 billion went into AI last year. From where you sit talking to these enterprises every week, what fraction of that is going into things that will still exist in two years that we will still use and maybe from

the perspective of the experimentation crowds, where are we still kidding ourselves when it comes to AI?

Juliana (05:32) I think I was just recently a few days ago reading this research from Gardner and it was basically saying that 60 % of all companies that have been investing in AI by the end of 2026 will abandon most of their AI projects because of the lack of foundation, know, data foundations, data warehouses, systems that have, you know, guardrails in place and so on.

I think it's a, I think, you know what, it is an age of exploration. think for the past 20 years, we had this age of optimization. We were just trying to fix things, but now I do think we're in an exploration phase. So a lot of testing is happening with this models with different, you know, tools and different technologies and so on. Well, it's kind of like they're great, but is it useful? Is it the question that you have to ask yourself? Is it really useful? I don't know. I probably is not. I do see a lot of.

Guido (06:23) Mm-hmm.

Juliana (06:28) It's like, cause I'm, I have ADHD, so I have a closet of abandoned hobbies because we like to start a lot of things all the time and start doing them, but we don't continue them. So I do feel like this whole AI thing is very similar to ADHD. It's a closet of abandoned ideas and level that you build. And actually it could give you a good example. I was scrolling LinkedIn.

Guido (06:45) Yeah.

Juliana (06:53) some weeks ago and I was feeling so like bad with myself because I haven't downloaded code in my computer to run specific things or build new things. And I was like, wow, like am I really falling behind? Like I used to be at the edge of this stuff. I used to build things, but like you see all these people that are not even technical and they're building these random tools. And then I started thinking to myself, like what do I want to build? And I realized like I don't really need anything else than what I have. So why would I start building?

I don't know, let's say I don't like Zoom anymore, so I'm just going to build a new Zoom. Where I don't like my Google Docs or Google Slides, so I'm just going to build a new tool. Back to your question, it makes you think because there was this HubSpot post about how their stock dipped because a lot of people are using Loveable, are using Replet or whatever to build new tools.

Guido (07:43) Mm-hmm. Yep.

Yeah, or building custom in-house tools, right?

Juliana (07:52) But the thing is this.

Yes, you can build the shell of it, but what is you need to think about the cybersecurity, about operations, about all the system engineering that requires. So like if you think about how much HubSpot would pay for people, for systems, for tools to keep, make sure everybody's data is safe, that you have delivery, that you have support, that workflows are working, like nobody would afford that type of thing, especially if you want to scale, if you want to build something and use it for yourself.

That's fine. But building a product is not just building the interface. That's the thing we tend to confuse the interface because it's something palpable. It's an abstraction of what is behind. But that's not only the thing. Like there's a lot of software engineering and system design principles in the back that we don't know because we're not software engineers. It's an interesting market. Yeah.

Guido (08:47) Nyeh.

Yeah, and I definitely also have that. You don't see it on camera, but there's a whole closet here, a wall of closets with abandoned IDs. very, very, very familiar. And you also said on, I think it was the analytics power hour, that's a good, it's the wrong question to ask if AI is good at data analysis. And your point was that,

Juliana (08:59) Exactly.

Guido (09:19) LLMs are probabilistic, right? So then if you ask an LLM the same answer twice, it will give you a different answer each time compared to linear regression today and tomorrow, it will give you the exact same answer, assuming the data is the same. So if we're not asking, if we shouldn't be asking that question, what should, especially experimentation teams, what should we be asking? Where does the...

Where does the LLM belong, big or small, where does the LLM belong in the analyst workflow?

Juliana (09:50) Hmm? Hmm?

Yeah, no, this is an excellent question. think a big problem we had in conversion optimization or experimentation was that we had tons of unstructured data to analyze and to come up with new hypotheses. So LLMs are really, really good at, let's say, survey analysis, cluster pattern analysis based on textual data, interview data, customer support data. There's so many nuances of text data that you can use right now and analyze it at scale, or like the word that I like to

used for LLM analysis, synthetis. You can synthesize a scale.

That's the best use case of this. Like this is one thing that we don't invest enough, I think, in experimentation is how we build our hypotheses and how we formulate them and whether we have enough data to come up with a new hypothesis based on whatever observations that we do. But this kind of expands the level of observation where you can come up with new ideas and ways to look at things. And that's what I would use an LLM on.

You have search data, you have Reddit data, you can scrape things so much faster now to get more qualitative data to your experiments. think we have enough quantitative data, enough funnel analysis, enough of all of that stuff that doesn't even make sense anymore because nobody just peruses the internet and ends up on your website. Like, Talia Wolf was saying something, I see something on YouTube, I'm going to verify on Reddit if it's true, and then I'm going to go to your website to check

if it's the same thing. And then I'm going to go back on YouTube to make sure that that's so there's so many touch points that happen right now that you cannot really understand if you just, you know, run a funnel analysis. So this is a great opportunity for people in CRO to use this model to synthesize textual data, reviews, search data. It's so much that you can do right now. Can you use it for numbers? No, you shouldn't. That's why you have math. That's why you have machine learning and

for sure LLMs are better in general.

then they used to be two years ago when I was working on the Starbucks project. Like I would not do the same thing, the same if it was for it to work. But it's just we, because we get a result in a, in a, in a chat interface, we think everything is so simple. So we start thinking like, Hey, well, I don't need to run a linear regression or this or run a confusion metrics. Who cares? I'm just going to ask JetGPT or I'm just going to ask Claude and it seems, it sounds good. So might as well do it.

Guido (12:15) Hmm?

Juliana (12:29) it. Yeah.

Guido (12:31) So what are the biggest things that you would do different for that project, for example?

Juliana (12:35) For what?

Guido (12:37) for the Starbucks project.

Juliana (12:38) I think maybe I would use a LLM for the synthetist because at that time Gemini was not very good to synthesize the review so we were using zero-shot classification so it's something that you hold inside the model so you give it one label, a specific label and then it looks in the reviews and tells you how many times that label was found in the review.

If I would do it now, I would add the nice reasoning of the review, what was done once is I would cluster some reviews like that and kind of tell the story, generate the hypothesis based on all the reviews. Like I would do more cool stuff. But at that time when me and Krasimir build this, you couldn't have used Gemini. Gemini is much better these days.

I don't use, and I had this conversation this morning with Simo and Krasimir and other people and I was saying, I don't use cloud or Gemini or anything from the chat interface unless I need to rewrite something because English is not my native language. So maybe I need to rewrite something or change the title, make it more SEO friendly. I use it as an API inside of a cloud infrastructure. So I don't really see these models. I just see the results.

Guido (13:45) Hmm.

Juliana (13:59) So I wouldn't do this, but now it's so much easier. You can deploy a pipeline in an hour where you ingest the reviews from a platform. You use ML generate text and Gemini to analyze and synthesize this boom, boom, boom, and you get the results.

It doesn't require that much manpower that it used to require back then. But for those that are interested, that pipeline for Starbucks is still alive as well. And well, it didn't die for three years. I think if it was a Gemini, if we would have done it with Gemini back then, it wouldn't have lasted this long. Yeah.

Guido (14:40) Yeah. Okay. So

we shouldn't be using LLM for data analysis, right? But you'll, please don't. But in a recent blog, you also, I think you presented about it at Superweek and Perform conferences, the haystack problem, right? I mean, you can just generate different variations endlessly, basically.

Juliana (14:48) Please, please don't.

Guido (15:08) So for those people who haven't seen that talk, what's the one minute version of the Haystack paradox? And why do you think that's also relevant for, also not a trap for experimentation?

Juliana (15:23) So what I'm trying to say with that talk is that it has never been cheaper and easier and faster to create an asset, a creative asset. And most people, when they think about the creative asset, they think about an ad, but they think image, carousel, video, that's it. But one of those assets is a very high dimensional point in a very big space that has context, that has content, that has a concept.

image will have a title, CTA, a person, an actor, a gender, location, a specific photography. So there's so many parameters to just one singular file that if you think when you test two images against each other, you will never know why an asset won. You will just know that it won, but was it the color? Was it the background? Was it the CTA? So I'm showing a way where you can basically decompose a creative asset like a video or an image and all its individual pieces.

and then test it and understand what is the performance of that across different platforms. So for YouTube, for example, this is something I'm working on right now. On YouTube, you have in-stream standard ads, which is the classic one that you see on a, you know, when before the video, but then you have shorts. So a short is very different than an in-stream standard ad. In YouTube and Google has this ABCD framework for attention, branding, connection, and direction. So each asset will have very specific

components that must rank very high for attention, for branding, for direction and connection so that they appear to you. So now in YouTube, you have YouTube AI, which basically means the Manjian and VRC. So both of them are connected. It's not the same thing. So this means that the only way to target people is kind of to create an asset that is relevant. So now because of the customer journey being so decentralized, it's so fragmented.

and multimodal and people are just everywhere at once. The image or the video, the creative asset is the only thing that follows people across where they are. So it's your storefront, it's your closer, it's your whole brand experience. So once you decompose these assets, for YouTube for example, as I was talking,

You know exactly where are the spots in the video or in the creative asset where you can make changes and amend them to fit with the platform best practices to fit with the algorithm. So you will have not spent 14 days in the learning phase when you launch a YouTube campaign and be day one ready from the start because you test your creative before lunch. But you can also test your creative in flight and then you can create this that post flight. So this way you kind of like build this organization. like to call it organizational memory.

of your assets. So every time you start a new campaign, you don't start from zero, you actually look back and say, well, this actually worked. This combination of creative elements worked. So I think this shouldn't be a strange thing or a new thing for CRO teams because this is kind of like what we do. The culture of experimentation is center of excellence and whatever we're calling it these days.

This is completely missing in media. So I found this spot in media where I really like to play Right now and YouTube is huge for me If you want to see that talk, it's on the super weak YouTube channel, but that talk changed so much in the last month So next time I will do it is probably gonna Yeah, it's it's so many things that I've discovered in the process, but YouTube is very important for me

Guido (18:53) Yeah.

Juliana (19:02) This is probably going to be most of my content about in the next period. That's why I'm quiet now on LinkedIn, because I'm, I'm cooking.

Guido (19:11) Well, we'll make sure

to link to your YouTube channel in the show notes for sure. But yes, I think also for the the Haystack problem, I mean, if you all have if you have unlimited traffic and you can afford to not know why, then sure, create 40,000 variants and go wild. But most of us don't have the luxury of unlimited traffic. So we need to be more.

Juliana (19:17) Yeah, I'm cooking.

Guido (19:38) We need to be smarter about the variants we pick and learn from them to be able to iterate, right? Otherwise, it's just a dead end after that. Nice. So my last question for you, Juliana. So you're running the whole day as mission control. What's the conversation that you hope that happens in the room that isn't happening at most AI and marketing events right now? What's the...

Juliana (19:43) exactly.

So I'm very excited

because it's the first time I'm the emcee. I'm always like the speaker. So I'm low key terrified. I think we've this before.

Guido (20:09) You'll do great.

You'll do great.

Juliana (20:12) I hope I have to wear this Darth Vader costume the whole day. So that was going to be interesting because this has this... That's the thing. So was thinking of genuinely not taking off my mask at all.

Guido (20:17) Oh, yep. So people won't even know it's you.

That's a secret.

Juliana (20:26) But then I have acne a problem, I'm probably going to be exposed on the face. But I'm excited. There's great speakers like Jim, like Sunny, Lina and Sarah. But I'm excited mostly about the on-conference part where you get other people to showcase what they're doing. What I'm hoping to get out of it is kind of like use cases, very practitioner led stories. I hope I'm not going to get slapped or

stuff about you know in the competitive landscape of the ever ever changing digital competitive landscape or something like that. Yeah I'm excited about use cases, excited about different case studies and things people are doing to get inspired. It's gonna be hard since I'm not speaking because I don't know what to do with myself so I hope I hope it will go well.

Guido (21:18) But as

the MC, you have the power to arrange everything in the room, right? So you can guide the questions and the conversation.

Juliana (21:25) I might even

just talk and not let nobody come on stage and just keep people there for seven hours and just talk about life.

Guido (21:31) Exactly.

If you want, could do an whole presentation. You're on stage anyway. No one's going to kick you off. Just force everyone to listen to your rant about anything. Yeah, you can do that.

Juliana (21:35) for everybody.

to listen to me. Otherwise I'm going to

choke them like this, know, and God is... Exactly, exactly. And it's going to be fun. It's going to be fun. For sure.

Guido (21:46) Exactly, from a distance. can choke everyone at the same time. Mind powers.

Good.

Thank you so much, Juliana. So yeah, Juliana is hosting the entire day at Space Academy on May 7th in London. Tickets are still available, link down below in the show notes. You can follow Juliana on LinkedIn. Her newsletter, Beyond the Mean, is on Substack and the Standard Deviation Podcast with Simon Ahava wherever you listen to your podcast, probably on the same platform you're listening to this podcast right now. And of course, like words, go subscribe to Juliana on

YouTube for all the future content and I guess the future version of the Haystack presentation as well. To be continued. I'm Guido Jansen, see you next time. Bye bye.

Juliana (22:28) to be continued.

Thank you, bye.