AI Visibility Explained: Why ChatGPT and Googlebot Can't See Your Website ft. Kyle Duck of Alli AI

speaker-0: All right folks, welcome back to

speaker-1: I said that.

speaker-2: Cut, cut, cut.

speaker-0: Do a nice little blooper at the beginning. That'll be nice.

speaker-2: Welcome to Control All PR, a podcast hosted by Eric Grau and Amber Krasinski, where PR, marketing, cybersecurity, and tech finally sit at the same table because most problems happen somewhere in the gaps between them. If you're looking for expert advice on how to handle AI and real conversations about its potential, consider this your hard reset.

speaker-0: So welcome back to Control Alt PR. This is where the world of marketing and public relations and crisis reputation management collides with cybersecurity and tech and IT and all that jazz. So today we have a really interesting episode. Eric, what do we got going on today?

speaker-1: ⁓ today's gonna be amazing. We're gonna break down something that most people have no idea is even happening under the hood in the internet. The interwebs.

speaker-0: So we're gonna be talking about the gap between what a website shows a real person and what AI actually sees and why the impacts of all of that AI agents, bots, crawlers, scrapers is much bigger than you actually think it is.

speaker-2: AI, AI, AI. Our guest is AI.

speaker-1: Our guest is Kyle Doc. He is founder and CEO of Alley AI, an expert on all things SCO, AE, AEO, GEO, AIEO, everything else, every acronym you can think of that.

speaker-2: Think there's five or ten more. Yeah. Yeah. Yeah.

speaker-1: You're a total techie from what I gather, right? I've read up. You're you're definitely in there. You've worked with agencies, you know, across the gamut, and you're gonna walk us through some of the visibility on all this stuff that we should all be paying a little bit more attention to. And it's not the same old, same old anymore, right?

speaker-2: ⁓ yeah, absolutely not. It's a totally different world now these last couple of years and it's ⁓ a different world that I guess most of us don't even realize we're living in and and maybe could get worse.

speaker-1: Yeah, or better or better. Let's be hopeful. Let's be hopeful, you're

speaker-2: Yeah, it'll we hopeful. Sorry, sorry. We're entrepreneurs, we have to be optimistic. My my apologies. We'll figure we'll sort it out. We'll sort it out.

speaker-1: So with that, in the beginning, we want to know a little bit like what has your professional journey been like?

speaker-2: Yeah, sure, sure. So I got into ⁓ what was in search engine optimization. I I guess going back a good 10 years ago. Back in the day, kind of making my own websites and trying to be kind of a solo entrepreneur and then figuring out how to get Google traffic. ⁓ eventually started consulting, eventually joined an agency, started my own kind of agency consulting brand, ⁓ did in-house work for ⁓ for Staples Europe here in Amsterdam, managing I think 20 something, 25, 27 sites, every language, every country throughout Europe. Over time I was ⁓ learned how to program and kind of built my own automation tools to, you know, kinda automate my job, just trying to scale my own consulting work. That turned into Ali AI, which I launched eight years ago. And ⁓ three kids since then and ⁓ maybe three different companies we might say as well. ⁓ but definitely have been a lot, you know, evolving with the landscape and and trying to innovate ⁓ as well within our industry whatever. Yeah.

speaker-1: Three kids. Yeah.

speaker-2: I thought to make it used to They do keep you busy. Yeah, yeah, they do keep you busy, but hey.

speaker-1: As I have my students conference in the store, yeah.

speaker-0: Yeah. ⁓ they're brilliant. Kids are fun. All right. So Kyle, give our give our listeners like the short version. What is Alley AI? What's your mission? What's your, you know, vision for all this?

speaker-2: ⁓ our mission originally, we were originally kind of the first ⁓ the people bringing AI into the industry or one of the first, ⁓ let's say. And that's evolved as the industry has evolved. Kind of the biggest problem that the the thing that's kind of keeping me up tonight and getting me up early in the morning right now is ⁓ how much everything has changed with AI visibility, specifically what we're gonna get into, which is ⁓ that AI crawlers, agents, bots aren't reading the full text of a website. Study or a a study from a company called Cited that ⁓ sites that are where their content is fully available are seven point eight times percent more likely to be cited by Chat GPT. So this is a number one factor in getting what's, you know, now the the the most growing search engine or information or treval you know engine in the world. So that's kind of our our big thing, our big focus now.

speaker-1: Yeah, that's that's ⁓ that's incredibly important on that. I also noticed did you do you do you have a percentage of like what sites AI doesn't right now on the internet s ⁓ you know, ⁓ I guess ⁓ scrape or or even pay attention to?

speaker-2: Well, in terms of not paying attention to, I mean, a lot of sites are unwittingly blocking Chat GPT crawlers, ⁓ because of Cloudflare. Or even like so I had like a kind of a hard number. Open AI uses ⁓ for a lot of what they're they're training crawls when they're training new bottles, they use the common crawl, which is going out and just crawling the internet and they kind of buy that data. And so there's ⁓ let me write his name that then we have a read his name to get his name exactly. Saganton ⁓ Mohanadsen, he's a ⁓ a kind kind of influencer in the industry. And he posted on LinkedIn that half a million sites, what he could tell, half a million sites from Crum and Crawl. And this was from last year before Cloudflare ⁓ changed the rules, were blocking ⁓ OpenAI from even seeing their site. That that's the entire website is missing. ⁓ we just finished up a study from we just did the Fortune 500, and we found that ⁓ of the Fortune 500, we did 20, 20,000 pages. ⁓ we were able to crawl 250 sites. That of those sites, 27% of the content totally invisible. That's of the biggest companies in the world, right? You can imagine for, you know, small businesses and and throughout the economy what that can do.

speaker-0: Absolutely.

speaker-1: Do you think the AI crawlers like ChatGPT open AI, do you think they crawl more than the Google bots?

speaker-2: Yeah, they definitely do. They definitely do. They're way more aggressive. We have pretty good data on this. So we're able it it goes and fits and spurts depending on like when a new model is being trained. But we're, you know, we're monitoring a whole network. We like have like forty million ⁓ hits from crawlers, from like bot crawlers and training agents over the last few months. And it's like becoming almost fifty fifty human and ⁓ i you know, human in versus bot traffic on the web. But We also see like we had ⁓ when when Facebook, I think they're training a new model. So we'll see a big s you know, a big spike in Facebook traffic. ⁓ what's the company's name about China? Huh Howei? I can't I know if I'm pronouncing that correctly. They were yeah. However, they were training a new model, I think, last month. Yeah, yeah, yeah. And they were trading a new model last month. So we saw a big spike from them. So ⁓ it depends. Now i in our case, there's probably like some bias there because for our network we are optimizing their sites and we are rendering their sites for AI visibility. So it's higher, like you know, the the round of chat GPT traffic will be higher than the non ⁓ you know, visible site or the average site. So there is like some selection bias there. But it it's it's only growing.

speaker-0: So let's dial it back, you know, just to kind of make sure we're all like we're talking Cloudflare, we're talking all this stuff. Like Okay. So if I'm someone who I know what SEO is, that's like Google stuff, right? What is the difference between AEO and GEO? Like there's all these terms that are floating around. People see them on LinkedIn. I mean, I know lots of colleagues that are trying to figure out like

speaker-2: Slide a number, slide a number. Yeah yeah.

speaker-0: What the hell is this?

speaker-2: I know it. Yeah, I mean sometimes I looked at LinkedIn and I'm wondering the same, you know, it's all and and they're they're all converging as well. ⁓ you know, SCO traditionally was just optimizing for Google, which was a bit different because with traditional Google search, they were rendering JavaScript past let's say 2015, 2016, something like that. This wasn't as much of a problem. It was still a best practice to to make sure that your content was being fully loaded and all that, but it wasn't kind of a mission critical thing. So now we have ChatGPT, we have Clyde. Most of the traffic is Chat GPT. I think seventy-eight percent of like AI search referrals ⁓ are gonna come from Chat GPT. ⁓ you know, Claude and Perplexity make up another percent of that. But so now we we consider AEO, GEO, pick your acronym, are optimizing specifically for those AI answer engines. And of course there's a lot of like overplay with traditional SEO, but not exclusive.

speaker-0: Is there difference between AEO and GEO? 'Cause I've seen fights in comment sections about that one little letter.

speaker-2: Yeah, you know, to me, no, I don't know. I don't care about fighting in the comments section. So to to me I'm like, Hey, let's just make sure everybody sees us, no matter what you call it. I I know I wanna be seen, I wanna be sighted. So ⁓ I I don't I try not to get it.

speaker-1: Yeah. Yeah. You you don't want to get into turf words. To be honest, I'm people still get a little confused and and and sometimes they don't know the difference. Like like you know, Annberg was saying it was a web crawl and a scraper, it just gets very confusing after a while and what you do, paying even attention to, right? If you're writing towards the AI, but then you also have to write towards people that are reading it, like how do you how do you draw that line? Like, you know, how much because I draw I do it myself, right? I go on my website, I try to do my things. And sometimes what what what comes up better for AI probably as an uptake might not make so much sense for a person reading it. And we could also go into how many people are even reading the websites in the next two years and all the AI. But I all I know is that when I see my my metrics, you know, I see things coming up as being cited by AI. So I'm doing something right. We're reading it. So like what what where is that line somewhere?

speaker-2: So I actually just this is just my opinion, but I actually think that, you know, with ChatGPT and Claude, it's this is a way better development for the industry because of that, because you know, people talk in natural language more to ChatGPT and and Claude or whatever else they're using. Then, you know, a Google search might have been, hey, what are the you know, best shoes, right? Whereas like in Chat GPT and Claude, you're having a discussion. And so when you know, when ChatGPT is going and looking for the content. They're trying to match what like a real person was using in real language. So now, you know, in terms of like marketers and content creators, we can use like actual normal human language in our content. And it is more likely to, you know, to match ⁓ what that kind of user query is what the prompt was. There are some tricks, like I think it's generally considered to try to, you know, if you have an idea of what that natural language like what somebody's actually saying, you put it higher in the page content and ⁓ Maybe in the header of the first paragraph, ⁓ things like this. But I to me I think it's all in all a good development for the internet.

speaker-0: Yeah. No, that makes sense. And I've heard that using FAQs can actually be a a good tactic as well because people are searching for questions or, you know, some specific surrounds

speaker-2: Yeah, that's definitely a a a best practice. And we ⁓ sometimes, you know, you can embed those. ⁓ some people embed ⁓ in Markdown, some people, you know, have ⁓ on every page. But I also think that's good for users anyway. You know, you have FAQs for every product you sell or every service you provide or general information. It's good just to have those. So I think we're getting a lot closer to overlap between humans and and machines here in a way. As long as the machine can read it, yeah, right. But

speaker-0: Yeah. So, okay. I wanna make sure everybody's clear because like there's all these terms agent, crawler, scraper. An agent, I could be totally wrong about this, you can correct me. An agent is something that you kind of program through something like Claude and it kind of does some work for you. It's not the it's not the same as an automation. Right? Yeah.

speaker-2: You know the joke that the the best like the you know the best marketing thing ever was calling ⁓ you know just calling an automation or calling a bot an agent, right? You just you had a bot so now it's an agent. So ⁓ well I'd say that's generally it's like an agent's taking an act.

speaker-0: Okay. And then within that there's crawlers and scrapers.

speaker-2: Yeah, yeah. I'll explain it with an example. When OpenAI is training a model, they're crawling, they're visiting every page or hopefully every page on your site, as many as possible. They're scraping, downloading the content to go into their training. ⁓ yeah, those are kind of like the two verbs and that's ⁓ that's what becomes a noun.

speaker-0: ⁓ so it's like they're just checking out the page versus actually like, Zoop, I'm pulling this directly.

speaker-2: Exactly, exactly. And then an agent might, you know, somebody has an agent that they want to check a website for price changes or for example on products. That's an agent that's going now, an agent's crawling the site, ⁓ scraping the price and then maybe taking someone taking that that price and giving an alert about it or buying it or or whatever else ⁓ kind of action you might want. That would be an agent.

speaker-0: So they can all be the same thing, technically kind of. Yeah.

speaker-2: Yeah yeah, pretty much. It gets you know, it just all blends together. Yeah, yeah.

speaker-1: Now do you now do you find that that the the number or the percentages of where AI is actually paying attention to your website or not is it is it the same? Does it hold steady across all industries or do you find that certain sectors are are worse than others or things like that?

speaker-2: Well, I think it has to do more with like page content type, right? Like informational content has definitely been been heard because now, you know, before if you're researching a topic, you would go to Google and you're you're trying to learn about something and you open ten pages and you maybe scan five of those and then you read one or two closely. Whereas now that entire conversation or the entire kind of research, ninety something percent of it is happening ⁓ within Chat GPT itself. So I don't see it as much by industry, at least just my own experience and my own viewpoint. I I think it's happening more depending on what the content of the page is like.

speaker-0: That makes sense. All right. Let's go back to this like study report thing that you talked about that crawled all of these Fortune 500 companies. So walk us through what made you want to go check out Fortune 500 and how this study actually worked.

speaker-2: worked. Part of it was just like, hey, we need to pick a group that's ⁓ and everybody knows what the Fortune five hundred is. It's ⁓ you know, almost arbitrary, but it's also kind of something that society has agreed upon is is a somewhat of a metric for business activity, right? Or like a fixed group. So that that was why Fortune five hundred it was kind of the easiest places to start. And we also knew like, hey, these are big companies. If they have a problem, then we know it's a problem. Right. Versus more likely that small business ⁓ you know, maybe less indicative of the overall economy. What we did for this is we scraped our crawler scrapes, ⁓ scraped all of the, you know, every sites or the ones that allow some, you know, ⁓ we don't break the rules of crawling. ⁓ so if they didn't allow us, we didn't do it. So that was 250 sites, crawled twenty thousand pages from those. And what we did is we looked at the raw HTML version of the page, which that's what Google sees on first pass. That's all that Chat GPT sees, that's all that. Cloud C's. That's just like look in the page, view page source, raw HTML. And then we compared that to the human ⁓ readable version, what you see in your browser, on your phone, on your laptop, whatever, on those pages, on about every site on the internet, except for Berkshire Hathaway, 'cause they have a site from the nineties. They're Yeah, yeah, yeah. And hey, they're perfectly visible. So the older you are, the better. You know, there's something to be said for that. Yeah, yeah.

speaker-0: I know it's awesome.

speaker-1: With giant plants.

speaker-2: So for just about all of them, they're loading ⁓ some version of the content or some amount of the content on the page is being loaded dynamically via JavaScript. So we kind of went through all the, you know, all the as many we could from five hundred and compared, hey, here's a human readable version, here's the machine readable version, how much is missing? Just compare raw text on the page, how much is missing?

speaker-0: Eric, this thing is gonna blow your mind. Cause like when I saw it, I'm like, this looks like a redacted government document when you look at a website.

speaker-2: Okay. Yeah. ⁓ BostonGlobe.com is a good ⁓ is a good indicator of this because they're a media site and I would imagine purposely blocking, you know, ChatGPT and others just because ⁓ they don't want their content indexed ⁓ or they don't want their content going into the the training without you know being compensated for it. So if you look right here on the left side, this is the human version of the page where you see everything. ⁓ and if you look here on the right side. You see these big squares. And these aren't like mapping one to one, but this is basically showing like, you know, more or less every section of the page. So this section matches to this headline. Basically every headline is is being blocked. If you scroll down, content block, content blocked, content blocked. So j just more or less I mean a few sections probably sneak in and maybe the images are are loaded or or hard loaded. ⁓ but certainly like all the headlines. all of the kind of sub-paragraph descriptions to get you to click on the they want you to click on there here in the metro section. So this whole, you know, we're not like totally covering it, but this section blocked, this section blocks. So you see that, you know, only nine per only nine percent of the text on the page is ⁓ is actually visible. So it's just almost it's just almost every article and every section and everything ⁓ that they will want you to see on the home page is is blocked.

speaker-0: This is something that is so critical is being able to be cited accurately, relevantly in something like chat. Being able to only see nine percent of the Boston Globe's website. Like if you're trying to find out, even if you're an everyday person trying to find out what's the latest on the Lindsay Clancy trial and it's pulling random other sites.

speaker-2: Yeah, it can be totally random. You know, it's just gonna be like the national enquire ⁓ equivalent of a of a blog nowadays and for who who for you know hasn't bothered to block their content. But also I think it hurts like newer smaller publishers because they they might be blocking these things by default and not realizing it. So they're you know, they're hurting themselves from getting traction in the first place.

speaker-1: Crowdfillar alone, that's one thing I'm gonna start talking to my clients about. It's like, hey, this is pay attention to this. This is serious stuff. And regardless of what you think about AI, you know, it's gonna affect your business.

speaker-2: We have this big picture stuff and we have the ethical stuff. I'm like, but in the meantime we all have to make a living. And i in the meantime, you know, anybody in business we want to be on there. And, you know, the Boston Globe has totally different incentives than they do. They're big enough and and all the huge media companies are big enough for they can try to leverage against helping AI. ⁓ anybody else below that tier, i you know, i isn't, right? You don't have leverage. So the the leverage goes the other way. Yeah.

speaker-0: Yeah. And it's also interesting because you think about a lot of, you know, going back to your study in the white paper and everything, you have these very large enterprises that are like, Yeah, we're AI first, we're AI driven. ⁓ by the way, you can't see a lot of our site or we have to do all these like duct tape band-aid fixes to the source code to actually show up properly or leverage these massive deals with open AI.

speaker-2: Yeah. And you know, there's like a good example and then ⁓ I think I did a I did a LinkedIn post about Walmart and maybe I haven't pulled up here somewhere. ⁓ well I have to reload the extension. ⁓ but Walmart, Walmart being on their e com pages, they have huge issues with ⁓ with visibility because it's you know, they're l it's an e com platform, so they're just, you know, they're pulling via JavaScript from ⁓ you know, a separate database and all that. And so what they've done is ⁓ It's kind of a little band-aid like Amber said, but they went in and put in like JSON code if you're ⁓ familiar with that kind of raw code, embedded in the page source so that that information, the price is there, the product price is there, the description's there, and then all the different like recommendation panels are all there. So they've ⁓ they've embedded that that in their site because I I don't know why they didn't go for the server side rendering solution. Yeah. Who knows? But You know, they've definitely taken action. I mean, Walmart's a great company, so but they've definitely taken action to reduce that. They realized there was a problem early on and have ⁓ have tried to solve it.

speaker-0: One of the things that I've seen a big uptick in is lovable sites and people coding something on a lovable site. It looks gorgeous. It's really, you know, cool interface. Are they actually showing up in AI search or not?

speaker-2: Kind of like the default out of this box ⁓ lovable site is zero percent visible to ⁓ Chat GPT. Lovable sites are essentially single page web apps, JavaScript web apps, where basically everything except for the title and a meta description ⁓ are loaded from JavaScript. So just out of the box, unless you make other changes, they are totally invisible. This is the case. Out of the box for every single site that comes from level at this moment. Maybe they'll figure out a w a fix, but that's just kind of the the the way a JavaScript web app works is it's all loaded dynamic.

speaker-0: Interesting.

speaker-2: And then that's just not just lovable, that would be just about any vibe coded tool you could you can think of, ⁓ for for just a website. So it's almost like your Facebook feed or your LinkedIn feed, except it's like totally unnecessary loading static content dynamically.

speaker-0: Cause even looking at tools like like Lighthouse, I use that on so one of my business friends, he posted all about, you know, building in public and here's my new website. And I mean, even just using Lighthouse on it, I was like, ⁓ this is this is rough. You didn't know what you were doing. So I mean, does there need to be like a warning label on some of those things? Or, you know, like I feel like people don't realize this a lot.

speaker-2: Wow. Yeah, people don't realize it and probably there might be kind of like an incentive ⁓ misalignment here where not many people involved want to necessarily advertise it because it's it's hard to fix and and not a simple fix. And ⁓ you know, I I don't want to say anything about lovable in particular because their own website is fully vis visible, but ⁓ th you know, th hopefully they'll get around to fixing it for all their their customers. But they're not the only ones by by any means. We were ⁓

speaker-1: But are they even paying attention to it? You think that's something they wanna do?

speaker-2: You know, that's kinda like that's kind of like why I'm coming on podcasts like this and trying to get the word out because I think people should know it's a huge factor of the future of the internet. And it's actually getting worse, ⁓ because of the you know, the changes that Cloudflare is making. We're talking right now, we're working with a ⁓ a CMS, a content management system, who had inadvertently blocked the all of their customers' sites from Chat GPT without realizing it just because of some of these default settings, right? And we're talking about thousands of businesses out there. ⁓ and so now they're you know hope they're gonna get fixed and all this. But ⁓ yeah, it's kind of a big problem getting bigger. And then, you know, if you're blocked from Chat GPT, one, you're not getting referenced. And two, if you do get referenced, you're just gonna be hallucinated because it can't see your price or you can't see your product descriptions or your service. So you know the CEO's name gets hallucinated and just all sorts of ⁓ bad stuff happens. So there's nothing good that's coming of it. Coming forward for businesses, that's for sure.

speaker-1: Let's take this out a little bit further. And what happens like if the gap never really closes? What happens in a in a year or two from now, like if it's still bad where people aren't catching up? How do you see this playing out?

speaker-2: I mean, the internet gets worse. AI gets worse. Like AI search for all growth went up five hundred and twenty seven percent in twenty five. And it like say like let's say the the rate continues. We go to AI more and more for answers. And if the source materials are blocked or hallucinated or outdated, then we're getting old information. We're getting bad information. It's just like a pollution of the public comments.

speaker-1: Perfect segue into just what's going on in Europe, like the US versus you know, versus Europe. ⁓ German court recently decided that Google Gemini was liable for incorrect information, which that's kind of how it plays in that scenario that you just described now. Like how differently is is Europe and the US treating AI accountability?

speaker-2: Obviously that court ruling is a big thing. And, you know, in Europe, misinformation, like and kind of disinformation has let's say a lot more like legal consequences, generally speaking, the US and every country is different, of course. And there's ⁓ more consumer protections and and this sort of thing. You know, so if you have an ⁓ a customer service AI agent and it can't read your site and it hallucinates a discount that you don't actually offer, or hallucinates a price that don't actually offer, your business could be ⁓ you know, ⁓ liable for that. It's definitely not a good development. You know, definitely from you definitely definitely in Europe. I think the ⁓ the rules are generally stricter and the business controls are stricter. And I you know, I would expect just with the misinformation and disinformation push and ⁓ in Europe that we, you know, could definitely on the business side be looking at real penalties for ⁓ for companies that don't solve this. The analogy I use is like the Americans for Disability Act, how you know. All of us have had to make our websites accessible for people with disabilities and rightfully so. We have an obligation to make our websites accessible for AI agents because that's the way we're now interacting with information in the world.

speaker-0: you see the US kind of following suit with the EU? Or do you think it's gonna be, you know, it's gonna take a minute and there's gonna be discrepancies between different sites?

speaker-2: Well, I think we probably probably look at and this is just me speaking off the cuff, you know, with GDPR for instance and the cookie banner, which I live in Europe and it's actually quite annoying, but nonetheless, it's a big enough economy that if Europe puts in regulatory requirements, American businesses will follow suit because you you have to just to be ⁓ compliant with everything. So it's a big enough market that if there's pan European legislation, you're gonna end up with American business compliance anyway. I won't get to I don't you know the politics and the regulation, all that. That's way above my pay grade. But ⁓ I I think just in practical terms, you know, you'll see businesses lead the lead the fold if that you know, if those kind of requirements come into place.

speaker-1: I personally hope it does a little bit. I think it needs a little bit of regulation.

speaker-2: Yeah. Yeah. Probably so. I mean, you know, this is how we learn about the world now. This is how we get our information. This is how, you know, people are using so much for our daily lives and not just like, you know, I am a businessman, I'm dealing with business, but just anything you can imagine. I I I want as a user, I wanna know that I'm getting the latest and new and newest you know, newest and most accurate information out of Yeah. Especially

speaker-1: Very important. All around. No matter what the vertical you're in, I I I think you know you're absolutely correct. It needs to be as accurate and as turned as possible. So Cloudflare, they they started blocking all the AI crawlers, right? By default. Have they lit have they leaned up a little bit or or are they still blocking

speaker-2: ⁓ they're it's getting more stringent. Now they're not going back and changing. Like if you have a site already on Cloudflare, your existing settings are kind of grandfathered in's. But starting from September fifteenth, every new site is gonna be blocking ⁓ just about everything by default. ⁓ Google bot chat GPT for like training information retrieve. Well they have they've kind of kind of different categories where it gets a little confusing, but ⁓ I send out a red alert to like all of our customer base and every even the people that aren't our customers we talked with before, like, hey, go check this because if you're putting anything new on Cloudflare from September 15th, you are pretty much just ⁓ you know, blocking everybody that you want to see your website, every, every agent or or bot that you want to see your website by default.

speaker-1: Making it an opt opt out or

speaker-2: You have to opt out from that. Yeah. It's by default. So by default you're blocking ⁓ everything. They have a business model particular publishers where they want publishers to be compensated by the AI bots for accessing that information. You know, I I know you two like work in media, so there's like different incentives there, entirely. So they're trying to kind of like ⁓ you know, spearhead this and and be have like a nice little toe gate in front of the internet. You know. From the business and I deal with businesses all day, we want everything to be read and and indexed and trained and ⁓ cited and agent available as much as possible. So

speaker-0: Makes a lot of sense. So I mean, I I'm really worried about agencies with this because I mean, if you're managing dozens or hundreds or even thousands of client sites and a lot of these are using Cloudflare or, you know, like not clear about these crawler defaults or things like that, their AI visibility disappears and they don't catch it. Like that's that's a huge liability. So I mean, I think like ethically. Whose job is it to really catch that? And like, is the average agency able to answer that question? Is there like winners and losers in all of this? Like what's the what's the responsibility ethically? I know we've kinda chatted about it. I

speaker-2: Yeah, sh sure, yeah, sure, sure. I mean I obviously if you're an agency, you know, that is your job. Like if somebody's ⁓ hiring you to help with their AI visibility or search visibility, I mean any kind of agency worth their salt, that's the first thing they're ⁓ they're looking for anyway. And if you kinda get into the weeds on this and search around like, hey, best practices and what's my to do list, what's my checklist? most of the good ones will have that as number one or number two on ⁓ on what to check for because if you're a business, you know, it it just is the most important. 7.8% more likely or or whatever the the number might need might be. You know, ethnically, I I honestly I think like to me, I call this a black hole at the heart of the internet. I think this is a big enough issue where we need elected officials involved. You know, I think this is just a major issue in in how humans are interacting with the world and by each other and by the total synthesis of human knowledge from the history of civilization. And if If only what if only seventy three percent of it is visible, we got problems here, right? So Yeah. Yeah. Yeah. So I think this is a kind of a huge issue in society for all of us.

speaker-0: Interesting that we're moving to this phase of the world where there's gonna be entire generations where you they don't trust we're already starting to see this, they don't trust what they see, what they hear, what somebody like says, you know. I mean it's it's wild. We've never really I mean, you go way, way back and you you look at like journalism and them making up like, you know, photoshopping things, but We've been doing this, but it's like a whole different level now. I mean, you can't even I can't even trust sometimes like what my dad shares with me from time to time. He's like, buddy, I saw this online and I'm like, Where'd you find it? And I look at the source, I'm like, ⁓

speaker-2: ⁓ yeah.

speaker-1: This has always been around. Growing up in the 70s, we had the National Inquirer, and there are people that really thought it was real. And that's just not even Photoshop, just picture cropping it out and pasting it really well. And that was it. Right. So there's always going to be people. And just recently, you know, Claude put in all the fingerprints down on the text and things like that. And one of our guests also has ⁓ you know, they they had a company that they text all the fingerprints on the images and videos and things like that. That's gonna become probably more prevalent though. So with that being said, like how does L AI solve these problems.

speaker-2: Yeah, we do a couple of things. Like the f first of all is ⁓ you know, when somebody signs up for us or somebody using our using our app, ⁓ we go through and we fetch the way the page looks to a human. We do the JavaScript rendering, we pay the money to do it, which is kind of a 10 to 20 percent ⁓ more of what the cost is than just doing the ⁓ kind of machine readable version, which is also ⁓ maybe another reason the the problem's not gonna get better because it costs a lot to do. We keep an index of that version, we keep it updated as you know our user's content changes. So then we kind of we plug in with the website itself. So then if ⁓ Claude or ChatGPT or Google or whoever then wants to access the page for a a user prompt or an agent action or training model, whatever, we serve that fully indexed or in fully readable human version of the page to the AI agent, to the bot, so that one hundred percent of the content is there.

speaker-1: Very valuable.

speaker-0: Yeah.

speaker-2: Yeah, for the people who know it's a problem. I mean it's ⁓ yeah, it's it's it's kind of a ⁓ an easy sales pitch if somebody comes in and like, hey, ⁓ you know, forty percent of our content is invisible. Okay, we'll press this button and now it's fully visible and then they're, you know, seven point eight times more likely to be seen by Chat GPT. Also, like as a business owner, it feels good to like, you know, be doing a good service for the world at large as well to to be to be honest with you guys.

speaker-0: That's really and that's so important. And speaking of the worlds, so we have a segment called ⁓ Eric, I'm gonna say it. I'm gonna say it the proper way again. I'm gonna do it. Used to call it the uh-oh scenario, and now I call it the ⁓ shit scenario. This this is a callback to my love of crisis communications, reputation management, all of that. So I'm gonna give you a total mess of a situation, just right here, no notes, you know, just

speaker-2: There we go, there we go.

speaker-0: Top of the dome, you know, how how you're like really gonna navigate a scenario. And yeah, then you just you just riff. So here's our scenario. So we have a US e commerce brand that's expanding into the EU and they launch, you know, some websites for European customers. It's the same platform, same content, they translate the pages, it's ADA compliant, they do localized pricing, all this stuff. But it turns out once they start getting some more customers from the EU that ChatGPT and Gemini are pulling product info and old quotes from an old pre-launch US version. It's got different pricing, different compliance language, even like a claim about a certificate that, you know, the EU version doesn't carry. A German tech journalist picks it up and starts writing about it and tries to go to the CEO for comet. What would you recommend they do? going forward and if you could time travel for this poor company, what would you tell them?

speaker-2: Yeah, yeah. First of all, obviously you're gonna have to, you know, ⁓ go by hand and look at the page and say, Hey, there's where the EU version ⁓ is and this is our kind of parameters that we operate within the EU and kind of show them the localized version. I guess that would be f you know, first of all, for handling that. And if you go back in time, you know, you make sure everything is ⁓ fully JavaScript earnable. We see this problem in e comm ⁓ all the time. Actually, it's a very common problem just because pricing and descriptions and so much and inventory is so dynamic. ⁓ So yeah, go back in time and make sure the site is ⁓ you know fully renderable by JavaScript. By you know, by your own means, whatever means you can find, or we're happy to help you as well. But I love that.

speaker-0: Very important.

speaker-1: I don't think we have a choice. I think we all have to do our part and and and try to either way it's gonna move in one direction, right? We I think we all try to help each other and coach each other so we do have the right things in place so we can expand humanity in some way or business.

speaker-2: Yeah, I know it. I mean it's just the Chat B GPT, it's just like, you know, the the library of Alexandria of Alexandria or whatever. But then you know it's like now there's just this big hole through it, and there's a hole through every book you look at, right? So ⁓ yeah, you know, my hope is is this is everybody gets their side fixed up. ⁓ you know, then that kind of library is made whole and these all these books are made whole. And it's tough to ask Chat GPT to do it themselves because you know, ⁓ you know, we look at all like there's so much consternation and so much kind of like up or over like these data center build outs. But for Chat GPT to do the human you know, human version of the page, you know, it's 10 to 20 times as expensive for their training models, for their agents, for their scrapers, everything. Like, and that's not just like expense in terms of money. That's also expense in terms of data usage and energy and everything else. So it's tough to ask them to do that as you know, as a company, right?

speaker-1: Do you think AI will help figure out those energy issues? And I know we already moved your shit scenario in the emotion, but I was just stepped into that. So now I'm like, I I wanna what do you think? 'Cause I I have my own opinion.

speaker-2: You know, I'm a big proponent of ⁓ of of nuclear energy. I have a a buddy down in Huntsville named Kirk Sorensen with ⁓ who who runs a flyb which ⁓ makes modular ⁓ thorium reactors. So I know a bit about this. So I'm hoping that hey, with AI one we can build better reactors and ⁓ you know, know about the safeguards and all that. So I'm kinda hoping that as the AI gets better and better that, you know, these advances we made in the digital world carry over into the physical world and we're able to have, you know, more clean and abundant energy ⁓ However that is. I think that's kinda the next steps with AI. ⁓ if we take like an AI maximalist point of view, is going from the digital to the physical world by just understanding everything better and ⁓ we don't have to worry as much about all the pollution and water usage and other problems.

speaker-1: I'm with you. I I personally think that this is a big serious problem, but I think it's a temporary problem. I think AI is gonna help us if it could help us solve mathematical problems that have been stumping mathematicians all over the world for fifty years, I really think it's gonna help us solve some of these.

speaker-2: Agree, yeah, I agree. I I sure hope so, yeah.

speaker-0: There's an AI doc that I was watching while I was on a flight recently, and the phrase apocalyptimist came up. So it's like you have the safeguards, and you're like, ⁓ snap, things could go really wrong, but I'm gonna hope for the best because that's the only space I can stay in. And I think I agree with you guys. I, Kyle, I'm curious what you think, especially given Zuckerberg's, you know, 6,500 big statement that he wrote out. Like, what could this future look like? like potentially and what's what's the role that some of these bigger players that do have the capacity to fix these things? Like what does that actually look like in terms of responsibility?

speaker-2: You know, I think right now kind of the model I evaluate l a lot right now, or I'm thinking about is kind of like a a nineteenth century, you know, late nineteenth century oil industry. We look at standard oil and some of these big players and certainly these are gonna be huge companies and I think over time the politics will get involved in in, you know, the U and the US and obviously in China as well, where these become kind of quasi state entities or become like heavily regulated commodities. ⁓ yeah, I think this is a really a problem for our elected officials to kinda sort out ⁓ you know, some of that ethics of this with the input of the people and and and kinda figure out how we want to organize our societies and and all this. Over time I think it's just it's gonna be such like a useful thing. It's just gonna revolutionize our lives in in so many ways that ⁓

speaker-1: It absolutely can and I'm with you. I think it's up to the politicians at all countries to really ⁓ try to think of what's best for humanity, for the country, for you know, for for everyone, which probably could lead me down a whole different path of discussions of like elected officials, which I am one of ⁓ for Board of Education town. We need to start we need to start electing people that know what they're doing.

speaker-2: ⁓ great, so Yeah, well.

speaker-0: To be honest about what the I mean, the White House put out all these new parameters for here's how we can vet if this is okay or not okay. And it's like okay, on one hand, we see like the, you know, the whole way that the railways, you know, grew in the eighteen hundreds and how that turned into monopolies. But then you also have here's these regulations that are going to be stifled by things like open weight models and you know what I mean? Like it's not you're not gonna be as competitive. So I think We're gonna need to figure out ways to to weigh those balances. And then we have everybody else in between that's like, I just want my business to show up properly or market.

speaker-2: It's a exactly, exactly.

speaker-1: By subscribing to Alley AI, listening to our podcast, they learn some

speaker-2: There we go. Yeah. So that's step one. That's step one. That's step one. And then you go vote and

speaker-1: So along with the OSHIT scenario, which, you know, we have another tradition here, ⁓ which is called the pass it forward. Each guest answers a question left by a previous guest or ⁓ audience. No context. Could be AI, non-AI, it could be, you know, what's your favorite recipe, doesn't matter. ⁓ but for you, here's yours. What's something you believed five years ago that you'd argue against today?

speaker-2: You know, five years ago, I was more pessimistic on agencies and consultants than I am today. And now I'm way more optimistic about their future because I think broadly speaking, with AI, a lot of not both knowledge work and maybe in the future, physical work is becoming commoditized where, you know, ⁓ the kind of the price of a new piece of code is, you know, at some point converged with the the token price. ⁓ so then I think you you know, when people will start making buying decisions, particularly businesses, I think they're, you know, gonna go relationship first because if anybody can deliver a product or deliver a service, you're gonna go with the person you like and the the person you know and the person who knows your business and your ⁓ your industry or your area the best. So yeah, that's my that I guess that's my point. Like, hey, tell me, tell me I'm wrong here, but I'm very ⁓ I I'm very bullish on the future of like, you know, agencies and consultancies of all all stripes.

speaker-1: agree more with you. I I think that human experience AI can bring a lot of things to the table, but ultimately I think we're gonna need some people to guide other people by

speaker-2: Absolutely. Absolutely.

speaker-0: So Kyle, thank you so much for breaking down, you know, this topic and you know, just something that folks aren't always aware is happening. It kind of feels like a if you know, you know situation. So where can people find out more about you?

speaker-2: Yeah, sure thing. ⁓ first of all, our website is ⁓ ollyai dot com. ⁓ we got everything everything that's ⁓ you know about our product and also a lot of information on there and everything else. That's the first point. You can also check me out on LinkedIn, just search ⁓ Kyle Duck, K Y L E D U C K. And ⁓ there's only a couple of us, but look for the one with Ali AI. And I just started ⁓ experimenting a little bit on ⁓ was it X, the Everything Out, no longer Twitter. ⁓ if you check me out like X dot com slash D Yeah, it's yeah, same, same. ⁓ I started posting a little bit there, kind of enjoying it. So you can look me up at ⁓ X dot com slash D T H E Kyle Duck and ⁓ you know, just ⁓ drop me a line there and say hello. So

speaker-1: Thank you so much again. This was really good. And thank you, audience, for listening to today's episode. And before we sign off, here's our inevitable call to action, which we just say, please remember to subscribe to Control PR on YouTube, Spotify, Apple for the latest episodes. ⁓ like them, subscribe, follow us on LinkedIn, Instagram. It's the only way that we can get this going, get more attention, and you can learn more really good advice from all these amazing people that we have. ⁓ on our show. ⁓ and if you're interested in becoming a guest or learning more about us, then just head over to control PR, fill out the form and we'd be happy to ⁓ to entertain that and hopefully have you on the show.

speaker-0: Yeah. So thank you everyone. We'll see you next time on control LPR. And remember, agents, scrapers, and crawlers are all kind of the same thing.

speaker-1: Thanks for listening to the Control AltPR podcast. Subscribe so you don't miss what's next. You can find us on Spotify, Apple Podcasts, YouTube, and controlAltPR.com. For more hot takes and resources, check out our LinkedIn and Instagram. See you next time.