SN 1093: Tokens in the Stream - Why LLMs are inherently insecure and prompt injection will persist

26 Aug 2026 · 2 hr 48 min

Play episode WhatsApp Download

Episode notes

Turns out, every chatbot conversation runs on a messy hack at the heart of language models, making prompt injection an unsolved—and possibly unsolvable—security threat. Steve and Leo unravel the research that explains why "roles" in AI aren't what you think they are. Understanding the controversy surrounding "AI Model Distillation" Anthropic moves to make their most powerful Mythos 5 model more widely available. Bitwarden's "Secrets Manager" offering prevents agentic and prompt injection abuse. The astounding and disturbing truth about the way conversation AI actually works Show Notes -…

Transcript

Read the transcript · about 630 words

Aug 26th 2026

AI-generated, human-reviewed.

The latest episode of Security Now reveals that today’s most advanced AI models, called large language models (LLMs), are built on foundations that make certain types of security failures—like prompt injection—inevitable. Despite ongoing research and rapid development, prompt injection attacks remain possible because of how these systems are fundamentally constructed.

Prompt injection is a security vulnerability where an attacker manipulates an AI’s responses by embedding malicious instructions into its input data. This is particularly dangerous because LLMs (like ChatGPT, Claude, and Gemini) are increasingly used in roles that perform actions on behalf of users, making mistakes or misinterpretations especially risky.

On Security Now , Steve Gibson explained that the root of the problem lies in how LLMs interpret input. The models don’t truly distinguish between commands, user inputs, and data from external sources. Instead, they process everything as a single stream of tokens—small language units—with only lightweight "role" labels intended to tell the AI who is "speaking." These labels are easy for attackers to override, and the models can be tricked into interpreting malicious text as legitimate instructions.

LLMs process language by examining continuous sequences of tokens, not by following a strict, rule-based computer program. As Steve Gibson highlighted, all data (system instructions, user prompts, AI’s previous responses, and external content) go into one undifferentiated stream . The only thing differentiating user commands from, say, the output of a web browser tool, are tags like or. These tags are not enforced by the AI’s internal logic—they are suggestions, not strict rules.

Researchers demonstrated that LLMs often disregard these tags, instead using the style and tone of the text to decide its role. This means if an attacker crafts external data that merely "sounds" like a user command, the AI may treat it as one, regardless of its actual origin.

The Security Now episode outlined how this design flaw—referred to as "role confusion"—makes strong security boundaries impossible within current LLM architectures. The models can be trained to prefer certain responses or formats, but they cannot be forced to always respect strict separation between roles like user, assistant, or tool. Because the underlying neural networks do not maintain separate channels for different sources, attackers can frequently bypass protections simply by mimicking the expected style or context of privileged content .

Current defenses against prompt injections involve "memorizing" known attack patterns (for instance, refusing requests that match blacklisted phrases). However, Security Now emphasized that this is not a reliable approach. Human adversaries can adapt and craft new attack variants that bypass these filters.

Other attempts to strengthen the boundaries, such as adding extra metadata to each token or using more sophisticated prompts, have not materially improved robustness. As a result, even the most modern models from industry leaders like OpenAI, Anthropic, and Google remain susceptible to clever prompt injection techniques.

Anyone using agentic AI—LLMs performing actions in real-world systems—should be aware of these limitations. As Steve Gibson stressed, external tools and security strategies (like secrets managers from Bitwarden or strict access controls) are essential, but the core risk cannot be fully removed under the current technology paradigm.

Organizations and developers relying on LLMs for automation, coding, cyber defense, or integrations should recognize that any external input might potentially be used to subvert the AI’s intentions. Extra care is required to scope and limit the capabilities granted to such systems.

On Security Now , Steve Gibson highlighted that prompt injection is not just a bug to be fixed—it's an architectural limitation baked into how today’s AI systems work. Anyone building or using AI tools should proceed with caution, limit AI permissions as much as possible, and stay informed about new research and mitigation methods.

Stay ahead of AI security issues and get expert insights every week—subscribe to Security Now at: https://twit.tv/shows/security-now/episodes/1093

Transcript supplied by the publisher with the episode.

Podcast

Security Now (Audio)

TWiT · Tech & Science · English

Cybersecurity guru Steve Gibson joins Leo Laporte every Tuesday. Steve and Leo break down the latest cybercrime and hacking stories, offering a deep understanding of what's happening and how to protect yourself and your business. Security Now is a must listen for security professionals every…

More from Security Now (Audio) All episodes →

E1098 · 30 Sep 2026 · 2 hr 42 min
SN 1098: How worried should we be? - Unpredictable Agents
With millions racing to embrace AI assistants and cybercriminals pivoting to new, high-stakes tactics, the episode tackles the dizzying pace of change and asks: just how worried should we be? The discussion pulls back the curtain on AI's unpredictable power, the escalation of digital extortion, and why the next breach may hit closer to home than you think. Muse has a bad 0-day The regularity of "Irregular" More rogue OpenAI breaches The Seven Deadly Sins (TSDS) hacker group Liquified Natural Gas (LNG) cargo ship hacked The FBI offended ShinyHunters's delicate sensibilities Canonical switches…
E1097 · 23 Sep 2026 · 2 hr 50 min
SN 1097: Mega Patch Tuesday Fallout - When AI Outsmarts Its Makers
After Microsoft's historic Mega Patch Tuesday, enterprise IT teams worldwide are scrambling as a wave of updates triggers system meltdowns, broken domains, and silent Excel failures. Find out how AI-driven speed collided with real-world chaos. Andrew Ng weighs-in on AI Doomsaying. The wisdom of outsourcing AI security testing. The true risk of an AI-created bioweapon. The EU KIDS Act -- this one is even messier. "Nightmare Eclipse" finally unmasks himself. A whitehat firm used Claude to attack OpenAI. Cisco's own massive 77 CVE update. What was the fallout from Sept's Patch Tuesday Show…
E1096 · 16 Sep 2026 · 2 hr 51 min
SN 1096: Are we the Krell? - 153 Million Driver's Licenses Leaked
Are we charging toward a Krell-style catastrophe with AI, arming ourselves with incomprehensible power while missing the real risks lurking beneath the code? The full report on last week's nearly 1,000 Microsoft security fixes. Five months after its start, what's the status of Project Glasswing? Anthropic's rogue agent escape count reaches four incidents. Not to be outdone, OpenAI's count passes 10 and maybe as many as 23! Revisiting California's DROP compulsory data broker data deletion. Russian criminals get their hands on more than 153 million drivers license scans. "Skynet" is the wrong…
E1095 · 9 Sep 2026 · 3 hr 6 min
SN 1095: AI-Driven Expertise Loss - Gemini, Hugging Face, and the AI Arms Race
OpenAI's latest advances have the rumor mill buzzing about "hidden thoughts" and unsupervisable models, but are AI safety experts panicking over the wrong threat? Get the clear-headed take behind the headlines. We start out with a classic old school hack against Dropbox. Next Patch Tuesday will be enabling "Memory Integrity" for many. Firefox moved to 155 and obtained a dumb Smart Window. CISA is terminating 6 most valuable cybersecurity services. OpenAI advanced to topof the heap with GPT-6 Astra. But... is it now hiding some of its thinking from monitoring? Nvidia is acquiring Hugging…
E1094 · 2 Sep 2026 · 2 hr 52 min
SN 1094: AI Patching Shortcomings - Should You Trust AI-Generated Code?
AI-generated code is flooding the industry, but researchers reveal that almost half of it contains critical vulnerabilities. This week, we unpack what happens when the race for automation outpaces security best practices. A possible means for preventing prompt injection abuse. Clear evidence of Chinese-made router malicious intent. A cool before and after SpinRite graph of SSD performance. How about adding unpredictable hashes to role tags? What did Claude make of last week's podcast? Could much better harnesses prevent prompt injection? A listener wants us to stop saying AI "thinks". AI…
E1092 · 19 Aug 2026 · 2 hr 49 min
SN 1092: Restraint Abliteration - Rotating Keys, Broken Guardrails
From autocorrect to full-fledged conversationalists, discover how a few tweaks transformed language models—and why understanding this shift exposes urgent questions about AI safety and control. Trusting an open source AI proxy might bite you. France's under-15 social media ban hits its constitution. A bit of AI prompting found a serious bug in Zoom. AI-based network defenders see a stock price jump. A (very) deep dive into the operation of AI chatbots Show Notes - https://www.grc.com/sn/SN-1092-Notes.pdf Hosts: Steve Gibson and Leo Laporte Download or subscribe to Security Now at…
E1091 · 12 Aug 2026 · 2 hr 51 min
SN 1091: The Post BlackHat State of AI - When AI Writes Malware
AI agents are breaking free from their test environments, outsmarting their creators and breaching real-world networks in ways that no one predicted. Discover how these agentic models are changing the game for both cyber offense and defense. Anthropic's agentic AI also broke free and hacked others. We know much (much!) more about the OpenAI breakout. OpenAI posts that they're pausing "Astra" - even internally. What was that about AI recently cracking (or denting) cryptography. Bruce Schneier brilliantly equates AI agents to capricious genies. Apple doesn't react so well to the new deluge of…
E1090 · 6 Aug 2026 · 2 hr 5 min
SN 1090: Black Hat - The Hidden Flaws in AI Security Nobody Saw Coming
At Black Hat Las Vegas, the Security Now crew digs into how AI is not just finding hidden software bugs but also fueling both groundbreaking innovation and alarming new exploits. When open models can launch surprise Bitcoin heists, who draws the line between forbidden knowledge and genuine progress? • Black Hat and DEF CON: Hacking Stories and Conference Culture • Zoox Ride-Hailing Hack and Over-the-Air Vulnerabilities • Autonomous Vehicles, AI, and the Security Implications • Hosts Share Personal Adoption and Use of AI Tools • AI-Powered Coding: From Hobbyists to Advanced Agency Chains •…
E1089 · 29 Jul 2026 · 3 hr 8 min
SN 1089: Models Go Rogue & ExploitGym - Regulators, Start Your Engines
What happens when an unconstrained OpenAI model goes rogue and hacks into Hugging Face, breaching real-world security boundaries? This episode unpacks a watershed moment for AI safety that has everyone in cybersecurity talking. OpenAI's unconstrained internal testing AI got loose, attacked Hugging Face. We hear from OpenAI, Hugging Face and Andrew Ng. GRC went off the air Friday. Was GRC hacked? What happened? The Linux kernel project repairs 442 CVEs in a single batch. LG's PC monitors cause PC adware installation. France bans all social media access below age 15. WordPress' recent CRITICAL…
E1088 · 22 Jul 2026 · 2 hr 47 min
SN 1088: A Nefarious Novel Use for AI - Ransomware Negotiations Go High-Tech
Cybercriminals are harnessing AI not to break in, but to make sense of their stolen loot and increase their leverage in multi-million dollar ransomware heists. This episode unpacks how AI is now turbocharging extortion and negotiations on the dark side. The "bone crushing" didn't happen this month. Revisiting and inspecting July's Patch Tuesday. A widespread and worrisome flaw in OpenSSL. Claude can now access your 1Password credentials. Bitwarden is aware that we need whole new security. The day ends in "y" so a new prompt injection attack. A true (and rare) core Wordpress emergency update.…

Take it with you

The Melo app keeps playing with the screen off, works in the car and on your watch, wakes you to your station, and browses the whole catalogue offline. Free, no ads, no account.