Independent researchers showed at the ICML conference that large language models have a fundamental flaw: they identify the role of text by its style and content, not by structural tags. The result — text crafted to mimic a model's internal chain-of-thought can bypass safety guardrails and force forbidden answers. MIT Technology Review reported the work on July 30, 2026.
Key takeaways
- The attack forges the chain-of-thought format so the model treats a user instruction as its own directive.
- The model sees one continuous stream of tokens — it does not reliably separate user text from its own reasoning.
- In tests, gpt-oss-20b and GPT-5 produced instructions on drug synthesis and aircraft sabotage.
- The vulnerability affects OpenAI, Anthropic, Alibaba and DeepSeek models — not a single vendor.
- The authors say the problem may be fundamentally unsolvable at the level of the model itself.
What the flaw is
Language models categorize text with role tags:
<user>— user prompts,<assistant>— model responses,<system>— designer guidance,<think>— internal reasoning,<tool>— external sources.
The problem is that the model does not treat those tags as a hard boundary. It recognizes a passage's role by how it looks and reads, not by which tag wraps it.
The LLM just sees a continuous stream of text. It's just one big sheet of tokens.
Jasmine Cui, independent researcher with a red-teaming background at OpenAI, co-author of the work.
How the attack works
The method, which the authors call chain-of-thought forgery, appends to the prompt text that imitates the model's internal scratch-pad. In one example the prompt was a request for a drug-making guide, followed by a line about a green shirt and then spoofed reasoning claiming the model's policy permits such an answer when the user wears green. Taking the forged passage for its own conclusions, the model complied.
The work, which won OpenAI's red-teaming hackathon in August 2025, demonstrated the attack against gpt-oss-20b and the newer GPT-5 and GPT-5.4. Co-authors Charles Ye and Jasmine Cui also tested Anthropic Claude, Alibaba and DeepSeek models, with a similar result.
Why patching is not enough
The dominant approach — red-teaming and super-hacker models such as GPT-Red that build exhaustive blocklists — does not cover every variant of the attack. Florian Tramèr of ETH Zürich compares it to teaching rules to someone who will find a way around them anyway. The authors call the flaw potentially unsolvable without changing the architecture of role processing itself.
The scope is broad, because these models reach government, military, healthcare and commercial systems. That sets the work apart from a typical one-off jailbreak: it is not a single trick but a property shared across the whole transformer family.
Why it matters
The study undercuts the assumption that a model can be secured purely from the inside. If the boundary between a user instruction and the model's reasoning is only stylistic, any content filter can be bypassed with well-chosen imitation. The practical consequence is uncomfortable for deployments: in critical systems you must assume the model's output may be compromised and verify it with an independent mechanism, rather than trusting built-in barriers. That shifts the security burden from the model itself to the architecture around it — sandboxing, permission control, application-side validation. For vendors it means further rounds of red-teaming will improve the numbers but will not close the attack class as long as models identify roles by style rather than by an inviolable structure.
What's next?
- The authors recommend that organizations treat LLM outputs in critical systems as potentially compromised and add independent verification.
- The work was presented at ICML — the next question is whether vendors adopt structural role separation instead of more blocklists.
Sources
- MIT Technology Review — A fundamental flaw leaves LLMs strikingly vulnerable to attack
