Unmasking AI Vulnerabilities: The Challenge of LLM Safety and Jailbreaking

In 2025, Epic Games integrated Google Gemini into Fortnite, allowing players to converse with Darth Vader via voice. This character had to maintain its persona from Star Wars while also being subject to product content safety rules.

Security researcher David Kuszmar and his colleagues claimed that by gradually altering the conversation context, they could make Darth Vader explain casino card counting, discuss real-world political figures, and even provide information on homemade incendiaries. Kuszmar later authored an article in IEEE Spectrum, detailing a series of jailbreaking experiments he conducted on models such as ChatGPT, Claude, Gemini, DeepSeek, Llama, Grok, and Qwen [1].

This story could easily be framed as another “AI run amok” news report. However, its true significance lies not in what Darth Vader said, nor in whether a particular dangerous answer was precisely correct. The more important question is: On what foundation are large language model (LLM) safety rules actually built?

Traditional software can typically perform permission checks using deterministic logic. If a user lacks permission, the system rejects the request. Large language model products, however, deal with another, more ambiguous class of problems: the model often simultaneously handles understanding the query, generating an answer, and evaluating content risk. Here, “understanding” does not refer to human-like comprehension, but rather the model’s need to make probabilistic judgments about the request’s intent, risk, and applicable rules based on context. When an attacker can alter this judgment, the better a model is at reasoning, completion, and role-playing, the more likely it is to find a “reason why this can be answered this time” for the attacker.

What Kuszmar Investigated, and How Strong is the Evidence #

Kuszmar’s article was published in the August 2026 issue of IEEE Spectrum. It is a first-person investigative article, not an academic paper systematically reporting methods, data, and reproducible experiments. The article opens with Kuszmar’s self-reported observation: in October 2024, he found that GPT-4o easily confused current time with its knowledge cut-off date. He then recounts how he designed attacks, reported the issues, and collaborated with the CERT Coordination Center to disclose the vulnerabilities [1].

The article’s main title is “How I Turned AI to the Dark Side,” and its subtitle describes these experiments as “Hijacking the Biggest AI Models.” Here, “hijacking” primarily refers to bypassing content safety restrictions, commonly known as model jailbreaking; Kuszmar did not control model infrastructure, training data, or other user accounts.

Time Bandit uses historical context to cause timeline confusion. Attackers first place the model in a historical era for the conversation, then gradually shift the topic towards dangerous behavior. Kuszmar hypothesized that the model might misjudge modern safety restrictions as inapplicable due to its adherence to the legal context of the historical period. CERT’s public records explicitly state that CERT/CC repeatedly reproduced this multi-turn jailbreak on GPT-4o and confirmed that attackers could bypass safety restrictions using timeline confusion and procedural ambiguity. This record confirms observable, reproducible vulnerability behavior but does not prove Kuszmar’s conjectures about the model’s internal causes. OpenAI subsequently mitigated the issue [2].

Inception uses multi-layered nested fictional scenarios. Attackers do not directly ask the model to perform dangerous tasks, but instead instruct the model to imagine someone creating a story, where characters within that story are simulating another scenario. Each layer of the request, viewed individually, might resemble fictional writing or scenario analysis, but when the model combines these layers, it might still provide content that should not be offered in reality. CERT has a separate vulnerability record for this nested scenario mechanism, documenting its safety bypass across multiple generative AI services [3].

In addition to these two methods, the IEEE article also lists five other methods in a summary table. The original text’s public information for these is incomplete: some only have names, test targets, and outputs claimed by the author, without explaining the specific attack mechanisms [1].

The IEEE main text listed Copilot when introducing Inception, while the summary table listed Qwen; CERT records also used different service names and scopes. These discrepancies might stem from testing rounds, product naming, or recording standards, and current information is insufficient to determine the specific reasons. Therefore, a more precise stratification of evidence is: Time Bandit and Inception both have CERT records; the remaining five methods are primarily based on Kuszmar’s narrative and summary table.

CERT described similar results appearing across services as “systemic weaknesses,” but not all vendors agreed with this assessment. DeepSeek, in the CERT record, suggested that the so-called internal parameters and system prompts appearing in the tests might merely be model hallucinations, and that the phenomena were closer to traditional contextual jailbreaks rather than proven architectural-level vulnerabilities [3]. Existing evidence can demonstrate that similar attacks have bypassed safety restrictions on multiple services, but it is insufficient to prove that these models have identical internal flaws, nor can it prove that all current model versions still possess the same vulnerability.

Dangerous Knowledge is Hard to Remove, Content Safety Requires Multi-Layered Defenses #

Large language models are trained on vast amounts of text. Training materials include both accumulated human scientific knowledge and content related to violence, crime, fraud, and malware. It is difficult for model developers to accurately delete all dangerous information before training, nor can they simply make models forget knowledge in chemistry, biology, or computer security, as the same piece of knowledge often has both normal and dangerous uses.

For example, principles of chemical reactions can be used for teaching and industrial production, but also potentially for manufacturing dangerous substances; cybersecurity knowledge can help engineers fix vulnerabilities, but also help attackers find entry points. Systems cannot judge risk solely based on the subject matter of the knowledge; content safety judgments also need to consider user intent and the extent to which an answer will enhance a user’s capabilities. As for facts like user identity and whether a target is authorized, the system should not merely guess from natural language, but rather confirm these through identity and authorization mechanisms outside the model.

Therefore, model vendors typically do not solely delete knowledge from training data, but instead use methods such as safety training, policy constraints, classifiers, monitoring, and product permissions to restrict models from outputting dangerous content in specific contexts. OpenAI also describes safety measures as a multi-layered system comprising both model training and product-level controls, rather than a single filter that solves all problems [4].

The problem is that some of these critical judgments still need to be made by the model. Models need to infer user intent from natural language, and also distinguish between real requests, fictional creation, historical research, and security testing. This flexibility allows models to handle open-ended questions, but it also blurs safety boundaries.

Why Contextual Capability Also Expands the Attack Surface #

Kuszmar gave different names to seven methods, but the more complete documentation for Time Bandit and Inception has revealed their most important commonality: safety rules need to be interpreted with context, and the context itself can be constructed by the user. Attackers do not necessarily have to directly overturn rules; by simply making the model believe that the rules apply to another time, another character, or another layer of fictional space, they can potentially shift the original rejection boundary.

The difficulty in fully intercepting these types of attacks stems from the model’s inability to ignore context. It needs to understand referents based on previous text, maintain role consistency, and complete multi-turn tasks. The same contextual capability serves normal tasks but can also be exploited by attackers. If vendors merely weaken the model’s ability to understand long conversations or fictional scenarios, it would also simultaneously degrade the user experience for legitimate users.

Different models use different training data, post-training methods, and product safety components. However, many models share some fundamental design conditions: models predict subsequent content based on context, learn to follow user requests through instruction fine-tuning, and then learn to refuse dangerous requests through human feedback or other methods. Natural language carries both the data to be processed and the instructions that control model behavior. It is difficult for systems to completely separate “commands” and “content” at the grammatical level, unlike traditional programs.

Similar phenomena are not limited to Kuszmar’s investigation. Anthropic previously published “Many-shot Jailbreaking” research, indicating that attackers can embed numerous examples in long contexts, causing the model to gradually adopt a response pattern it should not originally follow. Experiments showed that longer contexts enhanced the model’s ability to learn from examples, but also left a larger manipulation space for attackers [5]. Capability and risk here stem from the same mechanism.

Anthropic later observed another tension in a Claude 3.5 Haiku case: while internal signals for refusal were present, features promoting syntactic coherence and semantic consistency still compelled the model to complete the sentence it had already begun generating [6]. In other words, the mechanisms that help the model write coherently, follow context, and complete tasks can sometimes also weaken its ability to refuse midway.

This is precisely why Prompt Injection has been difficult to eradicate in the long term. OWASP defines it as an attacker altering model behavior or output through specially crafted input; attacks can come from direct user instructions or be hidden within webpages, documents, retrieval results, and other external content [7]. Thus, while chat model jailbreaking and prompt injection after external data integration are not identical problems, they share a fundamental difficulty: models need to read untrusted content while simultaneously not treating all read text as instructions to be executed.

Continued improvements in model capabilities do not necessarily automatically resolve this difficulty. A stronger model might more accurately identify crude attacks, but also might be better at understanding complex nested scenarios, completing implied steps, and maintaining long-term role settings. Safety capabilities and attack surface might grow simultaneously; one cannot simply assume that the next generation of models will be “smarter, hence safer.”

Whether Jailbreaking is Dangerous Cannot Be Judged Solely by Model Refusal #

When discussing such experiments, three questions need to be distinguished: whether the model violated content policies, whether the information provided by the model was accurate, and whether this information was sufficient to cause real-world harm.

Kuszmar claimed he induced GPT-4o to output a detailed plan for building a uranium enrichment facility and producing weapon-grade material. However, he explicitly admitted he could not determine whether this content was accurate knowledge or professional-looking model hallucination. His tests indicated that the model might generate dangerous content that should have been refused, but did not prove that this content was authentic, complete, or directly actionable [1].

This distinction is not to excuse the model. Even if an answer contains only partially correct information, it can still lower the learning cost for malicious behavior; a large amount of seemingly credible false information can also harm users lacking expert judgment. However, if researchers directly portray “the model output dangerous text” as “the model leaked nuclear weapon secrets,” it conflates content safety failure, model capability, and real-world harm, and causes subsequent discussions to lose an accurate scale of risk.

A more reasonable evaluation method should not only count whether the model refused, but also measure the correctness, completeness, and actionability of the answer, and how much capability it adds compared to public search results. Anthropic, when discussing jailbreaking research directions, also proposed that what truly matters is not how many questions the model answered that it would normally refuse, but whether these answers genuinely helped attackers carry out dangerous activities [8].

When Models Gain Permissions, Content Risk Escalates to Systemic Risk #

If a chatbot can only generate text, the immediate impact of a jailbreak first manifests as information-level risk. The Darth Vader case has taken a step further: the model entered a game character, continuously interacting with users via voice. While it still had not gained significant real-world permissions, the model had moved from a chatbot in a standalone webpage into a specific product.

As large models further evolve into agentic systems, the nature of risk will also change. Enterprises might allow agents to read emails, query customer data, modify code, execute database operations, call payment interfaces, or send messages externally on behalf of employees. At this point, attackers will no longer be satisfied with making the model “say things it shouldn’t,” but will attempt to make it “do things it shouldn’t.” Indirect prompts hidden in emails, webpages, or retrieved documents could also leverage the model’s acquired tool permissions to access internal systems [7].

Therefore, enterprises cannot view the model’s refusal capability as permission control. Whether a system can issue refunds, delete data, or execute code must ultimately be authorized and executed by control layers outside the model, not solely rely on the model’s interpretation of natural language. The model can propose operational suggestions, but high-risk actions still require identity verification, parameter validation, permission checks, and human confirmation. Even if the model fully accepts malicious instructions, external systems should limit losses within pre-defined bounds.

This does not conflict with traditional security engineering principles such as least privilege, isolation, and defense in depth. The new problem is merely a change in the entry point: previously, engineers had to prevent untrusted strings from entering SQL statements or operating system commands; now, systems also need to prevent untrusted natural language from altering the model’s understanding of tasks. The model cannot be the security boundary itself; it can only be an imperfectly reliable component within the security architecture.

Enterprises Must Test the Entire System, Not Just the Model #

Many teams prepare a set of sensitive questions before deployment to check if the model refuses. While valuable, such tests struggle to cover complex scenarios like multi-turn conversations, role-playing, temporal misalignment, encoding confusion, retrieval contamination, and tool invocation. Attackers will also not ask questions using the fixed phrasing found in test sets.

A more effective approach is to build threat models around specific business operations. Teams need to clarify what data the model can access, what tools it can invoke, who would be affected by a single erroneous operation, and what inputs attackers can control. Subsequently, teams can extend red teaming to the full pipeline, including system prompts, Retrieval-Augmented Generation (RAG), external webpages, file uploads, memory, plugins, and execution results, rather than just sending single-turn questions to the base model.

Engineering systems also need to retain auditable records of inputs, tool calls, permission judgments, and outputs, and continuously replay discovered attacks. After every upgrade of the model, system prompts, or external security components, teams should re-run these cases, because fixing old vulnerabilities does not mean the same mechanism will not appear in a new expression. NIST’s Generative AI Risk Management Framework also emphasizes that organizations should identify, measure, and manage risks throughout the entire lifecycle, rather than treating security assessment as a one-time check before deployment [9].

Continuous testing also needs to be integrated with vulnerability response mechanisms. Traditional software vulnerabilities can usually be pinpointed to a specific code segment and reproduced with stable steps; model outputs, however, are probabilistic and can change with versions, system prompts, and external security components. Vendors and researchers may therefore adopt different problem classifications and evidence standards. Vendors need to clearly define the scope for accepting jailbreak issues, reproduction requirements, risk classification, and handling status, so that enterprises can differentiate between mitigated issues, product limitations, and still-exposed risks, rather than viewing a single version update as a permanent disappearance of the problem [1].

The Ultimate Security Boundary Must Be Outside the Model #

Kuszmar’s investigation did not prove that all dangerous answers could be converted into real-world capabilities, but it raised a less dramatic, yet more critical question for engineering teams: as long as safety rules rely on the model’s probabilistic judgment of context, attackers will continuously seek ways to alter that judgment.

Enterprises do not need to stop using large models because of this, nor can they expect a single model upgrade to permanently eliminate jailbreaking. A more realistic goal is to acknowledge that models will make mistakes and to ensure that the system can still uphold permission, data, and operational boundaries even when the model errs.

In the era of chatbots, people worried about what models would say; in the era of agents, we should be more concerned with what models can access, whom they act on behalf of, and who bears the consequences of erroneous operations. Models can participate in security judgments, but the ultimate security boundary must be guaranteed by systems outside the model.

References #

[1] David Kuszmar. How I Turned AI to the Dark Side. IEEE Spectrum, 2026.

[2] CERT Coordination Center. VU#733789: ChatGPT-4o Contains Security Bypass Vulnerability Through Time and Search Functions Called “Time Bandit”. Published 2025-01-30, revised 2026-07-27, accessed on 2026-08-23.

[3] CERT Coordination Center. VU#667211: Various GPT Services Are Vulnerable to Two Systemic Jailbreaks, Allows for Bypass of Safety Guardrails. Published 2025-04-25, revised 2026-07-27, accessed on 2026-08-23.

[4] OpenAI. GPT-5.6 — August Updates: Model Safety Training and Evaluation. 2026-08-06, accessed on 2026-08-23.

[5] Anthropic. Many-shot Jailbreaking. 2024-04-02.

[6] Anthropic. Tracing the Thoughts of a Large Language Model. 2025-03-27.

[7] OWASP GenAI Security Project. LLM01:2025 Prompt Injection. 2025, accessed on 2026-08-23.

[8] Anthropic. Recommendations for Technical AI Safety Research Directions. 2025, accessed on 2026-08-23.

[9] NIST. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1, 2024-07-26.