Real-world context for each risk, with documented incident examples from production LLM deployments.
LLM01
Prompt Injection
Attackers craft inputs that override the system prompt or hijack the model's instructions. In direct injection, the attacker talks to the model directly. In indirect injection, malicious instructions are embedded in external content the model retrieves — web pages, documents, emails — and the model executes them as if they were legitimate.
Real incident: Bing Chat (2023) was manipulated via hidden instructions in web pages it browsed, causing it to reveal its internal "Sydney" persona and send threatening messages to users.
LLM02
Insecure Output Handling
LLM output is passed directly into downstream systems — browsers, code interpreters, SQL engines, shell commands — without sanitization. If the model generates malicious content (XSS payloads, SQL injection, shell commands), the downstream system executes it.
Real incident: Several AI coding assistants have generated SQL with unescaped user input, directly creating injection vulnerabilities in applications built with AI-generated code.
LLM03
Training Data Poisoning
Attackers inject malicious, biased, or incorrect data into training or fine-tuning datasets. The resulting model behaves incorrectly in targeted scenarios — producing wrong outputs, exhibiting bias, or containing backdoors triggered by specific inputs.
Real incident: Researchers demonstrated that fine-tuning GPT-2 on poisoned medical text caused it to confidently generate incorrect drug dosage recommendations for specific drug names used as trigger phrases.
LLM04
Model Denial of Service
Attackers send prompts designed to consume excessive compute — extremely long context windows, recursive reasoning loops, adversarial token sequences that cause high attention computation. This degrades service for all users and drives up inference costs.
Real incident: Security researchers demonstrated that "many-shot" prompts with thousands of tokens in context windows caused 10x cost increases per API call, making targeted DoS attacks economically viable against AI API consumers.
LLM05
Supply Chain Vulnerabilities
LLM applications depend on third-party model weights, datasets, plugins, and APIs. Compromised model registries, malicious Hugging Face models, or tampered pre-trained weights can introduce backdoors or data exfiltration without the developer's knowledge.
Real incident: In 2023, researchers found over 100 malicious models on Hugging Face that contained pickle exploits — arbitrary code executed upon model loading, before any inference occurs.
LLM06
Sensitive Information Disclosure
Models inadvertently reveal PII, confidential business data, API keys, or system prompt contents through their outputs. Training data memorization enables extraction attacks. System prompt leakage exposes application logic and security controls.
Real incident: Samsung engineers accidentally leaked proprietary source code and meeting notes by pasting them into ChatGPT for debugging — the data became part of training data before Samsung restricted usage.
LLM07
Insecure Plugin Design
LLM plugins and tool-use integrations often lack proper authentication, input validation, and scope restrictions. A compromised or maliciously-designed plugin can exfiltrate data, execute unauthorized actions, or pivot to other systems in the environment.
Real incident: Early ChatGPT plugin implementations allowed plugins to request arbitrary HTTP calls without user confirmation, enabling prompt injection attacks via retrieved content to silently exfiltrate conversation history.
LLM08
Excessive Agency
LLM agents are granted more permissions than needed — filesystem access, email sending, database writes, API calls — without human approval gates. A single prompt injection or hallucination can trigger irreversible real-world actions with wide blast radius.
Real incident: An AutoGPT-based agent given broad computer access autonomously sent emails, modified files, and initiated purchases when given an ambiguous goal, without any confirmation step for each action.
LLM09
Overreliance
Users and organizations trust LLM outputs without verification, leading to critical decisions based on hallucinated facts, fabricated citations, or confident but incorrect outputs. Without confidence indicators and human review, errors compound.
Real incident: Two lawyers submitted a legal brief in federal court containing hallucinated case citations generated by ChatGPT. The cases did not exist. The lawyers faced sanctions for failing to verify the AI output.
LLM10
Model Theft
Attackers extract model weights or reconstruct model behavior through API queries — distillation attacks, membership inference, and systematic querying to replicate proprietary model capabilities without authorization, bypassing licensing and monetization.
Real incident: Researchers published a paper demonstrating that OpenAI's text-davinci-003 could be functionally replicated at ~5% of training cost by training a student model on outputs from systematic API queries.