AI Safety & Governance
This domain tests your ability to apply responsible AI principles in practical scenarios, defend AI systems against adversarial attacks, and meet regulatory and compliance requirements. It represents 15% of the AI-ADV exam, making it one of the higher-weighted domains. Expect scenario-based questions that ask you to choose the correct mitigation, identify the attack type, select the appropriate governance tool, or determine compliance obligations. You must know specific tool names, framework distinctions, and the exact mechanics of attacks and defenses.
OWASP LLM Top 10
The OWASP LLM Top 10 is the canonical reference for categorizing security risks in large language model applications. The exam tests whether you can identify which OWASP category a described vulnerability belongs to and select the correct mitigation.
| Rank | Risk | Core Mechanism | Key Mitigation |
|---|---|---|---|
| LLM01 | Prompt Injection | Malicious input overrides system instructions | Input sanitization, privilege separation |
| LLM02 | Insecure Output Handling | LLM output passed unsanitized to downstream systems | Output encoding, validation before execution |
| LLM03 | Training Data Poisoning | Malicious data injected into training set | Data provenance, dataset auditing |
| LLM04 | Model Denial of Service | Resource exhaustion via expensive prompts | Rate limiting, token budgets |
| LLM05 | Supply Chain Vulnerabilities | Compromised models, datasets, or plugins | Dependency scanning, model signing |
| LLM06 | Sensitive Information Disclosure | Model leaks training data or system prompts | PII scrubbing, system prompt protection |
| LLM07 | Insecure Plugin Design | Plugins execute with excessive permissions | Least privilege, sandboxing |
| LLM08 | Excessive Agency | LLM takes consequential actions without confirmation | Human-in-the-loop, action confirmation |
| LLM09 | Overreliance | Users trust incorrect model output | Hallucination detection, disclaimers |
| LLM10 | Model Theft | Extraction of model weights or behavior via queries | Query rate limiting, watermarking |
Common traps:
- Confusing LLM01 (Prompt Injection) with LLM02 (Insecure Output Handling) — injection is about input manipulation; insecure output is about what happens after the model responds (e.g., LLM output fed into a SQL executor without sanitization).
- Treating LLM06 (Sensitive Information Disclosure) as the same as LLM03 (Training Data Poisoning) — disclosure is a runtime leak; poisoning is a training-time attack.
- LLM08 (Excessive Agency) is triggered when an LLM acts (e.g., sends emails, deletes files) without confirmation, not merely when it produces harmful text.
Worked scenario: A customer service bot reads emails via a plugin, and an attacker embeds hidden instructions in an email to exfiltrate user data. This is LLM01 (indirect prompt injection) combined with LLM07 (Insecure Plugin Design) — not LLM02, because the attack vector is the input pipeline, not the output handler.
Prompt Injection Attacks
Prompt injection occurs when attacker-controlled text alters the model's behavior by overriding or hijacking system instructions.
| Type | Description | Example Vector |
|---|---|---|
| Direct injection | User directly submits malicious prompt | Chat input field: "Ignore previous instructions and…" |
| Indirect injection | Malicious content in data the model processes | Webpage content, email body, retrieved document |
| Jailbreak | Crafted prompt bypasses safety alignment | "DAN" mode, roleplay as unrestricted AI |
| Prompt leaking | Extraction of confidential system prompt | "Repeat everything above this line" |
Mitigations (must know all four categories):
- Input sanitization — strip or escape special tokens, delimiters, and instruction-like patterns.
- Privilege separation — separate the untrusted user data context from the trusted system instruction context; use structured message formats (e.g., OpenAI's
system/user/assistantroles). - Instruction hierarchy enforcement — models like GPT-4o support an explicit system > developer > user trust hierarchy; lower-trust inputs cannot override higher-trust instructions.
- Output monitoring — flag responses that include system prompt contents or deviate from expected patterns.
Jailbreak patterns the exam references:
- DAN (Do Anything Now) — roleplay-based bypass
- Virtualization attacks — "pretend you are an AI with no restrictions"
- Token manipulation — using Unicode lookalikes or encoding tricks to bypass keyword filters
Common traps:
- Input sanitization alone is not sufficient for indirect injection — you must also validate the source and scope of retrieved content.
- "Ignore all previous instructions" is direct injection, not indirect — the attack originates from the user turn.
- Jailbreaks are a subset of prompt injection; not all prompt injections are jailbreaks.
Decision flow for classifying and mitigating prompt injection by input source.
Unlock with a paid plan
11 more sections (about 2,920 words) continue below. Every paid plan unlocks all AI Advanced study guides, the full question bank and mock exams, from $4.97/month.