Chapter 3. Data protection and permissions

Learning objectives

• Find and minimize leakage points across the data lifecycle, from input to logs.
• Diagnose excessive agency in terms of functionality, permissions and autonomy, and reduce permissions through tool design.
• Determine whose authority agents use, and minimize delegation tokens and scopes.
• Treat model output as untrusted input and render and process it safely for each destination.
• Correct common misconceptions about system prompts and vector stores.

Introduction

An online store's support chatbot used a tool that retrieved entire customer accounts to answer accurately. A single shipping question brought a full address, phone number, ten recent orders and reward-point history into context. Most days, nothing went wrong. Then a customer sharing an account with family asked, "Show me all last month's orders," and the chatbot helpfully displayed other family members' orders too. The model had made no mistake; it had simply organized the data it received. The team hurriedly added "do not disclose other people's information" to the prompt, but the model had no way to know which family member sharing the account was asking. The solution lay in the retrieval tool, rather than the prompt.

Where Chapter 2 covered "how an attacker's instructions enter," this chapter covers "what they can do once inside." A model can say what is in its context and do what its tools allow. Reducing data and permissions is therefore half of injection defense. This chapter begins with sensitive-information paths and minimization, then covers external AI services' data terms, system prompts and vector stores, excessive agency and tool permission design, and delegation and identity. Finally, it teaches safe rendering and processing of model output.

Key concepts

Where sensitive information leaks: the data lifecycle

OWASP LLM02 (Sensitive Information Disclosure) covers the risk of personal information, financial information, health records, confidential business data, credentials and legal documents appearing in model output. It also notes that user-supplied data may later reappear in model output. Looking only at the final answer misses many points, so follow the entire lifecycle.

OWASP warns that system prompt restrictions such as "do not return this data" can help but are not always followed and can be bypassed through injection. The most reliable protection is never showing the data to the model in the first place.

Minimization, redaction and pseudonymization

Data minimization means "pass only the fields needed for the task, only when needed." Shipping inquiries need shipment status and an expected delivery date, not a full address. Implement this through tool design rather than prompts. Replace tools returning whole customer records with tools returning only "this order's shipping status." Korea's Standards for Measures to Ensure the Security of Personal Information likewise require specifying the purpose when outputting personal information and minimizing output fields accordingly. Apply the same principle when passing data into LLM context.

Redact values that must otherwise be passed.

With tokenization, the mapping becomes a critical new asset. Keep it separate from the model's storage, encrypted, with independently controlled and logged access. Redact before external transmission, and restore original values after receiving responses only where necessary.

Terms for data sent to external AI services

Using an external provider's model takes data beyond the organization's boundary. Ask whether it is used for training, how long it is retained for abuse monitoring, how long application state (conversations, files, vector stores) remains, what conditions apply to reduced-retention options and where it is stored.

OpenAI's data controls documentation provides an example. API data is not used for model training unless the user explicitly opts in. Abuse monitoring logs are retained for up to 30 days, but may be kept longer if legally required or necessary to protect the service and third parties from harm. Zero Data Retention and Modified Abuse Monitoring require prior approval. Some features retain application state until users delete it. Terms vary by feature and contract, so review official documentation and contracts at the feature level and record the date checked. If sending personal information to an overseas provider, review the cross-border transfer requirements of PIPA Article 28-8 (separate consent or statutory exceptions) with your privacy officer.

System prompts are not secrets

The central point of OWASP LLM07 is that a system prompt should not be considered secret or used as a security control. The same document explains that the real risk is the sensitive information and permission structures placed inside it, rather than exposure of the prompt wording itself.

Perform privilege separation and authorization outside the model in deterministic, auditable ways. Split tasks requiring different access levels among agents with least privilege. The goal is "a design where publishing the whole prompt has no consequences." OWASP's 2026 edition broadens this item to LLM08:2026 Hidden Context Exposure, applying the same principle to developer instructions, retrieved policy text and tool schemas.

Permissions in RAG and vector stores

OWASP LLM08 (Vector and Embedding Weaknesses) identifies access control failures and leakage, cross-context leakage between user groups sharing a store, embedding inversion that reconstructs significant parts of source text, and index data poisoning. Mitigations include permission-aware vector stores and strict partitioning, knowledge-source validation, review and classification of combined data, and immutable retrieval activity logs.

The design principle is "apply permissions at retrieval, before generation." Letting the model read unauthorized documents and then telling it not to mention them repeats the mistake of using a prompt as a control. Apply pre-filtering by adding the requester's authorization conditions to the retrieval query itself, so only allowed chunks return. When document permissions change or a document is deleted, update index chunks and caches too. Documents deleted at the source commonly remain searchable in indexes. Treat embeddings as data that is just as sensitive as the original text. Being an array of numbers is no protection.

Excessive agency: three roots and eight preventive measures

OWASP LLM06 (Excessive Agency) is a vulnerability that allows harmful actions in response to unexpected, ambiguous or manipulated model output. Whether the cause is hallucination or injection, the key question is what can happen when the model makes a wrong decision. There are three roots.

The 2025 edition gives eight preventive measures: minimize extensions (tools), minimize functionality per tool, avoid general-purpose tools, minimize tool permissions, execute in the user's context, require user approval, apply complete mediation, and sanitize input and output. The 2026 edition (LLM03:2026) removes input/output sanitization from this list, leaving seven. Logging, monitoring and rate limiting are separately classified as measures that reduce impact but do not prevent excessive agency itself.

Complete mediation means "the downstream system, rather than the model, decides whether to allow every request." It is like having a door mechanism check access badges instead of relying on a receptionist's memory. The model may not be able to forge a badge, but it can persuade the receptionist.

Turning least-privilege tools into a design

Least privilege must appear in tool specifications, not just as a principle. The OWASP AI Agent Security Cheat Sheet recommends defining permissions per tool by read/write action and target resource, giving agents at different trust levels different tool sets, and failing closed for unknown tools or missing approval requirements.

{
  "name": "reschedule_my_booking",
  "description": "Changes the date of one booking belonging to the logged-in user. Requires user confirmation before execution.",
  "input_schema": {
    "type": "object",
    "properties": {
      "booking_id": { "type": "string", "pattern": "^BK-[0-9]{8}$" },
      "new_date": { "type": "string", "format": "date" }
    },
    "required": ["booking_id", "new_date"],
    "additionalProperties": false
  }
}

The schema helps the model avoid malformed arguments; the real control is on the server. On every request, the server checks again whether the booking belongs to the logged-in user, the new date is allowed and the user has confirmed it.

Delegation and identity: whose authority is used?

When agents call other systems for users, "whose authority" must be clear. If every call uses a powerful shared service account, a user who tricks the model can reach every record accessible to that account, while logs fail to identify the requester. The OWASP Agentic Top 10 says true least privilege cannot be enforced without agents having managed identities of their own, listing this as a cause of ASI03 (Identity and Privilege Abuse). OWASP LLM06 recommends acting for users in their context with least privilege, for example through minimally scoped OAuth authorization. MCP security best practices spell out delegation pitfalls.

Token passthrough undermines downstream API rate limits and audit records and can let a token thief use the intermediary as an exfiltration path. Obtaining broad scopes up front opens all tools as soon as one token leaks.

Output is untrusted input

OWASP LLM05 (Improper Output Handling) is the failure to validate, sanitize and handle model output before passing it to downstream components. Since prompts can manipulate output, using it unchanged is equivalent to giving users indirect access to downstream functionality. OWASP recommends a zero-trust approach that treats the model like any other user.

Impact increases when the model has greater privileges than the user, is vulnerable to indirect injection, or output is not encoded for its context. OWASP also covers installing malicious packages that attackers publish under hallucinated package names in this item and LLM09.

Rendering output safely

Rendering model output on screen follows the same rules as displaying user input. Applying the OWASP XSS Prevention Cheat Sheet to LLM output gives the following.

1. Render text by default. Use methods that treat data only as text, such as textContent rather than innerHTML.
2. If formatting is necessary, disable raw HTML in the Markdown renderer and filter the result through an allowlist-based sanitizer. OWASP recommends DOMPurify for HTML sanitization.
3. Do not use framework escape hatches such as React's dangerouslySetInnerHTML without sanitization.
4. Allowlist links and images. Disabling automatic external-image loading blocks exfiltration through image URLs. Check link schemes (such as javascript:) and domains, and show the actual address before clicking where necessary.
5. Content Security Policy (CSP) is an additional defense layer. OWASP advises against making it the primary defense.
6. Streaming output is rendered before all chunks arrive. Check that unsanitized intermediate output never reaches the screen.

<!-- Bad: interpret model output directly as HTML -->
<div id="answer"></div>
<script>answer.innerHTML = modelOutput;</script>

<!-- Better: render as text; use only sanitized results for formatting -->
<script>answer.textContent = modelOutput;</script>

Model-chosen addresses and code: SSRF and code execution

When a server sends requests or executes code using model-generated values, output-handling risks shift to the server side. If the server fetches a model-chosen URL unchanged, injection may redirect it to an internal administration page or cloud metadata address. This is Server-Side Request Forgery (SSRF). For clients fetching OAuth-related URLs, MCP security best practices recommend requiring HTTPS in production, blocking private, loopback and link-local address ranges, checking redirect destinations by the same criteria, and using an egress proxy that blocks internal destinations. They also warn against implementing IP validation yourself because encoding tricks are easy to miss. The same principles apply to every tool in which the model chooses URLs.

Running model-generated code or commands without adequate validation and isolation leads to ASI05 (Unexpected Code Execution) in the OWASP Agentic Top 10. The default principle is "do not execute if execution can be avoided." If necessary, use a dedicated least-privilege container or virtual machine with restricted network and file access and no secrets. Anthropic's computer use documentation likewise lists a dedicated least-privilege VM or container, domain allowlists and isolation of sensitive data such as login credentials as preventive measures.

Going deeper

Binding approval to execution

Even with human confirmation, approval loses meaning if "what the user saw" differs from "what actually executes." The screen may show "change meeting schedule" while the real arguments add ten attendees. Or a long-running task may continue under approval granted at its start even after the user's permissions have been revoked. This is a Time-of-Check to Time-of-Use (TOCTOU) problem.

Approval bound to execution has the following properties. The request contains the actual target, arguments and expected effects. Approval is stored with a hash of those arguments; immediately before execution, arguments are compared and any change requires fresh approval. Approval expires and is single-use. Current user permissions are rechecked immediately before execution. Action previews and rollback, recommended by the OWASP AI Agent Security Cheat Sheet, go alongside this design.

Isolating derived data: caches, memory, summaries and exports

Data can leak across access boundaries through derived paths even when source-store permissions are perfect. If a response cache key omits user or organization identifiers, another organization asking a similar question receives the first organization's answer. If several teams reuse a shared summary, confidential content within it reaches unauthorized teams. Conversation exports and shared links often bypass the original conversation's permission checks.

Ask one question: "Does this derived artifact inherit the original's permission boundaries?" Answer it for cache keys, memory stores, summary stores, export files and embedding indexes. Wherever the answer is "no," add user or organization scope, or prevent reuse. The OWASP AI Agent Security Cheat Sheet separately covers memory isolation between users and sessions.

At work

Onsaemiro Hospital: removing medical records from a booking agent

Onsaemiro Hospital, a general hospital, plans to introduce an appointment agent. The draft design uses a service account with read/write access to the entire patient database, places patients' full recent medical records in context to improve answers, and executes appointment changes immediately. Agent-written guidance is rendered as HTML in the app, and the system prompt contains the internal booking API URL and key.

The security reviewer diagnoses the three roots. For functionality, the reviewer asks whether booking needs medical-record retrieval; only available appointment times and the patient's own bookings are needed. The medical-record tool is removed. For permissions, write access is limited to the appointment table, and calls use the logged-in patient's own token. The booking server directly checks on every request whether this patient can change this appointment. For autonomy, cancellation and changes to a different department execute only after the patient confirms the date, department and clinician on screen.

The reviewer also checks output and secrets. Guidance is rendered as text with a sanitizer allowing only bold and lists; links and images are limited to the hospital domain. The API URL and key move from the system prompt to server configuration and a secret store. The exposed key is revoked and replaced.

The objection that "medical records improve doctor recommendations" is answered with a tool returning only two fields: department and attending doctor. Even if tricked, the agent can now only show the patient's own bookings and change them with confirmation. The register records "patient approves without reading the confirmation screen" as residual risk, leaving a feature to reverse cancellation within 24 hours for next quarter.

Whale Market: one sentence is risky in three different ways

The secondhand marketplace Whale Market builds a feature that writes listing titles and descriptions from sellers' photos and short notes. Model-written descriptions go to three destinations: Markdown rendering on public listing pages, the operations team's review screen, and a weekly CSV report opened in a spreadsheet.

Two issues emerge during beta. A model copies an HTML fragment from a seller's note into a description, breaking formatting and displaying script tags in the review screen. Fortunately, that screen renders text, so nothing executes. Some descriptions also contain external payment-site links induced by seller notes. Operations proposes telling the model not to write HTML or links.

The security lead accepts that only as a supporting measure and applies separate controls at each destination. Public pages use a Markdown renderer with raw HTML disabled and an allowlist sanitizer; only platform-domain links remain, and only platform-uploaded photos display. The review screen renders text and shows the original note beside the model's description. CSV export prefixes cells starting with formula characters with an apostrophe or tab so they are interpreted as text. Following OWASP's warning that spreadsheet behavior varies and there is no universal sanitization method, the team tests results in the program operations actually uses.

The lesson is that the same output has different risks and controls at each destination. Relying on a single model instruction would expose all three when it was ignored.

Common mistakes

1. "Prohibitions in the prompt provide permission controls." The model is still responsible for following them. Downstream systems must decide authorization on every request.
2. "Our own model produced it, so we can trust it." Output reflects external content and user input read by the model. Treat it like user input before rendering or execution.
3. "Users cannot see the system prompt, so it can hold keys." Invisibility is no protection. Keep keys beyond the model's direct reach.
4. "More data means better accuracy." Every addition increases exposure. Build tools that return only necessary fields.
5. "Embeddings are numbers, so they are safe." Embedding inversion can reconstruct significant portions of the original. Apply the same access controls as for source text.
6. "Logging and rate limits solve excessive agency." OWASP classifies both as impact-reduction measures. Prevention reduces functionality, permissions and autonomy.
7. "Schema-valid arguments are safe." Format can be correct while the target or amount is wrong. Servers must recheck business rules and object-level access permissions on every request (OWASP Authorization Cheat Sheet).
8. "CSP lets us skip output encoding." CSP is an additional layer. Context-specific encoding and sanitization come first.

Summary at a glance

Check the official docs

• OWASP LLM02:2025: sensitive information disclosure, sanitization and redaction.
• OWASP LLM06:2025: three roots of excessive agency and preventive measures.
• OWASP LLM05:2025: risks and controls by output destination.
• OWASP LLM07:2025: what to keep out of system prompts.
• OWASP LLM08:2025: permission-aware retrieval and partitioning.
• OWASP XSS Prevention Cheat Sheet: encoding, sanitization and the role of CSP.
• MCP Security Best Practices: delegation tokens, SSRF and scope minimization.
• OpenAI — Data controls: training use and retention terms.

Self-check

3-1. Name three lifecycle points other than the final answer where sensitive information remains or leaks, and give one basic control for each.

3-2. Which two measures does OWASP LLM06 classify as "reducing impact" rather than preventive measures, and why do they not count as prevention?

3-3. Define complete mediation in one sentence and give an example of a design that violates it.

3-4. List four items to check per feature when reviewing an external LLM API provider's data terms.

3-5. In one sentence each, state the default way to display model output on the web, the approach when formatted HTML is essential, and how CSP relates to both.

3-6. What do MCP security best practices prohibit regarding state handles, and what must the server do instead?

3-7. Design task: an internal assistant reads employees' calendars and books meeting rooms. Design no more than three tools, listing each tool's functionality, permissions and confirmation requirement in one line.

Hands-on tasks