Chapter 5. Operational security and incident response

Learning objectives

• Design audit logs that can answer incident investigation questions, and protect logs themselves as sensitive data.
• Detect cost/resource exhaustion attacks and runaway agents, and constrain them through budgets and blocking mechanisms.
• Reduce supply chain risks in models, model files, plugins and MCP servers through adoption and change procedures.
• Establish LLM incident response procedures with owners, containment measures, evidence and recovery criteria for each stage.
• Outline legal notification and reporting requirements to check when personal-information leakage is suspected.

Introduction

One Friday night, a company's customer support agent began sending ten times its usual email volume. The on-call operator received an alert and opened the logs, but they contained only line after line of "tool call succeeded." Nothing recorded which session read which input or sent what to whom. With no switch for disabling only the sending tool, the operator took down the whole service, and customer inquiries piled up until Monday morning. Investigation eventually established the cause, but never conclusively determined how many customers were affected or where their information had been transmitted. The operator had done nothing wrong. What can be done on the day of an incident depends on the logs and switches built beforehand. Operational incidents can also happen without attackers. One temporary setting or overnight update can become a privacy or cost incident.

Operational security is only as effective as the preparations made before an incident. This chapter starts with audit logs that can answer investigation questions and their protection, then covers controls for cost/resource exhaustion and runaway agents, supply chain management for models and plugins, and detection signal design. It concludes with LLM incident response, containment switches and notification/reporting requirements when personal-information leakage is suspected.

Key concepts

What must audit logs prove?

The first incident question is "when, in whose session, did which input arrive, and what did the model do?" The OWASP Logging Cheat Sheet recommends recording when, where, who and what for every event, along with an interaction identifier connecting related events. In LLM applications, one request branches into retrieval, multiple model calls, tool calls and approvals. Without trace identifiers connecting them, investigations break down.

Logs should prove "what the system allowed and executed" rather than "how the model thought." MCP security best practices likewise warn that investigations are blocked when downstream logs identify only an intermediary server instead of the actual requester.

Protecting the logs themselves

Logs are sensitive data too. The OWASP Logging Cheat Sheet recommends removing, redacting, hashing or encrypting session identifiers, access tokens, passwords, encryption keys and other critical secrets, database connection strings, payment information and sensitive personal information. It also requires log protection: tamper detection, copying to read-only media as soon as possible, recording and monitoring access, and restricting and regularly reviewing viewing rights. Sanitize newlines and similar characters in data arriving from outside trust boundaries to prevent log injection. Model output and external documents are typical examples.

Korean standards take the same direction. The Standards for Measures to Ensure the Security of Personal Information, a notice by the Personal Information Protection Commission, require retaining and managing access records for people accessing personal-information processing systems (excluding data subjects) for at least one year. Systems processing information on 50,000 or more data subjects, unique identifiers or sensitive information, among others, require at least two years. Records must be stored securely against tampering, theft and loss. If agents access a personal-information processing system, establish with the privacy officer how their access records correspond to the records required by this standard.

Whether to retain full conversation text is a separate decision. A common compromise keeps basic logs as redacted structured fields and stores original content essential for investigation briefly in a separate, tightly restricted store.

User identification and abuse tracking

Preventing abuse requires knowing who repeats it. OpenAI recommends attaching a safety_identifier parameter to requests to uniquely identify users, hashing usernames or email addresses so identifying information is not transmitted. A session identifier can stand in for users who are not logged in. The same document says knowing your customer (KYC), for example through signup and login requirements, reduces risk.

Anthropic recommends notifying users who repeatedly trigger the same kind of refusal that they are violating policy, and responding with rate limits or bans. This requires logging guardrail judgments with hashed or internally pseudonymous user identifiers and managing the mapping to real identities separately.

Forms of unbounded consumption

OWASP LLM10 (Unbounded Consumption) covers risks when users can trigger excessive, uncontrolled inference. It identifies seven sub-risks.

Agents can do the same without an attacker. Retries after tool errors, subtask branching and loops of agents delegating to each other grow costs exponentially. This is propagation described by OWASP Agentic Top 10 ASI08 (Cascading Failures), expressed as cost. Denial of Wallet differs from ordinary denial of service because damage accumulates while the service appears healthy.

Budgets and blocking mechanisms

OWASP LLM10 lists fifteen mitigations, including input validation, limiting logit/log-probability exposure, rate limits, resource allocation management, timeouts and throttling, logging/monitoring/anomaly detection, watermarking, graceful degradation, access control and a centralized model inventory. OpenAI also advises limiting user input length and output tokens to reduce abuse.

Application operators should first establish the following mechanisms.

Distinguish alerts from blocking. A cost alert only informs people, and on Friday night no one may see it. A budget cap stops execution in code. To counter model extraction, OWASP recommends limiting or obfuscating exposure of detailed probability information such as logit_bias and logprobs in API responses, and monitoring per-account usage patterns. The OWASP AI Agent Security Cheat Sheet also recommends tracking tokens and costs per session and using circuit breakers to prevent cascading failures.

Supply chain 1: models and model files

OWASP LLM03 (Supply Chain) lists outdated third-party packages, licenses, unmaintained models, backdoored pre-trained models, weak provenance, malicious LoRA adapters, exploitation of collaborative development, tampering with on-device models and unclear terms of service. Models are opaque binary black boxes, limiting what static inspection can guarantee.

Model file formats also create risk. Hugging Face warns that loading pickle files, the default format for PyTorch weights, can execute arbitrary code. It recommends loading models only from trusted users and organizations and checking provenance through signed commits, while noting that signatures establish origin rather than file safety. Hub security scans are not complete either. Its documentation introduces safetensors, designed to store tensors safely unlike pickle.

Build the adoption procedure in this order.

1. Review the provider, data provenance, terms and privacy policy, and revisit them regularly.
2. Use models from verifiable sources and check integrity with signatures and file hashes.
3. Open files that can execute code on loading only in isolated environments.
4. Validate using red teaming and evaluations for your own use, rather than benchmarks. Models can be fine-tuned to evade public benchmarks (OWASP).
5. Establish a patching policy and rely on actively maintained APIs and models.

Supply chain 2: tools, plugins and MCP servers

Agents load tools and other agents at runtime. Unlike LLM03's focus on static dependencies, ASI04 (Agentic Supply Chain Vulnerabilities) in the OWASP Agentic Top 10 covers compromise of runtime-composed tools, MCP servers, agent registries and update channels. A tool reviewed when installed can become a different tool after an update.

MCP security best practices explain local-server risks in detail. Local MCP servers run on the user's computer with the client's privileges, so a malicious startup command or server can cause arbitrary code execution and data leakage. Clients offering one-click local-server setup must show the full, untruncated command, explain that this is a risky operation and obtain explicit user approval. The best practices recommend running server commands in a least-privilege sandbox with restricted file and network access.

There are four operational principles: pin tool and server versions; compare names, descriptions, parameters and requested scopes on updates; approve expanded privileges as if adopting a new component; and prepare revocation procedures that disconnect and revoke tokens together on suspected compromise.

Configuration inventories, AI BOM and the model lifecycle

To answer immediately "where are we using the affected component, and at which version?" during an incident, you need an inventory. OWASP LLM03 recommends keeping a signed, current software bill of materials (SBOM) and considering an AI BOM (ML SBOM) covering models and data, with OWASP CycloneDX as a starting point. CycloneDX ML-BOM represents AI/machine-learning datasets, models and configurations and records provenance. LLM applications should also inventory prompts, tool definitions, connectors and permission scopes, classifiers and retrieval-index sources.

Models have lifecycles too. Anthropic's documentation says each model ID points to a fixed version whose weights do not change (older-generation aliases that follow the latest snapshot are an exception), but serving infrastructure such as request routers or safety classifiers can change and slightly alter observed behavior. Models move through deprecated to retired; requests to retired models fail. Anthropic provides at least 60 days' notice before retiring public models and recommends testing applications against replacements well in advance. From a security perspective, model replacement triggers the Chapter 1 review. Rerun the Chapter 4 regression set on a new model before switching.

Designing detection signals

Detection focused on actual privilege abuse and signs of damage lasts longer than detection of suspicious wording.

Alerts without owners simply accumulate, so assign an owner and first action to each. Baselines for goal states and tool-use patterns help detect unexpected goal changes or tool sequences as deviations (OWASP Agentic Top 10).

Incident response procedure

LLM incidents follow the same structure as general security incidents, adding tool permissions, memory and indexes, and prompt/model versions. In April 2025, NIST released SP 800-61 Rev. 3, aligning incident response recommendations with Cybersecurity Framework (CSF) 2.0 and risk management activities. NIST AI RMF's Manage function also includes incident response, recovery and communication plans and calls for notifying relevant parties of incidents and errors.

Containment precedes full scoping. If leakage may be ongoing, starting with log analysis allows damage to grow. Conversely, hurriedly resuming without establishing the cause reopens the same path. OpenAI recommends immediately revoking and replacing keys that may have been exposed or misused. The OWASP Secrets Management Cheat Sheet warns that restarting an application does not revoke stolen credentials. It also recommends centralized secret management and automated rotation to speed revocation and replacement.

Containment switches and safely reduced operation

A system with only a "turn off the whole service" switch pays heavily even for small incidents. Prepare switches at several levels.

1. Tool level: disable only a specific tool (message dispatch, payments, external fetching).
2. Connection level: disconnect a specific MCP server or connector and revoke its token.
3. Feature level: disable memory writes, external-link rendering or file uploads.
4. Mode level: switch to read-only mode or prepared static guidance.
5. Task level: cancel queued tasks and stop ongoing work.

For each switch, record who can operate it, what stops and what happens to ongoing tasks in the runbook, and test it regularly. Also test whether the user-facing "stop" button actually stops server-side work. Quarterly tabletop exercises, walking through the runbook verbally without affecting real systems, reveal gaps such as "there is no switch for one tool" or "we do not know how to contact the provider."

Personal-information breach notification and records

An LLM incident involving personal-information leakage creates legal notification and reporting requirements. This section outlines the decision framework and is not legal advice. In actual incidents, the privacy officer and legal team make the judgment.

Article 34 of Korea's Personal Information Protection Act requires controllers learning of loss, theft, leakage, forgery, alteration or damage to personal information to notify data subjects without delay. Notifications include affected information, timing and circumstances, ways data subjects can reduce harm, the controller's response and remedies, responsible department and contact details, and legal rights such as compensation claims and dispute mediation and how to exercise them. Amendments effective September 11, 2026 also introduce notification when a possibility of such an event becomes known. The Enforcement Decree sets a 72-hour deadline for notification in writing or equivalent form (or immediately after urgent-response reasons cease). If specifics are not yet established, notify confirmed facts first and supplement later. Article 40 of the Enforcement Decree requires reporting to the Personal Information Protection Commission or Korea Internet & Security Agency within 72 hours of awareness if any of three conditions applies: information of at least 1,000 data subjects, sensitive information or unique identifiers, or leakage through unlawful external access. There are deadline exceptions for unavoidable reasons and a proviso allowing no report where recovery or deletion has substantially reduced the likelihood of harm to rights and interests. Health information is sensitive information under Article 23 of the Act.

There are two practical implications. Answering "which fields, how many people, through which path?" within 72 hours requires logs prepared for scoping. Notification may also need consideration before leakage is confirmed, so bring the privacy officer into response early. NIST AI 600-1 likewise explains that logs, change-management records and version histories support incident information sharing.

Going deeper

Balancing evidence preservation and privacy

Investigation needs originals, while privacy calls for minimizing them, so use separate stores. Retain basic audit logs as redacted structured fields for longer periods; keep originals briefly in a separate evidence store requiring approval for access. When an incident is declared, freeze originals from the relevant period there and record collection, access and handover. Analyze copies rather than originals and attach integrity-check values to copies too.

Different system clocks prevent timeline reconstruction, so align timestamps to one standard and connect events with trace IDs. Define investigation-channel rules to avoid secondary exposure from pasting originals into messaging apps or tickets.

Recovering poisoned memory, indexes and caches

Agent contamination can persist after the cause is removed. Poisoned documents remain in indexes, "preferences" originating in hidden instructions remain in long-term memory, and contaminated answers remain in response caches. Think through recovery in this order. Stop writes from the contamination source first. Then identify affected memory entries, index chunks and cache entries by the period and path of contamination. Stored provenance makes this easier. Exclude affected items from retrieval to quarantine them, then delete them; rebuild the necessary index scope from trusted originals. Rotate credentials issued or exposed during the contamination period. Finally, run regression scenarios reproducing the same contamination before reopening writes.

A store without an answer to Chapter 1's question, "how do we clear it if poisoned?," may have to be discarded entirely at this stage. Recording provenance and creation time in memory and indexes greatly reduces recovery costs.

At work

Hanmaeum Dental Network: debug logs were the leakage path

During a routine check of Hanmaeum Dental Network's booking and consultation chatbot, a reviewer finds that debug logging enabled three months earlier for a quality investigation is still on. Entire original conversations are being sent to an external log analytics service. They contain patient names, contact details, pain locations and treatment histories. Almost twenty accounts, including outsourced developers, can access the logging project.

The reviewer contains the issue first: disable debug logging, stop external transfers and reduce project access to the minimum necessary personnel. Next, the reviewer identifies the transmission period, counts conversations and unique patients, and uses the log service's access records to establish who viewed logs during that period. There is no evidence of unlawful external access, but personnel without a business need have access to sensitive treatment information.

The privacy officer and legal team join the same day. Treatment histories may be sensitive information, and under the Enforcement Decree, leakage of sensitive information can be reportable regardless of the number of affected individuals. Legal reviews whether notification and reporting are needed and reaches a conclusion within 72 hours, using the reviewer's period, fields, counts and access records.

To remove the cause, basic logs now retain only redacted structured fields (trace ID, tool name, judgment, token count). Original text requires approval and stays in a restricted internal store for no more than two weeks. Debug settings get expiration dates, and outbound data is reviewed quarterly. The review records: "logs are sensitive data too, and temporary settings must expire."

Wave Media: Denial of Wallet without an attacker

The content company Wave Media uses a research agent when drafting articles. It gathers sources through an external-search connector and delegates summaries to subagents. Early one Tuesday, an alert reports that 40% of the monthly budget has been spent in six hours. The alert goes only by email; no one sees it before staff arrive in the morning.

The on-call operator finds no attack in the logs. An overnight connector update changed the response format. Unable to parse it, the agent classifies it as a search failure and retries. Each retry starts new subagents to summarize again; those agents call the same connector, fail and retry too. One task expands into hundreds of model calls.

Containment has two steps: disable the connector and cancel all ongoing research tasks. Checking whether cancellation actually stops subagents takes time; some continue after their parent is canceled. The team records this as a separate defect.

Prevention focuses on shifting from passive alerts to automated enforcement. Each task gets caps on model calls, subagent count, retries and cost. Retry intervals increase progressively; a circuit breaker stops calls after five consecutive identical tool errors. Canceling a parent now stops its children. Connector versions are pinned, and updates require response-format comparisons and regression tests. At the organization's daily budget limit, research degrades to read-only summarization. The review records: "unbounded consumption happens without malice" and "when no one is watching, caps work; alerts do not."

Common mistakes

1. "A cheaper model solves cost attacks." Lower unit prices do not stop unlimited requests from growing costs. The cause is uncapped consumption.
2. "Cost alerts prevent Denial of Wallet." Alerts only inform people. When no one is monitoring, caps enforced in code must stop execution.
3. "A popular model is verified." Downloads and benchmarks do not prove integrity. Check provenance, signatures, hashes and evaluations for your use.
4. "Pinning the model ID means behavior never changes." Serving infrastructure can slightly change behavior despite fixed weights, and models eventually retire. Run regression evaluations periodically.
5. "More logging is always better." Logs are sensitive data too. Decide what to record, redact and retain, for how long and who may view it.
6. "Contain only after fully identifying the cause." Ongoing damage requires containment first. Establish the cause before resuming.
7. "Restarting the service rotates keys." Restarting does not revoke stolen credentials. Revocation and replacement are separate actions.

Summary at a glance

The day before the exam, read each sentence aloud to check your understanding.

• Audit logs connect when, where, who and what with input sources, tool calls, approvals, judgments and versions through trace IDs.
• Remove or redact tokens, passwords, keys, connection strings and sensitive personal information; control tampering and access.
• Track abuse through hashed user identifiers and guardrail judgment records.
• Unbounded consumption may have no attacker. Retry, branching and delegation loops grow costs.
• Alerts inform; caps stop. Establish per-task budgets and circuit breakers.
• Verify model-file provenance, signatures and hashes; isolate formats that can contain executable code.
• Pin tool and MCP server versions, reviewing description, parameter and scope changes on updates.
• Incident response has six stages. Stop ongoing damage before fully scoping it; resume only after fixing causes and passing regression tests.
• Prepare and actually test switches per tool, connection, feature, mode and task.
• On suspected personal-information leakage, involve the privacy officer early for notification and reporting decisions under the 72-hour standard.

Check the official docs

• OWASP LLM10:2025 Unbounded Consumption: sub-risks and mitigation strategies.
• OWASP LLM03:2025 Supply Chain: supply chain risks and AI BOM.
• OWASP Logging Cheat Sheet: what to record, omit and protect.
• OpenAI — Safety best practices: safety_identifier, key revocation and reporting channels.
• Hugging Face — Pickle Scanning: code execution risks in pickle files.
• Anthropic — Model deprecations: model lifecycle and retirement notices.
• NIST SP 800-61 Rev. 3: incident response recommendations aligned with CSF 2.0.
• Personal Information Protection Act: Article 34 notification and reporting of loss, theft, leakage and related incidents (with Enforcement Decree Articles 39 and 40).

Self-check

5-1. Name four types of data to remove or redact under the OWASP Logging Cheat Sheet, and one alternative item that can be recorded for investigation.

5-2. Explain the difference between a cost alert and a budget cap, and give two caps for a single agent task.

5-3. Distinguish "continuous input overflow" and "resource-intensive queries" under OWASP LLM10, and give a suitable control for each.

5-4. Based on Anthropic's documentation, explain why behavior for the same prompt can change even with a pinned model ID, and give one operational action.

5-5. State two things a client offering one-click local MCP server setup must do under MCP security best practices, and the recommended execution environment.

5-6. List the three reportable conditions and deadline in Article 40 of the Enforcement Decree of the Personal Information Protection Act.

5-7. Design task: design four containment switches for a support agent with long-term memory. For each, state what it disables, who may operate it and what stops.

Hands-on tasks