| Stateless |
An approach where the server does not remember earlier requests, so each request must carry all needed information |
| Response contract |
A document expressing input assumptions, required output, prohibitions, hold conditions and acceptance criteria as verifiable conditions |
| System prompt |
Roles and rules common to requests. Sent as top-level system in Anthropic and system_instruction in Gemini |
| instructions |
Per-request higher-priority instructions in the OpenAI Responses API; not carried over from earlier turns |
| Prefill |
Filling the beginning of an assistant message to determine how an answer starts; unsupported by Claude 4.6 and later models |
| Trust boundary |
The line separating our instructions from external material to process |
| Indirect prompt injection |
An attack that tries to change model behavior through instructions hidden in external material such as web pages, emails or files |
| Token |
The unit in which a model processes text; the basis for billing, rate limits and context windows |
| Context window |
The total capacity shared by input, output and reasoning tokens in one request |
| Context rot |
Declining accuracy and recall as the number of context tokens increases |
| Maximum output |
max_tokens/max_output_tokens; a ceiling on generated tokens, not a target length |
| Reasoning (thinking) tokens |
Tokens used for internal reasoning before answering; billed as output and counted toward the output ceiling |
| Stop reason |
Why a response stopped: Anthropic stop_reason, OpenAI status/incomplete_details |
| Refusal |
A safety classifier or model declines to answer; arrives as HTTP 200 in Anthropic |
| temperature |
A sampling parameter that sharpens or flattens the probability distribution of next-token candidates |
| Reasoning effort |
A control for token use and reasoning amount: Anthropic output_config.effort, OpenAI reasoning.effort |
| Compaction |
Reducing a long conversation's context to the state needed for the next turn |
| Structured output |
Constraining output to follow a specified JSON schema |
| JSON mode |
An output mode guaranteeing valid JSON, but not schema compliance |
| Strict mode |
An option constraining responses or tool arguments to a supported schema. Handle refusals, output interruptions and provider exceptions (such as enum-value case) separately |
| Semantic validation |
Checking that correctly formatted values satisfy business rules involving totals, date ordering, state and similar relationships |
| Tool calling (function calling) |
The model proposes a tool name and arguments; the application executes it |
| tool_choice |
An option determining whether and which tools the model uses |
| Parallel tool calls |
Requesting several tools together in one response |
| Reasoning items/thinking blocks |
Information reasoning models return with tool calls; must be returned unchanged in the next request |
| Excessive agency |
Harmful actions caused by more functionality, permissions or autonomy than necessary (OWASP LLM06) |
| TTFT |
Time from a request to arrival of its first output fragment |
| p95 |
The boundary at which the slowest 5% begin when measurements are sorted |
| Prompt caching |
Reusing processed request prefixes to reduce input cost and latency; output is generated anew each time |
| Response cache |
A cache returning earlier answers unchanged; semantic caching also matches similarly meaningful questions |
| Batch API |
An API processing large volumes of nonurgent requests asynchronously at a discount |
| Service tier |
A choice to process the same model's requests more cheaply and slowly, or more expensively and quickly (service_tier, for example) |
| Escalation |
Reprocessing with a larger model when a small model's result fails validation |
| Cost per business task |
Total call, retry and review costs divided by tasks passing quality criteria |
| Unknown outcome |
No response was received, so it is unknown whether the server executed the operation |
| Rate limit |
A cap on requests/tokens within a time period (RPM, ITPM, OTPM, TPM and others) |
| Token bucket |
A limiting method whose capacity refills continuously rather than resetting at a fixed time |
| Spend limit |
A spending ceiling for a set period, such as a month; retries do not clear it |
| Exponential backoff |
Increasing the waiting time exponentially with each retry |
| Idempotency |
Multiple identical requests have the same result as one |
| Idempotency key |
A unique value per operation ensuring it is processed only once; reuse the same key for retries of one operation |
| Dead letter queue |
A queue isolating repeatedly failing inputs so they do not block other work |
| Circuit breaker |
Temporarily stops calls when failure rates are high and sends requests to a fallback |
| Bulkhead |
Resource isolation by feature to prevent failures from spreading |
| Success criteria |
Specific, measurable, achievable, relevant pass conditions |
| Evaluation set |
Fixed inputs, expected results and grading methods for comparing quality before and after changes by the same standard |
| Rubric |
A table of judgment conditions followed by an LLM or human grader |
| Model snapshot |
A specific model version with fixed behavior |
| Configuration bundle |
Model ID, prompt/schema/tool versions, parameters and SDK version managed together |
| Request ID |
A unique identifier attached by a provider to each response; the Claude API uses the request-id header |
| Pseudonymization |
Deleting or replacing personal information so a specific person cannot be identified without additional information (Personal Information Protection Act, Article 2) |
| ZDR |
Zero data retention agreement; coverage varies by provider and feature |