Chapter 1. How generative AI works and its limits

Learning objectives

• Explain tokens, the context window, probabilistic generation and the knowledge cutoff with everyday analogies.
• Distinguish models, products and tools, and identify whether an answer was generated or retrieved.
• Explain why hallucinations happen and identify situations where output must not be taken at face value.
• Choose between reasoning and standard models, and select model tiers based on task complexity, speed and cost requirements.
• Explain the uses and error types of multimodal input such as images, PDFs and audio.

Introduction

Min, a librarian at a district library, used AI to create a recommended reading list for a career exploration reading week for young people. It neatly presented twelve books with their authors, publishers, publication years and one-line descriptions. But a colleague checking the list against the shelves was puzzled. Two books could not be found in any bookshop; another had the right author but the wrong title. The remaining nine were correct. Had the list gone straight onto the noticeboard, students would have searched for books that did not exist.

What surprised Min was that the incorrect books were presented in exactly the same tone and format as the correct ones. The AI did not say, "I am unsure about these two." How had it created this list? Why were nine books correct and three wrong? What signs might have revealed the errors in advance?

This chapter answers those questions. It first explains the units AI uses to process text (tokens), how much it can see at once (the context window), how it produces answers (probabilistic generation) and the time boundary of its knowledge (the knowledge cutoff). It then covers the causes and types of hallucination, the differences between reasoning and standard models, and the limits of image and document input. Finally, a checklist brings together situations where output must not be taken at face value. This chapter builds the foundation for Discernment in AI fluency.

Key concepts

Distinguishing models, products and tools

Three things need to be distinguished first when discussing AI.

• Model: The computational engine that reads text and generates subsequent text. In a car analogy, it is the engine.
• Product: The service built so people can use a model. This includes the chat interface, file uploads, conversation history and sharing features. ChatGPT, Claude and Gemini usually refer to products, within which multiple models operate. In the car analogy, it is the body and dashboard.
• Tool: A separate function the model can call when producing an answer. Examples include web search, code execution for calculations, internal document search and connections to external apps. In the car analogy, it is the navigation system.

The response to an error depends on where it arose. Even with the same model, products with search enabled and disabled can give entirely different answers about current information. This chapter starts with the habit of going beyond "the AI was wrong" to ask, "Did the model make a mistake, was a tool disabled, or did the product supply outdated material?"

Tokens: the units into which a model splits text

A model does not read text word by word as people do. It cuts text into pieces called tokens and processes those. OpenAI explains that a token may be a single character, part of a word, a whole word or punctuation, and that spaces also affect how text is divided. Think of LEGO: the number of bricks needed for the same house depends on which set you use.

Token counts do not correspond one to one with character or word counts. The same text can have different token counts depending on the model, its encoding (the rules a tokenizer uses to turn text into tokens) and the language. Companies' rules of thumb are based on English.

It is natural for a Korean document and its English translation to have different token counts. Applying an English estimate directly to Korean material can produce a large error. When an exact count is needed, use the token-counting feature provided by the product or developer tools.

Tokens fall into three broad categories.

Tokens are units for measuring usage limits and costs. The amount a model can handle at once is set in tokens, and basic language-model API charges are usually calculated in tokens. OpenAI adds that other features may use other billing units. It also explains that reasoning tokens are billed as output tokens even though they do not appear in the answer, so a short answer can still use many tokens.

Context window: the workspace on your desk

The context window is the total amount of text a model can refer to at once while producing an answer. Anthropic describes it as working memory, distinct from the vast material the model learned during training. Instructions, conversation history, attached images and documents, tool results and the answer currently being generated all occupy this space. If training data is the library built up over a long time, the context window is the desk where the books are open right now.

A bigger desk does not mean better work. Anthropic calls the decline in accuracy and recall as token counts increase context rot, emphasizing that selecting what to include matters as much as the size of the space. It also notes that chat interfaces may manage context on a first-in, first-out basis, removing older content as conversations grow. This is why conditions established early in a long conversation can fade. Being able to include material is different from using every part of it. Omissions are especially common when conditions are scattered across documents, obsolete versions are mixed in, or a footnote contains an exception that changes the conclusion.

Another concept to know is the output limit. A model also has a maximum number of tokens it can generate in a single response. OpenAI explains that this limit is separate from the context window, and that reasoning models must leave room for reasoning tokens as well as the visible answer. A long document may be accepted as input without an equally long result being returned in one response. Think of input as desk space and output as the number of pages you can write in one sitting. Anthropic states that API requests return an error if the input alone exceeds the window.

Probabilistic generation: a very powerful autocomplete

Generative AI looks at the text so far and uses probabilities to choose a plausible next token, continuing the text. Think of a much larger, more sophisticated version of autocomplete on a phone keyboard. Two properties follow from this.

1. Answers to the same question can vary slightly. If several candidates are plausible, a different one may be selected each time.
2. The generation process includes no built-in step to verify whether something is true. The model continues based on how natural the text is, without separately checking its truth.

Developer APIs may provide a setting called temperature to control randomness. Google explains that lower temperatures suit tasks needing a fixed answer, while higher temperatures suit tasks needing variety. However, Anthropic's API documentation states that even a temperature of 0 does not make results fully identical, and some recent models do not accept or ignore temperature settings. Above all, lowering temperature does not ensure accuracy. It may produce the same wrong answer repeatedly. Google also advises retaining the default for some models because changing it can reduce reasoning performance.

An answer therefore does not become fact simply because it is repeated. Conversely, contradictory answers to repeated questions are a reason for suspicion. Anthropic describes running the same instructions several times and comparing results, explaining that disagreement can indicate hallucination. Remember the asymmetry: consistency is not proof of truth, but inconsistency can be a warning light.

Knowledge cutoff: the last newspaper read before leaving the country

A model is trained on material up to a particular point in time. It knows little about later events, or knows them inaccurately, so they require outside evidence such as search or attachments. It is like someone returning from a long assignment at an Antarctic station whose knowledge stops at the last newspaper read before departure.

Anthropic's model documentation distinguishes two dates. The reliable knowledge cutoff is the point up to which the model's knowledge is broadest and most reliable; the training data cutoff describes the wider range of material used in training. The model may know something about events between these dates, but its knowledge may be shallow or inaccurate. Even events immediately before the cutoff may have limited training coverage because not enough has yet been written about them.

At work, judging whether the answer to this kind of question changes over time matters more than memorizing dates.

Information in the right-hand column may already have changed even if it predates the cutoff. Attach current material or enable search for these questions, and have the answer state its reference date.

Generation and retrieval: creating answers and looking them up

AI products produce answers in two ways. One is to generate an answer solely from the model's learned knowledge. The other is to retrieve material through search or document lookup, then base the answer on it. The latter is called grounding: anchoring an answer to actual material, like driving a stake into the ground.

Google explains that grounding Gemini with Google Search enables answers about recent information beyond the model's knowledge cutoff and returns citations connecting answers to sources for users to check. Features that answer by searching internal documents work on the same principle.

Grounding is powerful, but it does not solve everything. Even with search, these errors remain possible.

• The search results themselves are outdated or wrong.
• The model misinterprets results or mixes information from different sources.
• A citation is present, but the cited passage does not actually support the claim.

Looking at source references is therefore the start of verification, not its end. Chapter 4 covers checking sources against individual claims in detail.

Causes and types of hallucination

A hallucination occurs when a model plausibly produces content that is untrue or contradicts the material provided. Anthropic acknowledges that even the most advanced models can produce text that is factually wrong or inconsistent with the context. Distinguishing error types rather than simply calling them "wrong" makes the appropriate response clearer.

Anthropic recommends three basic measures: allow the model to say it does not know when uncertain; for long documents, have it quote the relevant sentences verbatim before answering; and require citations and sources for every claim, withdrawing claims without supporting sentences. You can also restrict scope by saying, "Use only information from the provided documents, not general knowledge." Anthropic adds that these techniques substantially reduce hallucinations without eliminating them, so important information and high-risk decisions always require verification.

Confidence in tone and strength of evidence

AI answers can be fluent and definite even when wrong, and people tend to trust confident speakers. The three incorrect books in the introduction used exactly the same tone as the nine correct ones. Confidence in tone and strength of evidence are separate things.

A related phenomenon is sycophancy. Anthropic's research defines it as behavior in which a model favors answers matching the user's beliefs over truthful answers. It attributes part of the cause to human preference ratings that favor plausibly written, agreeable answers over accurate ones. At work, it appears in these ways.

• When a user says, "Really? I thought otherwise," the model changes its answer without evidence.
• When a question contains the desired conclusion, the answer tends to support it.
• When asked to evaluate the user's draft, the model rates it more generously than warranted.

The response is simple. When an answer changes, judge which version is correct using external evidence, not the model's expressed confidence. Phrase questions neutrally, without revealing the desired conclusion. When seeking evaluation, explicitly make room for criticism: "Identify three weaknesses first."

Reasoning models and standard models

Most AI products now offer several kinds of model. A reasoning model internally works through multiple steps before answering, while a standard model starts generating its response directly after receiving instructions. OpenAI compares them to a planner and a workhorse, explaining that they are different rather than one being better, and that most workflows combine both.

Avoid two common misconceptions. First, reasoning models do not necessarily have more current knowledge. Reasoning ability is how well a model works with given information; freshness depends on the cutoff and use of search. Second, an answer can be wrong despite prolonged thought. Check the evidence and calculations behind reasoning models' conclusions too. Instructions for reasoning models differ somewhat from those for standard models; Chapter 2's "Going deeper" section covers this.

Choosing a model tier that matches task difficulty

Within a model family, companies offer tiers ranging from large, slow and expensive to small, fast and inexpensive. Anthropic's comparison describes the fastest tier as suited to high-throughput, latency-sensitive tasks such as classification, extraction and routing, and the largest tier as suited to demanding reasoning and long tasks.

Ask four questions when choosing a tier.

1. How complex is the judgment? Are rules clear, or must several conditions be weighed?
2. How much work, how quickly? Thousands of cases a day or a few a month? Must responses arrive within seconds?
3. How harmful would an error be? Can a person correct it immediately, or will it go outside the organization and be hard to reverse?
4. Have you checked whether a small model is sufficient? Have you compared small and large models on a few dozen representative cases?

The final question is central. A small comparison experiment using representative examples of your own work is more reliable than tier charts or names. Record accuracy, omissions, speed and human correction time to see which tier is needed in numbers. High-volume work also commonly uses routing, sending easy requests to a small model and only difficult ones to a large model.

Multimodal input

Multimodal means the ability to handle several input formats, including images, PDFs, audio and video as well as text. You can ask a model to read an amount from a receipt photo, describe trends in a chart image or summarize a meeting recording.

There are three properties to understand.

1. Images and sound also use tokens. Google states that images, video and audio are converted to token counts according to defined rules, while Anthropic divides images into small patches for token accounting. A photographed document still consumes context window capacity. Whether it uses less than text depends on resolution and model. Remember that reducing image size can make small text harder to read.
2. Images introduce distinct recognition and extraction errors. Anthropic states that low-quality, rotated or very small images may be misinterpreted, spatial judgments are approximate, and counts of many small objects may be inaccurate. Common workplace problems include small print, blurry scans, matching table rows and columns, and reading chart axes, units and legends.
3. Some judgments cannot be made. Anthropic states that its model cannot determine whether an image was AI-generated and should not be relied on to detect forged images.

Separate audio into transcription and interpretation. Names, technical terms and overlapping speech are vulnerable during transcription; determining who agreed to what is vulnerable during interpretation.

Anthropic advises that images work best when placed before text. When numbers matter, original files such as spreadsheets or source text are more accurate inputs than images.

Situations where output must not be taken at face value

The following checklist brings these concepts together. If any signal applies, verify the output before using it.

Verification also depends on output type. Check facts against original sources, calculations by another method, interpretations through their premises and counterexamples, and creative work against purpose, audience and constraints. Reviewing everything in the same way misses important errors.

Going deeper

Why tools should handle calculations and aggregation

A language model predicts the next token; it is not a calculating machine. It usually handles short additions correctly, but may produce plausible yet incorrect numbers for totals in long tables, ratios, rates of change and aggregations with several conditions. The problem is that incorrect numbers look just as natural as correct ones.

Google recommends code execution for arithmetic, counting and tasks requiring calculation. This feature lets the model write the calculation as a program and actually run it. People can inspect the formula as well as the result. If code execution is unavailable, it is safer to calculate in a spreadsheet and have AI only interpret or describe the results.

This does not mean "AI is poor with numbers, so never use it for numerical work." It means divide responsibilities. A reliable calculation tool performs the arithmetic, a language model handles interpretation and writing, and a person checks the connection. Numbers in a report should be traceable to who calculated them and how.

Models change even under the same name: versions and reproducibility

The same instructions may produce a different format today than yesterday. Probabilistic generation may be responsible, but the underlying model may also have changed. A chat product's default model can be replaced as the product updates. Developer APIs offer snapshot IDs that pin a model to a particular point in time. OpenAI recommends pinning a specific snapshot and maintaining an evaluation set to measure prompt behavior when production use requires consistency.

If your team's template suddenly behaves differently, check whether the model or settings changed before blaming the instructions alone. For regularly used AI features, keep an evaluation set of 10–20 representative cases and expected results. Run the same cases again whenever the model changes to identify differences.

At work

Dawn Bakery's purchasing team: a butter price report

Oh, a buyer at the head office of handmade dessert franchise Dawn Bakery, needs a price-trend report to decide next month's butter purchase volume. Using an internal AI tool with web search disabled, Oh asks, "Summarize domestic butter wholesale prices over the last three months and the outlook for next month." A neat table of monthly prices and forecasts appears.

Before pasting it into the report, Oh recalls three questions.

1. When is this information from? Prices for the last three months probably postdate the knowledge cutoff. With search disabled, the model could not see the actual prices.
2. Where did these numbers come from? The table has no sources. Specific numbers may be plausible values filled in by probabilistic generation.
3. How harmful would an error be? The figures guide purchasing, and a bad purchase can cause inventory losses.

Oh first considers two alternatives. Asking the same question repeatedly and using the numbers if they agree is rejected because repetition cannot replace fact-checking. Enabling search and asking again is retained only as a supplementary method: search results might mix retail and wholesale prices or differ from the prices the company actually receives.

Instead, Oh attaches PDFs of the last three months' price lists from three suppliers and asks again: "Use only numbers in the attachments. Beside every table value, state its source document and page number. Mark months absent from the material as 'No data.' Do not calculate averages or rates of change yourself; just identify the columns to calculate." Oh calculates averages and rates of change separately in a spreadsheet. The outlook becomes a separate request: "Describe trends in the table above, separating verified facts from estimates into different paragraphs."

The result is less impressive-looking than the first, but every number has evidence and missing months are visible as gaps. Oh checks the table once against the original PDFs before adding it to the report. The key is understanding the information the model cannot see and the checks it does not perform, then filling those gaps with evidence and tools, rather than merely doubting its ability.

Summer Forest Camping's operations team: reporting from dashboard screenshots

Kang, an operations manager at camping equipment online retailer Summer Forest Camping, submits a weekly performance summary for Monday's executive meeting. To save time, Kang provides three dashboard screenshots to AI and asks, "Summarize last week's performance in five lines." The dashboard includes a dual-axis chart with revenue on the left and conversion rate on the right, plus a category table in small print.

The summary sounds plausible, but a teammate identifies two problems. The conversion rate is ten times its actual value, and revenue for the "Tents" row has been replaced with the value from "Tarps" immediately below. Kang separates the causes: confusion between the axes' units is an interpretation error; the swapped rows are a reading error in matching rows and columns in a small-print table.

Kang weighs three choices. Higher-resolution screenshots could reduce reading errors but leave axis interpretation errors. Avoiding images entirely would prevent misreading but lose the benefit of an overview of trends. Kang therefore decides to use original files for numbers and images only for trend descriptions.

Kang attaches the CSV downloaded from the dashboard as the numerical source, along with totals and week-on-week changes calculated in a spreadsheet. Images are used only for this request: "Describe two noticeable changes in the chart in words. Do not use numbers." Because this is a lightweight task repeated in the same format each week, Kang spends four weeks comparing whether a fast standard model can meet the criteria instead of a large reasoning model. With equivalent results, the fast model becomes the regular choice.

The key is separating what images do well—explaining the overall picture—from what original data does well—providing exact numbers—and responding separately to reading and interpretation errors.

Common mistakes

1. "Specific means accurate." Case numbers, dates and decimal figures look trustworthy. But probabilistic generation easily produces fake information in plausible formats. Specificity is a signal to check.
2. "The same answer repeatedly means it is true." A model can repeat the same error. Repetition cannot replace verification, although disagreements can serve as warning signs.
3. "A temperature of 0 makes it accurate." Lowering temperature does not guarantee accuracy; even 0 may not reproduce results exactly, and changing the default can reduce some models' performance. Accuracy requires evidence and verification.
4. "A large context window means current knowledge." Context measures how much can be read; the cutoff marks when learning occurred. These are separate dimensions.
5. "A citation means it has been verified." Even a real source may not support the claim in the cited passage, and the search result itself may be old. A citation starts the check.
6. "It corrected itself after I objected, so now it is right." Models tend to change answers to match users' objections. External evidence determines which answer is correct.
7. "An expensive reasoning model is always better." For simple high-volume work, it may only add delay and cost. Reasoning models do not necessarily know more current information.
8. "Uploading a photo saves space and gives accurate numbers." Images occupy context too; savings depend on resolution, and small print, tables and chart axes are common sources of misreading.

Summary at a glance

Check the official docs

• OpenAI — Understanding and counting tokens: Token definitions, English estimates, reasoning-token billing and output limits.
• Google — Understand and count tokens: Gemini estimates and token accounting for images, audio and video.
• Anthropic — Context windows: The context window as working memory, and context rot.
• Anthropic — Models overview: Definitions of both knowledge cutoffs and uses of each tier.
• Anthropic — Reduce hallucinations: Allowing uncertainty, quoting first, withdrawing unsupported claims and comparing repeated runs.
• Anthropic — Vision: Image-input limits: low quality, spatial judgment, counting and identifying AI-generated images.

Self-check

1-1. Write one line each distinguishing what tokens, the context window and the knowledge cutoff measure.

1-2. A model's documentation lists March as its reliable knowledge cutoff and May as its training data cutoff. Explain how to treat its answer about an event in April.

1-3. A reasoning model gives a two-line answer, but the usage screen shows as many output tokens as a long report. Give a possible reason.

1-4. Match each output with the best verification method. Outputs: (a) promotional copy for a new product; (b) quarterly revenue totals; (c) a competitor's launch date; (d) an interpretation of customer churn causes. Methods: comparison with the original source; recalculation with another tool; checking premises and counterexamples; evaluation against purpose, audience and constraints.

1-5. You plan to photograph twenty handwritten paper surveys and have AI count responses to each question. Describe two possible errors and a response to each.

1-6. You ask the same question in five new conversations and receive the same answer every time. Write one sentence about what this tells you and one about what it does not.

1-7. A team has three tasks: standardizing English spellings of 500 product names daily, writing a monthly memo on entering a new market, and drafting internal schedule announcements twice weekly. Design a two- or three-line rule for choosing model tiers for each task, and describe how to test whether the rule is appropriate.

Hands-on tasks