Chapter 5. Adoption value and the organizational view

Learning objectives

• Explain subscription and token-based billing, read pricing units and choose the approach that fits the intended use.
• Compare total cost and time, including preparation, review and rework, rather than unit prices alone.
• Classify business use cases by impact and risk to choose the first adoption target.
• Demonstrate impact through baselines and measurable metrics, avoiding the traps of usage measures.
• Explain pilot sequencing, expansion/revision/stopping criteria and organizational governance.

Introduction

Tak, who runs a furniture workshop, bought paid AI subscriptions for all twelve employees last spring. Three months later, the accountant asked, "What has this monthly spending improved?" Tak could not answer. Staff said things were easier, but nobody knew which tasks were faster, by how much, or whether errors had declined. Only four of the twelve used AI every week. The accountant proposed canceling all subscriptions; frequent users wanted an even more expensive plan. There were no numbers to decide between them.

The problem was not that AI lacked value. The organization had not chosen tasks, measured the situation before adoption or defined success. It therefore had no basis for deciding whether to continue, expand or stop.

This chapter takes the manager's view of AI adoption. It begins with cost structures—subscriptions, token billing and total cost—and criteria for selecting the first task. It then covers metrics and baselines, designing and evaluating pilots, and organization-wide operations. This is useful both to managers and practitioners proposing AI to their teams. It extends Delegation and Diligence from individual AI fluency to the organizational level.

Key concepts

Subscription and token billing: a flat-rate plan and an electricity bill

AI costs broadly follow two models.

Subscriptions are often more predictable for routine document work performed by staff; token billing often suits high-volume automated processing. Remember that the two are separate budgets. OpenAI explains that API use follows API pricing separately from subscription-included usage, and that using your own API key incurs API rates.

Units to check in a pricing table

Compare pricing on equivalent units. Check the following.

Two points deserve emphasis. First, subscriptions do not mean unlimited access to every feature; most plans have feature-specific limits. Second, token billing does not have just one rate. OpenAI states that input, cached-input and output tokens may have different prices, and offers batch processing at 50% less than ordinary calls when immediate responses are unnecessary. Exact prices change often, so recheck official pricing just before deciding.

Total cost rather than unit price

OpenAI explains that a lower price per million tokens does not necessarily mean lower total cost. Models tokenize the same text differently and generate different amounts of output and reasoning tokens—the internal tokens discussed in Chapter 1 that are billed as output. Compare actual representative tasks, not just visible answer length.

Human costs add to this. AI adoption costs more than service fees.

1. Preparation: Organizing material, writing prompts and templates, creating test cases.
2. Training: Teaching staff and writing guides.
3. Operations: Administrator settings, permission management, policy updates.
4. Review: Time spent checking and correcting results.
5. Rework: Time spent starting over when results cannot be used.

A low-fee tool can become expensive if it requires substantial review and rework. A simple formula is: total cost = service fees + preparation, training and operations costs + (review time + rework time) × hourly labor cost.

Selecting use cases: one step of a task, not an entire role

"Introduce AI into sales" is too broad to test. Choose one step of the work, not a whole role. In sales, separate information gathering, proposal drafting, customer response drafting and contract-term review, then select one as the first target.

Good first candidates meet these conditions.

Value accumulates steadily in work that recurs every week and is easy for people to verify, rather than in work that stands out in demonstrations.

Classifying by impact and risk

Classifying candidates by impact—time savings and quality improvement—and risk reveals priorities.

Treat risk as a combination of four factors rather than a single number.

1. Likelihood of error: How often does AI get this task wrong?
2. Magnitude of harm: How much damage can one error cause?
3. Detection difficulty: Can people easily recognize the mistake?
4. Reversibility: Can an incorrect result be undone?

Frequent errors can still mean low risk if obvious and easily corrected. Rare errors can mean high risk if difficult to detect, sent outside the organization and irreversible. Meeting minutes, internal inquiry classification and drafting often occupy the high-impact, low-risk category. The high-risk decisions from Chapter 4 require human-review design first, even when impact is large.

Compare with solutions that do not use AI

AI is not the answer to every problem. Inconsistent document formats may need one standard template; repeated questions may first need a well-organized guidance page. Include a non-AI improvement option in adoption comparisons. AI's real contribution is the value remaining after better forms, ordinary search and calculation tools, and clearer work rules solve what they can. This is the same principle behind Anthropic's recommendation to start agent design with the simplest possible solution.

Four conditions for good adoption metrics

Anthropic describes four conditions for good success criteria.

• Specific: State what is to be achieved, such as accurate inquiry classification, rather than good performance.
• Measurable: Use numbers or a consistently applied rating scale.
• Achievable: Set realistic goals based on industry benchmarks, earlier experiments or expert knowledge.
• Relevant: Choose metrics matching the task's purpose and users' needs.

Anthropic also explains that criteria should usually span several dimensions. Measuring only processing time can miss declining quality; measuring only quality can miss rising cost. The basic approach is to measure time and quality together.

Baselines and measurement design

A baseline is the pre-adoption value of the same metric. Without it, improvement cannot be demonstrated. OpenAI also recommends using the evaluation set created during prompt refinement as a baseline for later comparison. Define the following when designing measurement.

1. What is measured? Define the metric and denominator, such as per case or per day.
2. When and how much? Choose before-and-after periods with comparable seasonality and workload.
3. Which sample? Use work of similar difficulty; do not select only easy tasks.
4. Who assesses quality, and how? Use the same rubric and evaluators.
5. How are exceptions recorded? Keep separate records of failures and cases involving manual rework.

Types of metrics and their traps

Three traps are common.

• Mistaking usage for outcomes: More logins or questions indicate interest, not proven value.
• Measuring generation time alone: A one-minute draft does not speed up work if review and editing take too long. Measure total time to completion, from preparing material through approval and delivery. Include time spent restarting unusable cases.
• Equating saved time with cost reduction: Less staff time does not necessarily reduce spending. Saved time is additional work capacity. Show how that capacity was used—handling more cases, reducing outsourcing or doing new work—to establish real value. Avoid counting the same benefit twice.

Pilot sequence and criteria for expanding, revising or stopping

A pilot is a small-scale trial operation. Follow this sequence to evaluate impact. Each step uses the previous one's results, so the order is difficult to change.

1. Gather candidate tasks.
2. Prioritize by impact and risk and choose the first target.
3. Define success criteria and stopping conditions in advance.
4. Measure current performance to establish a baseline.
5. Operate a small pilot and measure again in the same way.
6. Compare the values and decide whether to expand, revise or stop.

Set decision criteria before starting. If you define them later, the effort already invested can encourage favorable interpretations.

Do not generalize a small pilot directly to the whole organization. Results from a few enthusiastic users may not translate into equivalent gains for everyone.

Organizational governance and capabilities

When AI adoption becomes organizational rather than a matter of individual techniques, it needs an operating framework. The US National Institute of Standards and Technology's AI Risk Management Framework (AI RMF 1.0) is voluntary and organizes its core into four functions.

NIST also published the Generative AI Profile, NIST AI 600-1, in 2024 to address generative-AI-specific risks. Even small organizations can reveal gaps by filling out a one-page table for the four functions. If staff use tools unknown to the company, often called shadow AI, understanding current use comes before measurement or response.

Human capability is part of the operating framework too. The four AI fluency competencies introduced in this book—Delegation, Description, Discernment and Diligence—can guide training design. Teach how to decide what to delegate, give good instructions, evaluate results and act responsibly in balance, so adoption does not depend on a few people's techniques.

Going deeper

What if nobody uses the tools after adoption?

If licenses are distributed but few people use them, the cause is usually not the tool but uncertainty about applying it to one's own work. Buying more licenses or forcing use rarely helps. These measures are more effective.

1. Show task-specific examples. Concrete accounts of reducing the time spent on your team's weekly reporting motivate people more than general feature demonstrations.
2. Share tested templates. Make Chapter 2's templates and test cases team assets.
3. Designate departmental support contacts. Early adopters can serve as colleagues' first contact for questions.
4. Address fear. Job concerns can prevent use. Explain that the purpose is to reduce repetition and free time for judgment and customer interaction, while people remain responsible for results.

Usage rate is a useful supporting measure of adoption, but should not itself become the goal.

Avoiding dependence on one supplier

When an entire workflow is tightly fitted to one provider, product or feature, changes to its prices, features or policies can disrupt the organization. This is vendor lock-in. As Chapter 3 showed, product names and forms can change within months.

It cannot be eliminated completely, but its impact can be reduced. Store prompts and test cases in documents outside the product, keep original work material in company systems and connect or copy it into AI, and ensure important tasks can be run on other products using representative cases. Check export options for outputs and conversations before adoption too.

At work

Engrave Printing: a quotation-reply pilot

Jung, a manager at commercial printer Engrave Printing, is asked by the owner to show in numbers what AI adoption could improve.

Jung gathers six recurring-task candidates from employees. More than 30 quotation-inquiry emails arrive each day, and staff can review and edit the reply drafts, making the task high-impact and low-risk. Client credit evaluation also has high impact but high risk because errors could substantially affect business. Jung chooses quotation replies and first defines success—at least 30% shorter completion time per case without an increased repeat-inquiry rate—and a stopping condition: even one reply with an incorrect price reaching a customer.

For two weeks before adoption, Jung measures completion time and the proportion of customers asking the same thing again: 12 minutes on average and an 18% repeat-inquiry rate. Three employees then use AI drafts for four weeks. A suggestion to measure only generation time is rejected; Jung includes review and editing in completion time. Results average 7 minutes and a 15% repeat-inquiry rate. Five percent of drafts are unusable and rewritten from scratch; that time is included in the average.

Because staff use the interface directly, Jung chooses per-seat subscriptions and compares monthly fees for three people with gained capacity: 5 minutes per case × 600 cases monthly = 50 hours a month. Rather than reporting 50 hours as payroll savings, Jung explains how they were used: 30 hours for new-client consultations and 20 for overdue inventory organization. The report begins with "Five minutes saved per case, the repeat-inquiry rate improved by 3 percentage points, no stopping condition triggered," rather than staff satisfaction. The owner expands the quotation-reply rollout and defers reconsidering client credit evaluation until human-review procedures are designed.

Sharing Bridge: choosing the first task

Local nonprofit Sharing Bridge has five full-time staff managing 3,000 donors. Its executive director considers three possible AI uses: writing public event notices, organizing internal meeting minutes and sending personalized thank-you messages to individual donors.

The director compares them in one table.

Thank-you messages appear to offer the greatest impact, but involve personal information and direct delivery to thousands of donors, making mistakes irreversible. A board member argues that starting with maximum impact would show results. The director defers the task because high-risk work needs controls first. Event notices are low-risk but too infrequent to demonstrate impact easily. Minutes recur often, are easy for people to verify and can be handled in the nonprofit's approved business environment.

The director selects minutes and records the previous month's preparation time and number of follow-up questions asking what was decided as a baseline. Thank-you messages will be reconsidered next quarter using AI only to create templates without personal data, with people inserting names and donation records.

Common mistakes

1. "A low unit price means low total cost." Models tokenize differently and generate different amounts of output and reasoning tokens. Review and rework also cost time.
2. "The subscription includes API use." Chat subscriptions and APIs follow separate pricing rules.
3. "More use means success." Usage indicates interest, not proven value.
4. "Faster drafts mean faster work." Judge total time to completion, including review and rework, rather than generation alone.
5. "Saved hours equal reduced costs." Time savings provide capacity; show how that capacity is used to establish value.
6. "Adopt first, measure later." Without a pre-adoption baseline, the ability to demonstrate impact is lost.
7. "Start with the greatest impact." High-impact, high-risk work needs controls first. Choose a high-impact, low-risk first case.
8. "Define stopping criteria after seeing results." Set criteria before starting to avoid being influenced by invested effort.

Summary at a glance

Check the official docs

• OpenAI — Understanding and counting tokens: Prices by token type and total cost rather than unit price.
• OpenAI — ChatGPT Work and Codex: Subscription usage versus API pricing rules.
• OpenAI — Batch API: Discounts and processing times for non-urgent bulk work.
• Anthropic — Define success criteria and build evaluations: Four conditions for success criteria and multidimensional assessment.
• NIST — AI Risk Management Framework: AI RMF 1.0 and Generative AI Profile overview.
• NIST — AI RMF Core: Explanations of the four functions.
• Anthropic — The AI Fluency Framework: Definitions of Delegation, Description, Discernment and Diligence.

Self-check

5-1. Explain subscription and token billing through everyday analogies and state which use each suits.

5-2. State the four conditions for good adoption-impact metrics.

5-3. Give two problems with reporting adoption impact without a baseline.

5-4. State the four factors considered together in risk assessment, and explain when frequently incorrect work may nevertheless be low-risk.

5-5. Identify the problem in "AI saved staff 40 hours this month, so we saved 40 hours of labor cost," and explain how to revise the report.

5-6. Three months after adoption, a report contains only weekly active users and total questions. What cannot be judged from it, and which metrics should be added?

5-7. Design a one-page pilot plan for one team task. Include the task step, baseline, time and quality metrics, duration, and conditions for expansion, revision and stopping.

Hands-on tasks