What are tokens in AI? How LLMs count, price and limit your text

Tokens are the small chunks of text, often a word or part of a word, that a language model reads and writes; models count, price and limit everything in tokens, not words. This guide shows how tokenizers split real text, how to count tokens, how per-token pricing and context windows work, and how to use fewer tokens without worse answers.

What are tokens in AI?

Tokens are the units of text a large language model actually processes: a whole common word, part of a longer word, a punctuation mark, a space plus a word, or a few digits. Before a model reads your prompt, a tokenizer converts the text into a list of token IDs from a fixed vocabulary, and the model writes its answer one token at a time, which the tokenizer turns back into text.

Tokens matter to you for three reasons:

• Cost: APIs charge per token, usually quoted per million.
• Limits: every model has a context window measured in tokens, and chat apps meter usage in tokens behind the scenes.
• Speed: generation time grows with the number of output tokens.

If you want the bigger picture of how a model uses these tokens, our explainer on how ChatGPT works walks through next-token prediction step by step.

How does a tokenizer split text?

Most modern tokenizers use byte pair encoding (BPE) or a close relative: they start from bytes and merge the most frequent pairs into larger pieces, so common words become single tokens and rare words are built from sub-word pieces. OpenAI's tiktoken README gives the classic example: BPE will often split "encoding" into "encod" and "ing," so the model sees the common "ing" piece across many words.

To show what this looks like, we ran a few strings through tiktoken's o200k_base encoding (the one tiktoken maps to GPT-4o). The vertical bars mark token boundaries:

Four patterns show up in almost every tokenizer:

1. Common English words are cheap. The English sentence is nine tokens for eight words.
2. Other languages cost more. The Turkish version of the same request is shorter in characters but uses 13 tokens, because the vocabulary has fewer Turkish word pieces.
3. Code and JSON are punctuation-heavy. Quotes, brackets and colons often become their own tokens.
4. Numbers fragment. Long numbers and dates split into chunks of a few digits, one reason models can stumble on arithmetic.

How many characters or words is a token?

For English text, one token averages about 4 characters, or roughly three quarters of a word. The tiktoken README says each token corresponds to "about 4 bytes" on average, and Google's token guide says a Gemini token is "about 4 characters" and that "100 tokens is equal to about 60-80 English words."

Use these rules of thumb (derived from Google's 60 to 80 words per 100 tokens) for estimates, never for billing:

Anthropic adds one more caution on its pricing page: Claude models from 4.7 onward use a newer tokenizer that "produces approximately 30 percent more tokens for the same text" than earlier Claude models. A count you measured on one model is not valid for another.

How do you count tokens exactly?

You count tokens exactly with the tokenizer or counting endpoint of the model you plan to use: tiktoken for OpenAI models, the token counting API for Claude, and the token counting method in the Gemini API. Every API response also reports the tokens a request used in its usage field.

For Claude, the token counting endpoint is free to use (it has its own rate limits) and returns the input token count for a full request, including the system prompt, tools, images and documents:

curl https://api.anthropic.com/v1/messages/count_tokens \  -H "x-api-key: $ANTHROPIC_API_KEY" \  -H "content-type: application/json" \  -H "anthropic-version: 2023-06-01" \  -d '{    "model": "claude-opus-5-5",    "system": "You are a scientist",    "messages": [{"role": "user", "content": "Hello, Claude"}]  }'

The response is a small JSON object with an input_tokens field. Anthropic notes the result is an estimate that can differ from actual usage "by a small amount."

For OpenAI models, tiktoken counts locally in Python:

import tiktokenenc = tiktoken.get_encoding("o200k_base")text = "Please summarize this report in three bullet points."print(len(enc.encode(text)))  # 9

In Claude Code, /context shows what is filling the context window during a session, which is the quickest way to see how many tokens your files, tools and conversation take up.

How does per-token pricing work?

AI APIs charge separately for input tokens (what you send) and output tokens (what the model writes), quoted in dollars per million tokens (MTok), and output usually costs several times more than input. Your bill for one request is input tokens times the input price plus output tokens times the output price.

Claude API prices as of November 2026, from Anthropic's pricing page:

A worked example: a request with a 2,000-token prompt and a 500-token answer.

Three costs are easy to miss:

• Conversation history is resent. Chat APIs are stateless, so turn ten of a conversation pays again for turns one to nine as input.
• Thinking tokens are output. Anthropic's context window docs say thinking tokens "are billed as output tokens" and count toward rate limits, even though you may not see the reasoning text.
• Tools add hidden input. Tool definitions and the system prompt that enables tool use add input tokens to every request that declares tools.

Images count as tokens too. Claude splits an image into 28 by 28 pixel patches, so a 1000 by 1000 pixel image costs 1,296 tokens, according to Anthropic's vision docs.

On subscription apps such as ChatGPT Plus or Claude Pro you don't see a per-token bill, but long messages, large files and long answers still use up your usage limits faster. Our guide to Claude Code usage limits explains how that plays out for coding sessions.

What is a context window in tokens?

A context window is the maximum number of tokens a model can consider in one request, counting both what you send and what it generates. Anthropic describes it as the model's "working memory": the system prompt, every message, tool results, images, documents and tool definitions all count, and so does the answer, including thinking.

As of November 2026, current Claude models such as Opus 5.5, Sonnet 5.5 and Haiku 5.5 have a 1M-token context window and can generate up to 128k output tokens per request. If the input alone exceeds the window, the API rejects the request with a "prompt is too long" error.

Bigger is not automatically better. Anthropic's docs warn that "as token count grows, accuracy and recall degrade," a problem it calls context rot. A focused 20,000-token prompt often gets a better answer than the same question buried in 500,000 tokens of loosely related material. Our post on the context window covers this in depth, and context engineering covers how to choose what goes in.

How can you use fewer tokens?

You use fewer tokens by sending less irrelevant context, reusing the context you do send, and asking for shorter output. The list below goes roughly from easiest to most involved.

1. Send excerpts, not whole files. Paste the section that matters, not the 80-page PDF, when the question is about one section.
2. Start a new chat for a new task. Long threads resend their whole history each turn. In Claude Code, /clear between unrelated tasks and /compact in long ones.
3. Ask for the length you need. "Answer in five bullets under 100 words" cuts output tokens, which are the expensive ones.
4. Drop the filler in prompts. Repeated instructions, long pleasantries and pasted boilerplate all cost input tokens without improving answers.
5. Prefer compact formats. For data you pass in, CSV or a short table usually uses fewer tokens than verbose JSON.
6. Downsample images. Send screenshots at the resolution the task needs.
7. Cache repeated prefixes. With prompt caching, a stable system prompt or document is billed at the cache-hit rate after the first request: $0.20 instead of $4 per million on Claude Opus 5.5. See our guide to prompt caching.
8. Batch work that can wait. Anthropic's Batch API gives a 50 percent discount on input and output tokens for asynchronous jobs.
9. Use the smallest model that does the job. Classification and extraction often work well on a Haiku-class model at a fraction of the price.

Here is a prompt pattern that keeps both input and output tight:

Context: the pasted section is from our refund policy (section 4 only).Task: list every condition under which a customer gets a full refund.Format: a numbered list, one line each, no introduction, under 80 words.

For a fuller playbook on API spend, including routing and measurement, read our guide to reducing LLM API costs.

Token checklist for your next AI project

Use this before you build anything that calls a model in a loop:

• Count a realistic request with the counting tool for the exact model you will use.
• Multiply input and output tokens by current prices, then by expected daily volume.
• Log the usage field from every response, so you see real numbers, not estimates.
• Check that system prompt, tools and history fit comfortably under the context window, with room for the answer.
• Turn on prompt caching for any prefix you send more than once.
• Set a maximum output length that fits the task.
• Re-count when you switch models, because tokenizers differ.

Token costs add up fastest with coding agents, which read many files and run for hours. Orkio runs agents such as Claude Code on an always-on cloud box on your own Claude account, and its Insights view shows what that work costs per project, per model and per merged pull request.

Frequently asked questions