What are Tokens?

What are Tokens?

Language models don't read text the way you do. They don't process letters, words, or sentences as discrete units. Instead, they process tokens — chunks of text that sit somewhere between characters and words.

Understanding tokens helps you understand why AI models behave the way they do: why they count letters wrong, why long conversations get expensive, and why there are limits on what you can send in one go.

How Tokenisation Works

A tokeniser breaks text into tokens before the model ever sees it. Common approaches:

• "cat" → one token
• "unbelievable" → might be two tokens: "unbel" + "ievable"
• " hello" (with a leading space) → often a different token to "hello"
• Numbers and punctuation split unpredictably

As a rough rule of thumb, 1 token ≈ 4 characters ≈ ¾ of a word in English. So 100 words ≈ 130 tokens.

Other languages are different. Languages like Chinese or Arabic, which are less represented in training data, often require more tokens per word, making them more expensive to process.

Why Does Tokenisation Matter?

1. Counting and spelling tasks

Models are notoriously bad at counting letters (e.g., "how many r's in 'strawberry'?"). That's because they never see individual letters — they see tokens. "strawberry" might be one or two tokens, and the model has no direct access to the underlying characters.

As of October 2026: current reasoning models usually get the strawberry test right, because they reason step by step before answering rather than guessing from the token straight away. The underlying limitation hasn't gone away — it's just worked around for well-known examples like this one.

2. Context windows

Models have a maximum number of tokens they can process at once — this is the context window. Context windows have grown enormously: from about 4,000 tokens in 2022 to about 1 million tokens in 2026, across the frontier labs.

Everything you send (your message + the conversation history + any documents) has to fit in this window. If it doesn't, older parts get cut off.

3. Cost

APIs charge per token — both input tokens (what you send) and output tokens (what the model generates). A long document you paste in costs money. A short, focused prompt is cheaper.

Input and Output Tokens

When you send a message to a language model:

• Your message gets tokenised → input tokens
• The model generates a response, one token at a time → output tokens

Output tokens are typically more expensive than input tokens (generating is harder than reading).

The model generates tokens probabilistically — at each step, it's picking the most likely next token given everything that came before. This is why models can be "steered" by how you phrase your prompt — you're influencing which token the model thinks comes next.

A Quick Example

The sentence: "The quick brown fox"

Might tokenise as: ["The", " quick", " brown", " fox"] — 4 tokens.

Or for a less common phrase: "Tokenization is fascinating"

Might tokenise as: ["Token", "ization", " is", " fas", "cin", "ating"] — 6 tokens.

The exact split depends on the tokeniser, which is trained separately from the model.