Temperature and Randomness

Temperature and Randomness

Every time you ask a language model a question, it doesn't simply find "the answer" — it generates a response probabilistically, choosing from a distribution of possible next tokens at each step.

Temperature is the setting that controls how that distribution behaves — essentially, how creative or predictable the model is.

How Token Generation Works

When a language model generates text, it produces a probability distribution over all possible next tokens at each step. For example, after "The sky is ___", the model might assign:

• "blue" → 45% probability
• "clear" → 20%
• "grey" → 15%
• "dark" → 8%
• everything else → 12%

At temperature 0, the model almost always picks the highest-probability token ("blue"). Responses are near-deterministic — but not perfectly guaranteed to be identical every time, because of how models batch requests and run on hardware behind the scenes.

At temperature 1, the model samples according to the actual probabilities. "Blue" is likely but not guaranteed.

At higher temperatures, the distribution is flattened — unlikely tokens get boosted. Responses become more surprising, more varied — and potentially more nonsensical.

The exact range for temperature isn't universal — it's set by each provider's API. The Anthropic API accepts temperature between 0 and 1. Other providers allow a wider range.

As of October 2026: OpenAI's reasoning models (GPT-5 and onwards) don't accept a temperature setting at all — the model controls its own sampling as part of how it reasons before answering. Temperature mainly applies to non-reasoning models and to the final-answer generation step of some reasoning systems.

Practical Guidance

Low temperature (0–0.3): Use when you want consistent, factual, predictable output.

• Extracting data from documents
• Writing code
• Answering factual questions
• Classification tasks

Medium-to-high temperature: Use for creative tasks where you want variety and surprise.

• Brainstorming
• Creative writing
• Generating diverse options

Exactly what counts as "high" depends on the provider's range — on a 0–1 scale, that might mean 0.7–1.0; on a wider 0–2 scale, more like 1.2–2.0. Check the API you're using rather than assuming a universal scale.

Most consumer AI interfaces don't expose this setting — they pick a sensible default. When you use an API directly (where the model accepts one), you have full control.

Top-p and Top-k

Temperature isn't the only way to control generation. Two other common settings:

Top-p (nucleus sampling): Instead of sampling from all possible tokens, only sample from the smallest set of tokens whose cumulative probability reaches p. With top-p = 0.9, you're only choosing from tokens that collectively make up 90% of the probability mass — cutting out very unlikely options.

Top-k: Only consider the top k most probable tokens. With top-k = 50, the model picks from the 50 most likely options at each step.

These parameters are often used together to control the quality/diversity trade-off. Most users never need to touch them — but understanding they exist explains why AI outputs are probabilistic, not deterministic.

Key takeaway: AI outputs are not "looked up" — they're generated. That's why you can ask the same question twice and get different answers.