Parameters and Model Size

Parameters and Model Size

When people describe an AI model, they often quote a number of parameters: "a 7 billion parameter model", "GPT-3 has 175 billion parameters". But what does that actually mean, and does bigger always mean better?

What is a Parameter?

A parameter is one of the learnable numerical values in a neural network — a weight or bias. During training, these values are adjusted to minimise prediction error.

When training is complete, the parameters are fixed. They encode everything the model learned: grammar, facts, reasoning patterns, code syntax, world knowledge.

A model with more parameters has more capacity to learn complex patterns. But it also requires:

• More training data to use that capacity effectively
• More compute to train and run
• More memory to store

A 7B (7 billion) parameter model in 16-bit precision takes roughly 14GB of memory just to load. A 70B model takes ~140GB. This is why running large models requires expensive hardware.

Does Size Always Win?

Not anymore. Early scaling research suggested bigger models were almost always better. But recent work has complicated that picture:

Efficient smaller models: a small model trained with better data and techniques can outperform an older, much larger one. Size alone doesn't determine capability.

The Chinchilla finding (DeepMind, 2022): Most large models were undertrained. You get better results training a smaller model on more data than training a huge model on less data. Optimal model size and training data scale together.

Quantisation: Models can be compressed (quantised) to use less memory with minimal performance loss. A 70B model quantised to 4-bit precision takes roughly 35GB of memory — more than any single consumer GPU has, but well within reach of a multi-GPU workstation or a modest cloud instance.

Mixture of experts (MoE): Instead of one dense network, an MoE model splits into many smaller "expert" sub-networks, with a router sending each token to only a few of them at a time. This gives a model two different size numbers: its total parameters (everything stored on disk) and its active parameters (what's actually computed for a given token). A model can have a huge total parameter count while only using a fraction of it per token — much of the capability of a far larger dense model, at a fraction of the inference cost. Most frontier and open models now use this approach, which changes the memory maths above: you often need to fit the total parameter count in memory even though only the active parameters are computed per token.

Open vs Closed Models

Closed models: accessed via API only. You don't know the exact size or architecture. You pay per token.

Open models: weights are publicly available. You can run them yourself. The community can fine-tune and adapt them.

As of October 2026: closed frontier models include ChatGPT, Claude and Gemini. Open-weight models are led by DeepSeek, Qwen, Kimi and GLM, alongside Gemma, Mistral and OpenAI's own open-weight gpt-oss models. Meta's Llama 4 (April 2025) was the last Llama release — Meta's Muse Spark (April 2026) was the company's first closed-weights model.

Why model size matters for you:

• API usage: larger models cost more per token, but often produce better outputs
• Local models: the model needs to fit in your hardware's memory
• Fine-tuning: smaller models are cheaper to fine-tune for specific tasks
• Latency: smaller models respond faster — important for real-time applications

The right model for a task depends on capability requirements, cost, latency, and privacy needs. There's no single "best" model.