Foundation Models and Fine-tuning

Foundation Models and Fine-tuning

One of the most important ideas in modern AI is the foundation model — a large model trained on broad data that can be adapted to many different tasks.

This changed how AI is built and deployed. Instead of training a new model for every application, you start with a foundation model and adapt it. It's a bit like education: a person who learns to read, write and think clearly can be trained for many specific jobs, rather than each job requiring education from scratch.

What is a Foundation Model?

A foundation model is a large model trained on a huge, diverse dataset — typically a significant portion of the internet, plus books, code, and other text.

During this pre-training, the model learns general capabilities:

• Grammar and language structure
• Facts about the world
• Reasoning patterns
• How to write code
• How to follow instructions

ChatGPT, Claude, Gemini, and Llama are all built on foundation models. They weren't trained to do one thing — they were trained on everything, and that general capability is what makes them useful across so many tasks.

Fine-tuning

Fine-tuning is the process of taking a pre-trained foundation model and training it further on a smaller, specific dataset to improve performance on a particular task.

Examples:

• Take a general language model → fine-tune on medical texts → better at answering health questions
• Take a general model → fine-tune on customer service conversations → better at handling support queries
• Take a general model → fine-tune on legal documents → better at legal reasoning

Fine-tuning is much cheaper than pre-training. Pre-training a frontier foundation model costs well over $100 million — OpenAI's own CEO has said GPT-4 alone cost more than that — and today's frontier runs cost hundreds of millions of dollars or more. Fine-tuning a smaller model for a specific use case might cost hundreds or thousands.

RLHF: Making Models Helpful and Safe

The raw output of a pre-trained model isn't always useful or safe. It might complete any text it's given — including harmful requests — because it was trained to predict the next token, not to be helpful.

RLHF (Reinforcement Learning from Human Feedback) is the technique used to align models with human preferences:

1. Generate many responses to prompts
2. Human raters rank the responses (which is better?)
3. Train a reward model that predicts human preference scores
4. Use reinforcement learning to fine-tune the base model to produce higher-scoring responses

This is how ChatGPT, Claude, and Gemini became conversational assistants rather than raw text completers. RLHF (and variants like DPO) is why they refuse harmful requests, stay on topic, and respond helpfully.

Reasoning Models

Alignment isn't the only thing post-training is used for anymore. Many models are now also trained with reinforcement learning with verifiable rewards (RLVR) — practising on tasks like maths and code where an answer can be checked automatically, rather than relying only on human raters' judgement.

This produces reasoning models, which add a third stage on top of pre-training and post-training:

1. Pre-training: learn general language, facts, and patterns from huge amounts of data
2. Post-training: aligned with human preferences (RLHF) and sharpened on checkable tasks (RLVR), so the model gets measurably better at maths, code and logic — not just more pleasant to talk to
3. Test-time reasoning: before answering, the model generates its own internal reasoning — effectively "thinking" for longer on harder problems before producing a final response

This is different from asking a model to "think step by step" in your prompt (chain-of-thought prompting): test-time reasoning is built into how the model generates its answer, not something you have to request. Many reasoning models also let you set a reasoning effort — low for quick, cheap answers, high for harder problems where it's worth spending more compute to think for longer.

As of October 2026: reasoning models with adjustable effort are now the default way frontier labs ship their most capable models, rather than a separate, experimental category.

The Modern AI Stack

1. Pre-training: massive compute, massive data, general capabilities
2. Post-training: RLHF for alignment and helpfulness, plus RLVR for verifiable reasoning skills
3. Test-time reasoning: adjustable "thinking" effort at inference time, on harder problems
4. Deployment: API, app, or local — used by millions