What is a Neural Network?

What is a Neural Network?

A neural network is the mathematical architecture behind most modern AI. The name is inspired by the human brain — but the analogy is loose. Understanding what a neural network actually is (rather than what it's named after) is essential for understanding how AI works.

A neural network is a function. It takes some input, applies layers of mathematical operations, and produces an output.

Neurons, Layers, and Connections

A neural network is made of neurons arranged in layers:

• Input layer: receives the raw data (e.g., pixel values of an image, or numbers representing words)
• Hidden layers: do the processing — there can be just a few or thousands of these
• Output layer: produces the result (e.g., "cat" / "not cat", or a predicted price)

Each neuron takes a weighted sum of its inputs, applies a mathematical function, and passes the result forward to the next layer. "Weights" are just numbers — they control how strongly each connection influences the next neuron.

During training, those weights are adjusted millions of times until the network's outputs match the expected answers.

Why "Deep" Learning?

The word deep simply refers to the number of hidden layers. A shallow network might have one or two. A deep network might have dozens or hundreds.

More layers allow the network to learn increasingly abstract representations:

• Early layers might detect edges and basic shapes in an image
• Middle layers might detect features like eyes, noses, and ears
• Later layers might recognise whole faces

This hierarchical feature learning is why deep networks are so powerful for complex tasks. They don't need to be told what features to look for — they learn them from data.

What Makes a Neural Network "Learn"?

When you train a neural network:

1. An input passes through the network → produces an output
2. The output is compared to the correct answer → the difference is the loss
3. Backpropagation calculates how much each weight contributed to the error
4. Weights are nudged slightly in the direction that reduces the error (gradient descent)
5. Repeat billions of times across millions of examples

After training, the weights encode everything the network has learned. A trained model is essentially a very large collection of carefully tuned numbers.

How Big Are Real Networks?

• GPT-2 (2019): 1.5 billion parameters
• GPT-3 (2020): 175 billion parameters

After GPT-3, frontier labs largely stopped disclosing parameter counts — model size became a competitive secret rather than a published fact.

Each "parameter" is one of those tunable weight values. More parameters → more capacity to learn complex patterns — but also more data and compute required to train.

Mixture of Experts

Many modern models are built as a mixture of experts (MoE) rather than one single dense network. The model is split into many smaller "expert" sub-networks, and a router sends each token to only a handful of them.

This creates two different numbers for the same model:

• Total parameters — the full size of every expert combined (what's stored)
• Active parameters — how many are actually used to process a given token (what's computed)

An MoE model can have far more total parameters than a dense model of similar running cost, because most of the network sits idle for any single token.

As of October 2026: labs no longer publish exact parameter counts for their flagship models, and mixture-of-experts architectures (with total parameter counts well into the trillions, but much smaller active counts) are common across both closed and open models.