What is a Dataset?

What is a Dataset?

If machine learning is a recipe, data is the ingredient. The quality and quantity of your training data determines, more than anything else, how good your model will be.

A dataset is a structured collection of examples used to train, validate, and test a model. Every AI system you've ever used was built on one.

Quality vs Quantity

More data generally helps — but only if it's good data.

What makes data high quality?

• Accurate: labels and values are correct
• Representative: covers the range of situations the model will face in the real world
• Diverse: includes edge cases, minorities, rare events — not just the common cases
• Clean: minimal errors, duplicates, and corrupted entries

Low-quality data produces models that are confidently wrong. This is where the phrase "garbage in, garbage out" comes from.

Training a frontier model requires tens of trillions of tokens of text — Meta has disclosed that Llama 3 was trained on about 15 trillion tokens. But it wasn't just any text — the data was filtered, deduplicated, and curated extensively.

The Problem of Bias

Bias in AI comes primarily from bias in data.

If you train a hiring tool on historical hiring decisions, and those decisions historically favoured certain demographics, your model learns to replicate those patterns. The model isn't "trying" to discriminate — it's doing exactly what it was trained to do.

Real examples:

• Amazon scrapped a CV-screening AI in 2018 after finding it penalised CVs that included the word "women's"
• Facial recognition systems trained mostly on lighter-skinned faces perform significantly worse on darker-skinned faces
• Medical AI trained on data from Western populations can perform poorly in other populations

Understanding data bias is essential for anyone who builds or deploys AI systems — and useful for anyone who uses them.

Train, Validation, and Test Splits

A dataset is usually split into three parts:

Training set (~70–80%): The data the model learns from. It never sees the validation or test sets during training.

Validation set (~10–15%): Used during training to check how the model is doing on data it hasn't trained on. Helps tune the model's settings (hyperparameters) and catch overfitting.

Test set (~10–15%): Held back entirely until the end. Gives a final, unbiased measure of how well the model will perform on real-world data.

Why this matters: If you evaluated a model on the same data it trained on, it would look brilliant — it's just memorising. The test set reveals whether it actually learned or just remembered.

This is why benchmark results matter: they reflect performance on held-out data, not training data.