Bias and Fairness

Bias and Fairness

AI systems can be biased — producing outputs that systematically disadvantage certain groups. Understanding how bias arises, where it harms, and how to mitigate it isn't just an ethical nicety. It's a practical requirement for building systems that work reliably and fairly for everyone.

Sources of Bias

Training data bias: if training data reflects historical inequalities or underrepresents certain groups, the model will too. A hiring model trained on past successful employees will perpetuate the demographic profile of past hires.

Measurement bias: what you measure as "good outcome" shapes what the model optimises for. If "successful loan repayment" is your target, and repayment rates vary by group for reasons unrelated to creditworthiness (e.g., access to emergency funds), the model learns proxies for those group differences.

Representation bias: if group A appears rarely in training data, the model will perform worse on group A. Facial recognition systems famously performed worse on darker skin tones partly because training datasets overrepresented lighter skin.

Feedback loops: biased decisions create biased outcomes that feed back into future training data, amplifying the original bias.

Where Bias Causes Harm

Bias in AI causes serious harm when AI is used for consequential decisions:

Hiring: resume screening tools have been found to downgrade women's CVs (Amazon scrapped their tool in 2018 after this finding).

Criminal justice: COMPAS recidivism prediction scores were found to produce racially disparate false-positive rates.

Healthcare: clinical decision support tools trained on data that underrepresents certain populations perform worse for those populations.

Credit and lending: models can produce disparate outcomes for protected classes even without using protected attributes directly, through correlated proxies.

Content moderation: automated moderation can perform differently across languages, dialects, and communities.

Fairness is Technically Hard

There's no single definition of fairness, and some definitions are mathematically incompatible with each other:

• Equal accuracy: the model is equally accurate for all groups
• Equal false-positive rates: false positives are equally likely across groups
• Calibration: predicted probabilities mean the same thing for all groups
• Individual fairness: similar individuals are treated similarly

The ProPublica/Northpointe debate over COMPAS famously showed that two respected researchers, using different fairness definitions, reached opposite conclusions about whether the same system was fair.

Practical implications: be explicit about which fairness criterion you're optimising for and why. Audit your system's performance disaggregated by relevant demographic groups. Don't deploy high-stakes automated decisions without human review of edge cases.