Algorithmic Bias

Algorithmic bias is systematically unfair model behavior toward specific groups, and it's almost never caused by a biased line of code — it's overwhelmingly a property of the training data and the objective being optimized, inherited and sometimes amplified by a model that is, mathematically, doing exactly what it was trained to do. A model trained to minimize average error across a dataset will happily achieve low average error while performing far worse on an underrepresented subgroup, because nothing in that objective ever asked it to treat subgroup performance as a constraint. At a high level, bias enters and compounds through a pipeline like this:

COMPAS, a real recidivism-risk tool used in US courts, is the case study that made fairness metrics a hard mathematical problem rather than a vague ethical one. ProPublica's 2016 analysis found the tool was roughly equally accurate for Black and white defendants overall, but Black defendants who did not reoffend were nearly twice as likely to be incorrectly flagged as high-risk compared to white defendants in the same situation. The tool's maker countered that COMPAS satisfied a different, equally legitimate fairness criterion (calibration: among people scored at the same risk level, the actual reoffense rate was similar across races) — both sides were right about their own metric, which is exactly the problem.

Bias can be amplified, not just inherited, which is a distinct and often more dangerous failure mode. A 2017 study found image-captioning and object-recognition models trained on datasets where, say, 67% of cooking-related images featured women would predict "woman" for cooking scenes at rates well above 67% — the model's learned shortcut (cooking correlates with gender in training data, so lean harder into that correlation to minimize loss) overshoots the original data's actual skew. This matters because fixing amplification requires more than just collecting more balanced data; it requires understanding why gradient descent found the amplified shortcut to be loss-minimizing in the first place, since a model will re-amplify a correlation from even moderately imbalanced data if that correlation happens to be predictively useful.

Technical mitigation operates at three different points in the pipeline, and each has a different cost and a different limitation. Pre-processing reweights or resamples training data before training even starts (cheap, but can't fix bias introduced by the model architecture or objective itself); in-processing adds a fairness constraint directly into the training objective, trading some accuracy for a chosen fairness metric (more effective, but requires picking which fairness definition to optimize for, with the impossibility result above meaning that choice always excludes other valid definitions); post-processing adjusts a trained model's output thresholds per group after the fact (fast to deploy, but treats the symptom at inference time rather than fixing what the model actually learned).

Read algorithmic bias not as a bug to be patched out but as a direct, often mathematically provable consequence of what a model was asked to optimize for and what data it saw while doing so: a model minimizing average loss on biased historical data has no mechanism to know that's a problem unless fairness is explicitly written into the objective, and even then, the choice of which fairness definition to enforce is an unavoidable policy tradeoff, not a solved technical question.