Fairness in Machine Learning

In 2016, ProPublica published a groundbreaking investigation revealing that COMPAS, a widely-used algorithmic risk assessment tool for criminal justice, was twice as likely to falsely label Black defendants as high-risk compared to white defendants. This wasn't a bug in the traditional sense—the algorithm was optimized for accuracy. But accuracy alone doesn't capture what we intuitively understand as fairness.

This revelation sparked a profound reckoning in the machine learning community. How do we define fairness mathematically? Can algorithms be both accurate and fair? And when different notions of fairness conflict—as they inevitably do—how do we choose?

These aren't merely technical questions. They represent one of the most significant challenges facing modern AI: translating human values into mathematical constraints.

Machine learning models are increasingly deployed in high-stakes decision-making contexts that directly impact human lives:

• Criminal justice: Risk scoring for bail, sentencing, and parole decisions
• Financial services: Credit scoring, loan approvals, insurance pricing
• Healthcare: Disease prediction, treatment recommendations, resource allocation
• Employment: Resume screening, hiring decisions, performance evaluation
• Education: Admissions, scholarship allocation, dropout prediction
• Social services: Benefit eligibility, child welfare assessments

Unlike traditional software bugs that cause system crashes, fairness failures cause harm to people—often to those already marginalized by systemic inequities. When a credit algorithm discriminates based on race (even implicitly), real families are denied housing. When a hiring algorithm penalizes women, career opportunities evaporate.

The stakes demand rigorous understanding of what fairness means and how to achieve it.

The Fundamental Challenge:

Fairness is not a single, universally-agreed concept. Different stakeholders, contexts, and ethical frameworks lead to different definitions. A definition that seems fair from one perspective may be deeply unfair from another.

Consider a university admissions algorithm:

• One definition: Equal acceptance rates across demographic groups
• Another definition: Equal acceptance rates among equally qualified applicants
• Yet another: Outcomes that lead to proportional representation in the student body

These definitions can't all be satisfied simultaneously. Understanding this landscape of competing definitions is essential for any ML practitioner.

Before diving into mathematical definitions, we must understand the philosophical traditions that inform different fairness concepts. These aren't just academic distinctions—they fundamentally shape what we optimize for.

The Three Major Frameworks:

John Rawls and the Veil of Ignorance:

One influential framework comes from philosopher John Rawls. He proposed a thought experiment: imagine designing a society without knowing your position in it (behind a 'veil of ignorance'). What rules would you choose?

Rawls argued we'd choose two principles:

1. Equal Liberty: Maximum freedom compatible with equal freedom for all
2. Difference Principle: Inequalities are only justified if they benefit the least advantaged members of society

In ML terms, Rawls might suggest we should optimize for the worst-off group, not average performance. This leads to minimax fairness approaches that ensure no group is left behind.

Group fairness definitions compare outcomes across different demographic groups. These are the most commonly used fairness criteria in practice because they're measurable from data.

Let's establish formal notation:

• A = Protected attribute (e.g., race, gender)
• X = Other features used for prediction
• Y = True outcome (ground truth label)
• Ŷ = Predicted outcome (model output)
• R = Risk score (for probabilistic predictions)

The Three Fundamental Group Fairness Criteria:

While group fairness focuses on statistical parity across demographic groups, individual fairness addresses a more intuitive principle: similar individuals should be treated similarly, regardless of group membership.

Formal Definition (Dwork et al., 2012):

$$d_Y(M(x_1), M(x_2)) \leq L \cdot d_X(x_1, x_2)$$

Where:

• M is the ML model (mapping individuals to outcomes)
• d_X is a distance metric over individuals in feature space
• d_Y is a distance metric over outcomes
• L is a Lipschitz constant bounding how much outcomes can differ

In words: if two individuals are 'close' in relevant characteristics, their predicted outcomes should also be 'close.'

Example: Loan Approval

Consider two loan applicants:

• Applicant A: Income $50K, debt $10K, credit score 720, age 28, male
• Applicant B: Income $52K, debt $9K, credit score 715, age 29, female

These applicants are nearly identical on legitimate credit factors. Individual fairness demands they receive similar loan decisions—regardless of gender. If the model approves A but rejects B, it violates individual fairness even if group-level statistics look balanced.

Relationship to Group Fairness:

Individual fairness and group fairness are distinct concepts that neither implies the other:

• Individual fairness ⇏ Group fairness: You can treat similar individuals similarly but still have disparate group outcomes if the groups have different distributions of relevant features.
• Group fairness ⇏ Individual fairness: You can achieve equal selection rates across groups while still treating similar individuals very differently (random selection satisfies demographic parity).

One of the most important theoretical results in ML fairness is the impossibility theorem (Chouldechova, 2017; Kleinberg et al., 2016). It proves that certain fairness criteria cannot be satisfied simultaneously when base rates differ between groups.

The Setup:

Consider a binary classifier and two groups with different base rates:

• Group A: 10% positive rate (e.g., 10% recidivism)
• Group B: 30% positive rate (e.g., 30% recidivism)

The impossibility theorem states that you cannot simultaneously achieve:

1. Calibration: P(Y=1|R=r, A=a) = r for all groups
2. Equal FPR: P(Ŷ=1|Y=0, A=0) = P(Ŷ=1|Y=0, A=1)
3. Equal FNR: P(Ŷ=0|Y=1, A=0) = P(Ŷ=0|Y=1, A=1)

unless either:

• The classifier is perfect (no errors), or
• The base rates are equal across groups

Intuition Behind the Impossibility:

Imagine you have a calibrated risk score. For Group A (10% base rate), a score of 30% is already 3x their average—these are relatively high-risk individuals within their group. For Group B (30% base rate), a score of 30% is exactly average.

Now, if you set a threshold and predict positive for everyone above 30%:

• In Group A, you're catching the higher-risk individuals
• In Group B, you're catching everyone at or above average

These groups will have different error rate profiles because the same threshold means different things relative to each group's distribution.

To equalize error rates, you'd need different thresholds—but then the scores wouldn't be calibrated anymore (a 30% score would mean different things for different groups).

Practical Implications:

Beyond the fundamental trio of demographic parity, equalized odds, and calibration, researchers have proposed dozens of fairness definitions. Understanding this taxonomy helps you select appropriate criteria for specific applications.

Categorization by Statistical Relationship:

Interpretation Guide:

Independence-based (Anti-classification):

• Don't look at protected attribute
• Predictions should be independent of group membership
• Use case: When any group-based disparity is unacceptable (affirmative action contexts)

Separation-based (Error balance):

• Equal errors across groups
• Conditions on true outcome
• Use case: When false predictions have concrete harms (criminal justice, medical diagnosis)

Sufficiency-based (Calibration):

• Predictions mean the same thing for everyone
• Supports individual decision-making
• Use case: When people act on predicted probabilities (risk scores, insurance)

Counterfactual:

• Uses causal reasoning
• Asks 'what if this person belonged to different group?'
• Use case: When you have causal knowledge of how attributes affect outcomes

Most fairness research focuses on binary classification, but many ML applications involve ranking (search results, recommendations) or regression (salary prediction, credit limits). These require extended fairness definitions.

Fairness in Ranking:

For ranked lists (e.g., job candidates, search results), fairness involves exposure and attention allocation:

• Demographic Parity in Exposure: Each group should receive proportional exposure across ranking positions
• Equity of Attention: Users from different groups should have equal probability of appearing in top-k positions
• Position Bias Correction: Account for the fact that positions have different visibility (top results get more attention)

Fairness in Regression:

For continuous outcomes (salary, loan amount, estimated home value), fairness definitions adapt:

• Equal Mean Prediction: E[Ŷ|A=0] = E[Ŷ|A=1]
• Equal Residuals: E[Y-Ŷ|A=0] = E[Y-Ŷ|A=1] (no systematic over/under-prediction)
• Bounded Group Loss: Max group-specific loss is minimized

We've traversed the rich landscape of fairness definitions in machine learning. Let's consolidate the key insights:

What's Next:

Now that we understand the formal definitions of fairness, the next page examines protected attributes—the specific characteristics (race, gender, age, disability) around which fairness concerns arise. We'll explore legal frameworks, proxy discrimination, and the complex questions of which attributes deserve protection and how to handle them in ML systems.