RecruitmentAI.org

Home › Explainers

Explainer 7 min read

AI bias in hiring: where it comes from, what the research shows and how employers measure it

In short. AI bias in hiring is unequal treatment of candidates by screening, ranking or interview software because of a protected trait or a proxy for one. It usually comes from historical hiring data, proxy variables and associations inside language models. Employers detect it by comparing selection rates between groups, commonly with the four fifths rule, and reduce it with regular audits, job based rubrics and human review of rejections.

AI bias in hiring is what happens when a screening, ranking or interview tool treats candidates differently because of a protected trait such as race, sex, age or disability, or because of something that stands in for one. The software does not need to see a candidate's race to produce a racial gap. It only needs to learn from data where that gap already exists.

Where the bias comes from

Historical training data. A model trained on past hiring decisions learns the preferences of past hiring managers, good and bad. The best known example is the experimental résumé tool Amazon abandoned after it learned to downgrade applications that mentioned women's organizations, as Reuters reported in 2018.

Proxies. Postcodes, college names, gaps in employment, club memberships and even word choice can correlate with protected traits. Removing the name field rarely removes the signal.

Language models themselves. General purpose models absorb associations from the text they are trained on, and those associations show up when the model is asked to compare candidates.

How the tool is used. A cut off set too high, a knockout question that screens out people with caregiving gaps, or a video tool that struggles with some accents can all create unequal outcomes even when the underlying model is reasonable.

What the research found

Two peer reviewed studies from 2024 tested this directly. In an audit by Kyra Wilson and Aylin Caliskan, presented at the AAAI/ACM Conference on AI, Ethics, and Society, researchers ran more than 500 public résumés against 500 job descriptions across nine occupations using text embedding models of the kind used for retrieval. The models favored résumés with White associated names in 85.1% of tests and résumés with female associated names in only 11.1%, and Black male names were disadvantaged in up to 100% of comparisons.

A separate ACL 2024 study by Haozhe An and colleagues asked language models to write acceptance or rejection emails to applicants whose only difference was their first name. The models were more likely to accept White applicants than Hispanic applicants in many settings, but the pattern shifted from one prompt template to the next. That instability is the practical lesson: you cannot assume a model is fair because it was fair on one test.

How employers measure it

The standard first check in the United States is the four fifths rule from the federal Uniform Guidelines on Employee Selection Procedures. Divide each group's selection rate at a step by the rate of the group selected most often. A ratio below 0.8 is generally treated as evidence of adverse impact and a reason to look harder. If 50% of men pass an AI screen but only 35% of women do, the impact ratio is 0.70, which fails.

Impact ratios are a smoke alarm, not a verdict. Small samples make them noisy, and a passing ratio overall can hide a failing one for a subgroup, such as older women. Good audits look at each step separately, test intersectional groups, and repeat after every model or configuration change.

The rules employers now face

Existing anti discrimination law already applies to AI decisions: an employer is responsible for the outcome whether a person or a vendor's model made the call. On top of that, newer rules add specific duties, from New York City's annual bias audits for automated employment decision tools to Colorado's state law and the EU's treatment of recruitment AI as high risk. Our chapter on bias, regulation and candidate rights summarizes them, and we cover the Colorado AI Act and the EU AI Act in their own explainers.

Reducing bias in practice

  • Ask vendors for their audit. Request the impact ratios by group, the date of the analysis, who ran it and what data it used.
  • Run your own numbers. Your applicant pool differs from the vendor's test set, so compute selection rates on your own data at least once a quarter.
  • Score against the job, not the past. Rubrics built from the skills a role needs are less prone to copying old patterns than models trained on who was hired before.
  • Keep humans on adverse decisions. Let AI sort and summarize, and require a person to confirm every rejection after the first screen.
  • Give candidates a route out. Offer an alternative process for anyone who needs an accommodation, and tell applicants when AI is used.

The bottom line

AI can make hiring more consistent than tired humans reading the hundredth application, but consistency is not the same as fairness. Bias enters through data, proxies and configuration, and it changes when the model changes. Treat measurement as an ongoing job and keep a person accountable for the outcome.

Common questions

What is AI bias in hiring?

It is when AI recruiting software produces different outcomes for candidates based on race, sex, age, disability or another protected trait, directly or through proxies such as postcode or college.

Can removing names and photos stop AI hiring bias?

It helps but rarely solves it, because other fields such as schools, addresses, employment gaps and phrasing can still correlate with protected traits.

What is the four fifths rule?

A screening test from the US Uniform Guidelines on Employee Selection Procedures. If a group's selection rate is less than 80% of the rate for the most selected group, that is generally treated as evidence of adverse impact.

Is the employer or the vendor responsible for biased AI decisions?

Under US anti discrimination law the employer remains responsible for its hiring outcomes, even when a vendor's tool made the recommendation. Contracts can share costs, but not the legal duty.

How often should AI hiring tools be audited?

At least annually where the law requires it, and in practice after every model update or configuration change and on a regular schedule, such as quarterly, using your own applicant data.