Bias and fairness risks

Bias can enter before, during, and after model use:

• Problem framing: the team predicts “successful employee” without defining success.
• Criteria: historical preferences are mistaken for job requirements.
• Labels: past ratings reflect inconsistent managers or unequal opportunities.
• Data: groups are missing, misclassified, or represented by poor-quality records.
• Features: postcode, school, employment gaps, or language style act as proxies.
• Model behavior: errors differ across groups or unusual career paths.
• Interface: a ranked list encourages reviewers to trust position rather than evidence.
• Deployment: one team uses the output as advice while another treats it as a cutoff.
• Feedback: people selected by the old process generate the data used to justify it.

This chain explains why “remove protected fields” is insufficient. A model can reconstruct correlated information, and the target itself may encode past inequity.

Teams may care about equal access, consistent treatment, job relevance, comparable error rates, accommodation, transparency, or the ability to contest a result. These goals can conflict. Improving one aggregate metric may worsen another or hide intersectional differences.

Write a fairness claim that can be tested. For example: “For applicants who satisfy the published essential criteria, the evidence-extraction assistant should identify relevant passages with comparable recall across approved evaluation groups and application formats.” This is more useful than “the AI is unbiased.” It names the task, population, quality measure, and comparison.

Protected categories and lawful monitoring rules vary by jurisdiction. Work with qualified legal, HR, employee-representative, accessibility, and data specialists. Do not collect sensitive attributes casually. Fairness evaluation may require them, but collection needs a defined purpose, access controls, retention rules, and governance.

Selection rates can reveal disparities, yet they do not show whether the assistant correctly read evidence. Build a labeled evaluation set reviewed by qualified people and test:

• recall of relevant evidence;
• rate of unsupported evidence claims;
• “unclear” or unreadable rates;
• recommendation or rating differences, if recommendations are permitted;
• human override and reversal patterns;
• processing failures by file type, language, assistive technology, or career path;
• candidate complaints, correction requests, and accommodation outcomes.

Disaggregate where lawful and statistically responsible. Broad averages can conceal harms to small or intersecting groups. At the same time, tiny samples produce unstable results. Report uncertainty and avoid declaring victory from noise.

Use counterfactual tests carefully: change a name or demographic cue while holding job evidence constant and observe whether output changes. This can expose sensitivity, but names are imperfect proxies and one template cannot represent a population. Pair such tests with real workflow evaluation.

A predictive model trained to resemble “top performers” may learn attendance, manager access, sales territory, or promotion history rather than capability. Historical outcomes occurred under particular opportunities and management systems. They are not neutral ground truth.

Ask whether the target is valid, measured consistently, and causally connected to the job outcome. A high performance rating may reflect visibility or manager style. Tenure may reward people who could afford to stay. “Accepted offer” can encode differences in salary negotiation or location.

Prefer direct, job-related assessment over broad predictions. If the role requires interpreting a customer request, use a representative work sample with an accessible alternative. Do not infer that capability from social media activity, video expression, or resemblance to past hires.

Humans can reduce or amplify bias. A reviewer who sees a model rank may anchor on it. A reviewer under time pressure may approve every recommendation. A panel may use “holistic judgment” to reintroduce stereotypes after a structured assessment.

Give reviewers criteria, original evidence, uncertainty, and a clear obligation to disagree when warranted. Hide irrelevant model confidence or ranking. Require reasons tied to job evidence, sample decisions for audit, and investigate systematic override patterns. If reviewers always override the model for one group, determine whether the model, the reviewer, or the criteria are failing.

An appeal must be more than rerunning the same system. A qualified person should inspect the underlying data, evidence, and procedure, correct errors, and reverse downstream consequences where possible.

Fairness evaluation is not a launch certificate. Applicant populations, job descriptions, labor markets, model versions, prompts, parsers, and reviewer behavior change. Define triggers for reevaluation: a new role family, a vendor update, a new data source, an unexplained disparity, or a rise in unreadable applications.

Keep versioned records of criteria, prompts, tests, known limitations, incidents, and decisions. Pause the system when evidence suggests material harm or when required data for monitoring is unavailable. “We cannot measure it” is a risk signal, not proof of fairness.