Before you buy AI recruitment software, ask the vendor for three things: evidence that its scores predict performance in jobs like yours, a bias audit you can read and question, and a clear account of who carries legal responsibility when the tool screens people out. Most sales conversations supply none of these in usable form.
Three assumptions make that gap easy to miss. The first is that a product sold for hiring has already been tested the way a psychological test is tested. The second is that a published bias audit shows a tool is fair. The third is that legal risk sits with the vendor, because the vendor built the model. Each assumption is understandable, and each one fails when you look at the research. Validation, in the sense psychologists use the word, means gathering evidence that a score measures something relevant to the job and predicts something that matters. A bias audit, at least as most laws define it, compares selection rates between groups and says nothing about whether the tool works. And regulators on both sides of the Atlantic place duties on the employer who uses the tool, not only on the company that sells it.
This article sets out what the evidence says about each assumption and what a careful buyer should ask for. It does not try to settle whether software chooses better employees than people do. That is a separate question. The focus here is due diligence: what to demand, what to distrust, and where your exposure lies.
What Do Buyers Assume About AI Recruitment Software?
The pitch for recruitment software built on artificial intelligence rests on speed and consistency. A system can read thousands of applications overnight, score every candidate against the same criteria, and promise to remove the human biases that creep into a tired recruiter's judgment. Those benefits are real in the narrow sense that machines do not get tired. The trouble lies in what buyers infer from them.
A 2022 systematic review of 51 articles on the ethics of recruiting with artificial intelligence found the literature split between two pictures. In one, software applies the same skill focused criteria to everyone, writes more inclusive job advertisements and opens doors for candidates without elite credentials. In the other, bias flows in through training data and through ordinary looking variables that stand in for protected characteristics. A zip code, for example, can carry information about race. The review also found that only limited empirical research existed. Most of what had been written was practitioner commentary and conceptual argument. In plain terms, the claims had run ahead of the testing.
An earlier systematic review of 36 journal articles published between 2014 and 2020 reached a similar view. Tools trained on past hiring decisions tend to inherit the patterns in those decisions, even when sex or ethnicity are removed from the data, because other variables carry the same information. Years of work experience, for instance, track age closely. Most of the reviewed articles, again, did not test anything directly.
So the gap is this. Buyers assume that a product on the market has passed some independent bar of scientific evidence. In most places no such bar exists before sale. The burden of asking falls on the employer.
What Validation Evidence Should a Vendor Be Able to Show You?
Industrial and organizational psychologists settled long ago what counts as evidence for a selection tool. The professional principles for validation, published in their fifth edition in 2018 and approved as an official statement of the American Psychological Association, set out the expectations. A tool should rest on an analysis of the job, so that what it measures is linked to what the work requires. Its scores should be reliable, meaning a person would get roughly the same score if assessed again under similar conditions. And there should be evidence that scores relate to outcomes the employer cares about, such as rated performance or success in training, in settings similar to the one where the tool will be used.
Nothing about artificial intelligence changes these expectations. Three senior selection researchers argued exactly this in a 2021 call to action. Looking at online games, video interviews and models built from resumes and social media data, they listed the open questions. Do these tools measure anything tied to the job? Are they reliable and stable over time? Do they predict outcomes that matter? Are their validity evidence and comparison data kept current? Is what vendors tell employers and candidates about the tools accurate? Their conclusion was that existing testing standards should simply be extended to these products, including the expectation that developers record how a tool works and what evidence supports it.
The point about keeping evidence current matters for a buyer because many models are retrained as new data arrive. A validation study run three years ago on an earlier version of the model tells you little about the version you will license. Ask when the model was last changed and whether the evidence was refreshed after the change.
Ask, too, what the scores were validated against. A tool can predict something consistently and still predict the wrong thing. The 2020 review described above showed how a model trained to reproduce which applicants past recruiters advanced will learn to copy those recruiters, including their blind spots, rather than learn who performs well. The outcome a model is trained to predict is a design choice, and it is often the first place where bias enters. A vendor who cannot tell you precisely what outcome its model predicts, and where that outcome data came from, has not done the basic work.
Does a Consistent Score Mean an Accurate One?
Reliability is the floor of validity, not the ceiling. If a score jumps around when nothing relevant about the person has changed, it cannot be measuring a stable trait. That simple logic gives buyers a test they can run themselves, and one research team showed how revealing it can be.
An external stability audit published in 2022 examined two commercial systems that infer personality traits from resumes and online professional profiles. The researchers fed the systems material from 94 graduate students and then changed things that should not matter. For one system, merely switching the resume's file format moved scores further than accepted testing standards allow. Adding a link to the applicant's professional networking profile inside the resume shifted that system's scores a great deal. When the same person was scored from a resume and from an online profile, both systems produced results that agreed poorly. Scored again months later, the same people received noticeably different personality profiles from both systems. One system could not reproduce its own scores exactly even when given identical input shortly afterward.
The authors concluded that neither system could be treated as a valid instrument for high stakes hiring decisions. The sample was small and drawn from technical graduate students, so the findings may not extend to every applicant pool. The method is the lesson. Stability testing is cheap, and it does not require access to the vendor's code.
Some failures are more basic still. An analysis of functionality presented at a leading computer science conference on fairness in 2022 argued that deployed systems built on artificial intelligence often simply do not work, and that debates about ethics tend to skip the prior question of whether a product functions at all. Its case studies describe systems built carelessly, deployed without checks and promoted with claims that outrun the evidence. The authors treat functionality as the first protection for the people a system judges: a tool that does not work cannot be fair, however its selection rates look.
A 2022 perspective later published in a data science journal applied the same thinking to hiring. It proposed that audits examine the assumptions a hiring tool is built on, not only its outputs. If a product claims to read job suitability from facial movement or tone of voice, the first question is whether that link has any scientific basis. The authors warned that some products rest on outdated and essentialist ideas about human nature and ability, ideas that treat a trait as fixed and readable from surface signals. An audit that only checks whether the tool selects men and women at similar rates can pass a product whose underlying theory of human ability is unfounded.
What Does a Bias Audit Actually Prove?
Most bias audits of hiring tools rest on one number: the impact ratio. You take the share of each group that the tool advances, divide it by the share advanced from the most favored group, and check whether the result falls below 0.8. That threshold comes from federal selection guidelines adopted in the United States in 1978. They say a selection rate below four fifths of the highest group's rate will generally be treated by enforcement agencies as evidence of adverse impact. Adverse impact means a practice screens out one group at a noticeably higher rate than another, whatever anyone intended.
The same guidelines contain a caveat that audits often drop. Smaller differences can still count as adverse impact when they are meaningful both statistically and in practical terms. Larger differences can be discounted when the numbers involved are too small to trust. The four fifths rule was written as a rule of thumb for agencies deciding where to look, not as a definition of fairness.
A 2024 conference paper took this point further. It argued that the field of algorithmic fairness lifted the four fifths rule out of its legal setting, turned one part of discrimination law into a general fairness metric, and then wrote that metric into widely used software toolkits. In doing so, the authors argued, the field stripped away the context that gave the rule its meaning and created fresh ways to cause harm. A tool can clear 0.8 and still disadvantage a group in ways that matter, particularly a small group whose numbers are too thin to register in the ratio.
The deeper problem is that an impact ratio describes outcomes. It says nothing about causes and nothing about accuracy. A 2023 framework published in the flagship journal of the American Psychological Association set out what a full audit of a system that makes predictions about people should cover. It listed 12 components grouped into three categories. The first category covers the model itself: its source data, design, development, the features it uses, its processes and its outputs. The second covers how information about the model is presented and understood by the employer using it, by the people affected by its decisions and by outside observers. The third covers matters that cut across everything else, including cultural context, respect for persons and whether the research designs behind the developer's claims hold up. A comparison of selection rates touches only a small corner of that framework.
A field scan of auditors presented at the same fairness conference in 2022 explains why buyers cannot take the word "audited" at face value. The researchers catalogued 438 individuals and 189 organizations doing algorithmic audits, surveyed 152 of them and interviewed 10 industry leaders. Their central finding was that without shared standards or regulatory guidance, a claim that a system has been audited is hard to verify and may hide harm rather than reduce it. Their recommendations included independent audits against clearly defined standards, public disclosure of key findings, involvement of the people most likely to be harmed, and formal evaluation, possibly accreditation, of the auditors themselves.
Did New York City's Bias Audit Law Work?
New York City offers the clearest test of what happens when a bias audit becomes a legal requirement. Under Local Law 144, which the city began enforcing on July 5, 2023, an employer may use an automated employment decision tool only if the tool has had a bias audit within the past year and a summary of the results is publicly available. Candidates must also receive notice, which city guidance sets at 10 business days before the tool is used.
The early evidence is sobering. One field study sent student investigators to check 391 employers. They found published audits for just 18 of them, about 5%, and transparency notices for 13, about 3%. After removing duplicates from employers that used the same vendor, 13 distinct audits remained, which together reported 386 impact ratios. Only 9 of those ratios fell below 0.8.
Why so few audits, and why such clean results? The researchers pointed to the discretion the law grants. Employers decide for themselves whether a tool plays a large enough part in their decisions to be covered. An employer whose tool showed wide gaps could simply conclude the law did not apply and publish nothing. The authors called the result null compliance: a missing audit cannot be read as a violation, because the employer may have judged that it was not required to produce one. The study also found that 20% of impact ratios were left out because the groups concerned made up less than 2% of the data, an omission the city's rules allow, and that most of the omitted groups were Indigenous applicants. For job seekers, the disclosures were hard to find even after investigators clicked through an average of 17 to 19 web pages, and none of the audit reports explained their results well enough to help anyone make a decision.
A companion interview study, based on 16 interviews with auditors and practitioners, found only 19 public audits and described why the regime produced so little. The law sets no minimum impact ratio and requires no remedy when a gap is found, so its force depends on reputation. Auditors disagreed about what independence means, especially when the same firm offered both advice and audits to a client. Vendors cited trade secrets and confidentiality agreements to restrict auditors' access to data, and some clients had no reliable demographic data on their applicants at all.
Then came an official verdict. In December 2025 the New York State Comptroller published an audit of enforcement covering July 2023 to June 2025. The city agency responsible received only 2 complaints in that period. When it reviewed 32 companies, it identified 1 instance of noncompliance. The state auditors, looking at the same 32 companies, found at least 17 instances of potential noncompliance. The city had not consulted its own technology office when making judgments about these tools, even though its staff lacked the technical expertise to evaluate them. Penalties under the law run from $500 to $1,500 per day of violation.
The lesson for a buyer is not that bias audits are worthless. It is that a published audit tells you a vendor or employer met a narrow legal test, under its own reading of whether the law applied. It does not tell you that the tool is valid, that small groups were checked, or that anyone looked at whether the tool works.
How Does European Law Treat AI Recruitment Software?
The European Union took a different route. Rather than a single audit requirement, its Artificial Intelligence Act, adopted on June 13, 2024, classes systems used for recruitment and selection as high risk, along with systems used for decisions on promotion, termination, task allocation and the monitoring of workers. The regulation's reasoning is that such systems can shape careers and livelihoods and may carry forward historical discrimination against women, certain age groups, people with disabilities, and people of particular racial or ethnic origins or sexual orientations.
High risk status brings duties for the developer and for the employer who uses the system. Under the regulation, an employer deploying such a system must use it according to the provider's instructions, monitor how it performs, keep records, and assign oversight to staff with adequate competence and authority. Employers must inform workers and their representatives before putting such a system into use at work. People affected by decisions must be told that a high risk system was involved and of their right to an explanation. The regulation also permits the processing of sensitive personal data, under safeguards, for the specific purpose of detecting and correcting bias. That provision answers one of the practical obstacles the New York auditors described, since an audit cannot compare groups without knowing who belongs to which group.
The same regulation goes further for one category of product. It prohibits systems that infer the emotions of people in the workplace. Its stated reasons are serious doubts about the scientific basis of reading emotion from expression, the wide variation across cultures and individuals, and the imbalance of power between employer and worker. A vendor offering emotion detection in interviews to employers in the European Union is offering a use the regulation forbids.
Timing has shifted. A 2026 amending regulation moved the date from which the main high risk obligations apply to systems such as recruitment tools, from August 2, 2026 to December 2, 2027. The delay buys time. It does not change what will be required.
European discrimination law also works differently from the United States model. A legal analysis later published in a peer reviewed law and technology journal argued that European courts judge discrimination case by case, relying on context and judicial interpretation, so no single statistic such as an impact ratio can settle whether a tool is lawful. The authors proposed a particular statistical measure as a common starting point for evidence rather than as a pass mark. They also observed that discrimination by an algorithm is harder for victims to notice than discrimination by a person, which weakens the usual route by which unfair treatment comes to light.
That matters for anyone buying software built for another market. A 2020 conference study of three automated hiring systems used in the United Kingdom found that vendors' claims to detect and remove bias were rarely scrutinized, and that the few existing evaluations had been made almost entirely from a United States legal perspective. The authors concluded that such systems might obscure rather than reduce systemic discrimination when used under British and European law, and that bias correction meeting European legal standards would be hard to build.
Can a Tool Be Both Fair and Accurate?
Vendors often present fairness and accuracy as a trade the buyer must accept. The better question is whether the vendor has tested that claim at all.
A buyer is entitled to ask a vendor what it did to reduce group differences in scores and what, if anything, it gave up in prediction to do so. A vendor that cannot answer that question has probably never looked.
One legal analysis of discrimination and algorithms makes a related argument about transparency. Human judgment is opaque: a manager cannot fully explain why one applicant felt like a better fit. A well built algorithm, by contrast, can be interrogated. An auditor can change one input and see whether the decision changes. Algorithms also force their builders to state their priorities in numbers, such as how much predictive accuracy they will trade for fairer outcomes. But those advantages exist only if the training data, the outcome being predicted and the choice of predictors are stored and open to inspection, because bias can enter at each of those points. Without that record, an algorithm is as opaque as a person and operates at far greater scale.
The broader research picture remains unsettled. A multidisciplinary survey published in 2025 in a computing journal reviewed systems, sources of bias, fairness measures, methods for reducing bias, data sets and legal rules across the whole recruitment process. It concluded that whether, and which kinds of, algorithmic hiring can be less biased than the simpler methods they replace remains an open question. The literature, the authors observed, is pulled between two stories: one in which algorithms rescue hiring from biased recruiters and one in which they automate discrimination. Neither story is supported well enough to justify a purchase on its own.
What Ethical Questions Remain Once the Law Is Satisfied?
Legal compliance is a floor. The ethics review discussed earlier identified several concerns that no audit statute currently addresses.
The first is homogenization. When many employers license the same model, a candidate who scores poorly may score poorly everywhere at once. A single biased interviewer affects one employer. A single biased model could close a whole sector to someone. The same review flagged the related risk that highly qualified applicants are screened out because they do not resemble past hires.
The second concerns what employers and applicants are told. The three researchers behind the 2021 call to action treated the accuracy of information given to candidates and employers as an ethical requirement, not a courtesy. If a vendor's materials describe a game as measuring problem solving, the evidence should show that it does. Marketing language that outruns the validation evidence is an ethical problem before it is a legal one.
The third is accountability when something goes wrong. The interview study of New York auditors found that the law had created a market for consulting on responsible use of artificial intelligence that, for some firms, was more lucrative than the audits themselves. That is not necessarily bad, but it means the firm advising a vendor on how to pass may also be the firm certifying that it passed. Ask who paid for the audit and what other services the auditor sells to the same client.
The ethics review also found the evidence base thin. Of the 51 articles it covered, roughly 22% fell into the descriptive category that included experiments and case studies. The rest were practitioner commentary, technical proposals, legal analysis or theory. Employers making decisions about thousands of applicants are relying on a body of evidence that is still small, and they should buy accordingly.
What Does This Mean for You When You Sit Across From a Vendor?
If you are the person signing the contract, the research points to a short list of things to insist on. Ask for the validation study, not a summary of it. Check that it was run on jobs similar to yours, on a recent version of the model, and against a measure of job performance rather than past recruiter decisions. If the vendor cannot produce such a study, you are being asked to run the validation yourself, on your own applicants, without telling them.
Ask for the bias audit and read it with the New York findings in mind. Which groups were left out because their numbers were small? Was the audit run on an applicant pool like yours or on the vendor's pooled data from other clients? Who paid the auditor, and what else does the auditor sell? An impact ratio above 0.8 is a starting point for questions, not the end of them.
Run a stability check before you commit. Submit the same resume in two file formats, score the same volunteers twice a few weeks apart, and see whether the results hold. If they do not, nothing else the vendor tells you about validity can be true.
Read the contract for who carries responsibility. In New York City the duty to have an audit and give notice falls on the employer. In the European Union the employer that deploys a high risk system carries its own obligations for oversight, monitoring, record keeping and informing workers. An indemnity clause can move money after the fact, but it does not move those duties.
And if the product claims to read emotion, honesty or character from a face or a voice, ask what scientific theory connects that signal to job performance. For emotion, the European Union has already decided that the answer is not good enough.
Key Takeaways
- A product being on sale proves nothing about its validation. Ask for the full study and confirm it covers jobs like yours and the version of the model you will actually use.
- Bias audits built on the four fifths rule test one narrow question about selection rates. They do not show that a tool measures anything useful or that small groups were protected.
- New York City's audit law produced few published reports, disclosures that job seekers struggled to find and weak enforcement, so a posted audit is a minimum rather than a guarantee.
- European rules treat recruitment systems as high risk, put duties on the employer that uses them, ban emotion inference at work, and now apply the main obligations from December 2, 2027.
- Cheap stability checks, such as changing a resume's file format or rescoring the same people later, can expose unreliable scoring without any access to the vendor's code.
- A well documented algorithm can be questioned in ways a manager's judgment cannot, so ask any vendor what it did about fairness and what that cost in accuracy.
Implications for Practice
Start before the sales meeting. Write your evidence requirements into the request for proposal so that every vendor answers the same questions in writing. Following the professional principles for validation and the 2021 call to action, ask each vendor to show the job analysis that links its measures to your roles, evidence that scores are reliable, evidence that scores predict job performance in comparable jobs, the date of that evidence, and the version of the model it refers to. Ask what outcome the model was trained to predict and where that outcome data came from. Vendors who answer with marketing claims instead of studies have told you what you need to know.
Build a trial period into the purchase. Run the tool in parallel with your existing process for at least one full hiring cycle, recording its scores without using them to make decisions. During that period, run the stability checks the external audit research used: the same resume in different formats, the same volunteers scored on two occasions, the same person scored from different sources. At the end of the trial, compare selection rates across groups yourself, and remember the federal guidelines' warning that small numbers can hide or exaggerate differences. Where a group is small, pool data across several hiring cycles before drawing conclusions rather than leaving the group out.
Commission audits that go beyond the impact ratio. Use the 12 component framework from the 2023 audit research as a checklist, covering the source data, the features the model relies on, how its outputs are explained to the people affected, and whether the vendor's own research designs stand up. Choose an auditor who is paid by you rather than by the vendor and who does not also sell the vendor advice on passing. Ask the auditor to report results for every group, including the small groups that New York City's rules allowed audits to omit. If an audit is published, make sure it sits somewhere a candidate could find it in a few clicks.
Map your legal exposure by where your candidates are, not where the vendor is based. If you hire for roles in New York City, put the annual audit, the public summary and the advance notice to candidates into your hiring calendar, and do not rely on a narrow reading of whether your tool is covered, given the state comptroller's findings on how little enforcement had happened and how much potential noncompliance it uncovered. If you hire in the European Union, prepare now for the high risk obligations due from December 2, 2027: appoint a specific person with the competence and authority to oversee the system and override it, keep the system's records, inform workers and their representatives before use, and prepare plain explanations for candidates who ask how a decision was reached. Check that no tool you license infers emotion. Wherever you hire, the outcome of a screening process still has to answer to local equal employment law, whatever software produced it.
Write the evidence into the contract. Require the vendor to notify you before any material change to the model and to provide fresh validation and bias evidence after such a change. Secure access to the data an independent auditor would need, since the New York interview study found that vendors' confidentiality claims were a main obstacle to meaningful audits. Keep the right to suspend use of the tool if monitoring shows a problem, and keep records of every case where a recruiter overrode the tool, so that you can see whether human oversight is real or ceremonial.
Finally, treat the purchase as the start of an evaluation rather than its end. Once a year, compare the tool's scores with the actual performance and retention of the people you hired, review selection rates by group, and decide on evidence whether to keep the tool. The research base on these products is still thin. Your own data, gathered carefully, may become the most relevant validation study you have.
