AI recruitment tools are not one technology, and the evidence for them is not one verdict. Some tools, such as software that scores written answers against expert ratings, have solid research behind them. Others, such as personality readings drawn from video or resumes, rest on thin or troubling evidence. Judge each tool on its own record.
Three assumptions shape most conversations about software in hiring. The first is that artificial intelligence in recruitment is a single thing that either works or does not. The second is that a machine is neutral because it has no feelings about anyone. The third is that when a supplier reports high accuracy, the tool must be good at finding people who will do the job well.
Each assumption breaks down once you look at the studies. A chatbot that books interviews, a game that measures reasoning, a model that scores an interview transcript and an advertising system that decides who sees a vacancy share very little except the label. They use different data, they make different decisions and they fail in different ways. Machines can carry bias without anyone intending it, sometimes because of simple economics, as the research on job advertisements shows. And the word "accuracy" in this field usually means that the machine agrees with a human rater, not that the people it favors go on to perform better at work. Those are very different claims.
This article takes the tools one at a time, in the order a candidate meets them, and asks what the peer reviewed evidence actually shows. Where the evidence is strong, it says so. Where it comes from a single study, a student sample or a working paper, it says that too.
Why Do Most Buyers Treat AI Recruitment Tools as One Product?
Organizations often decide to "use artificial intelligence in hiring" as if they were choosing one product. In practice the label covers at least six different jobs spread across the hiring funnel. Some systems decide who sees a job advertisement. Some search databases of past applicants and suggest who a recruiter should contact. Chatbots answer questions, collect details and book interviews. Games measure abilities or traits through play. Video and voice systems run interviews without a human present. Scoring models read written answers or interview transcripts and assign marks.
Each of these sits at a different gate, and each gate has its own risks. A tool at the top of the funnel, such as advertisement delivery, decides who even enters your applicant pool. Its errors are almost invisible, because you never see the candidates you failed to reach. A tool at the scoring stage makes errors you can, at least in principle, check, because you hold the answers it scored.
There is a second gap that matters even more. Most published work on recruitment software tests whether a machine can reproduce a human judgment. Can it score an essay the way an expert would? Can it rate an interview the way an interviewer would? That is a useful first step, but it is a low bar. A machine that perfectly copies a human rater also copies that rater's blind spots, and it tells you nothing about whether the people it rates highly will do the job well. Researchers call that second question validity, which in plain words means whether a score predicts what it claims to predict. Reliability, a word you will meet often below, means whether a tool gives the same answer when it measures the same thing twice.
When you keep these two gaps in mind, the evidence stops looking like a single thumbs up or thumbs down. It becomes a map with firm ground in some places and swamp in others.
Do Recruitment Chatbots Do More Than Answer Questions?
A recruitment chatbot is a program that holds a typed or spoken conversation with job seekers. Most answer questions about the role, collect application details and book interviews. Some go further and ask screening questions or try to read a candidate's personality from the conversation.
The best evidence on how recruiters actually experience these bots comes from an interview study published in 2022, based on conversations with 11 recruiters who used or planned to use chatbots and two people who built them. Recruiters said the bots helped them attract applicants, including people who were not actively looking, and made applying easier on a mobile phone, where many visitors browse but few finish a long form. The same study found a less advertised side. Bots produced larger pools of applicants to review, and short bot conversations often produced thinner applications, so recruiters had to chase missing details. Recruiters took on new work writing and testing conversation scripts, which one estimated at 15 to 60 minutes per bot. Candidates sometimes shared personal information with the bot that the employer had not asked for, and bots often sat outside the applicant tracking system, creating separate workflows. The study is small and qualitative, so it describes experiences rather than measuring outcomes, but it is a useful corrective to the idea that a chatbot simply saves time.
Some chatbots claim to infer personality from what candidates type. A 2023 psychometric study tested this directly with 1,444 university students who completed a 20 to 30 minute conversation with a chatbot and also filled in a standard personality questionnaire. The machine's personality scores were reasonably consistent and agreed moderately with the questionnaire. Two problems stood out. The machine's scores for different traits moved together more than they should, so it struggled to tell traits apart. And the machine scores did a poor job of predicting the students' grades and how well their peers judged them to be settling into college. Because the sample was students rather than job applicants, the findings may not carry over to hiring, but the pattern is a warning. A chatbot can produce numbers that look like personality scores without capturing the separate traits a personality test is meant to measure.
Can Software Read Personality From a Recorded Video Interview?
A one way video interview, also called an asynchronous video interview, is one where the candidate records answers to preset questions with nobody on the other end. Recruiters can watch the recordings later, or software can score them.
A 2022 research review of these interviews set out a conceptual model and a research agenda for how they are designed and used, a sign that employers adopted the format faster than researchers could study it. The model treats the interview as a set of design decisions, such as how much preparation time candidates get, whether they may rerecord an answer and how long they have to respond, and proposes that these decisions can shape how candidates perform and how they are judged. Much of that model still awaits testing, but the point stands: the design of the interview matters before any algorithm touches the recording.
The most careful study of automated scoring in this format is a 2022 validation study that trained models across several samples and tested them on new people. The models learned in two ways. Some learned to predict what interviewers thought of a candidate's personality. Others learned to predict what candidates said about themselves on a questionnaire. The models that learned from interviewer ratings did better at matching interviewer judgments. The models that learned from self reports gave mixed results. When the models were applied to candidates answering different questions from the ones they were trained on, they held up for some traits and not for others, and their consistency over time varied a great deal from trait to trait. Most participants were students in mock interviews, and the study did not test whether the scores predicted job performance or whether they disadvantaged any group. The plain lesson is that these systems learn to reproduce an observer's impression of a person. That is not the same as measuring the person.
A follow up study looked at how much training data such systems need. This 2024 methods study found that a few hundred scored interviews were enough for research purposes when the goal was to copy interviewer ratings, though employers may want more to make sure the model works across different groups and over time. When the goal was to predict self reported personality, the models showed only a weak relationship with the real scores even at the largest sample sizes tested. The study also found that newer language models, which read words in context, generally did better than older methods that simply count words.
Before a model can score a spoken answer, speech has to be turned into text. In 2025, a transcription study compared human transcription with four commercial speech recognition services across two samples of mock video interviews, involving 1,014 and 414 people. The services made more errors for people who speak English as a second language and for Black interviewees, and their accuracy also differed between men and women. The surprise was that the scoring models built on these flawed transcripts did not produce meaningfully biased final scores. That is good news with a catch. It held in these samples with these models, and it does not mean transcription errors are harmless everywhere. It means you have to test each link in the chain rather than assume.
Are Game Based Assessments Valid or Just More Fun?
A game based assessment measures abilities or traits through short games rather than conventional test questions. The appeal is obvious: candidates may find games less stressful and more engaging, and the games may be harder to rehearse.
Four linked studies published in 2023 on a commercial game based test of reasoning ability, scored with machine learning, drew on 11,574 completed assessments, most from graduate job applicants. Game scores overlapped moderately with a short traditional reasoning questionnaire, which is in line with earlier research that found game scores and test scores related but not interchangeable. When 102 people took the games again within about four months, their scores were fairly but not highly stable. The group results were encouraging. Women passed at 95% of the rate of men, Black applicants passed at about 90% of the rate of the highest scoring group, and applicants under 40 passed at 94% of the rate of older applicants. Each of these ratios sits above four fifths, the common rule of thumb for flagging unfair impact on a group. Among 4,778 graduate applicants asked about their experience, satisfaction averaged 4.3 out of 5. Yet the lowest ratings went to how relevant the games felt and whether they let candidates show their skills. Most important, the studies did not test whether game scores predicted job performance, and the authors flag this as the key gap.
Games aimed at personality have even less behind them. A 2024 pilot study turned a short Big Five personality questionnaire into a story based game and tested it with 98 Spanish university students. The game version was about as consistent as the questionnaire and explained self rated job performance about as well, and participants found it more attractive. But people scored noticeably higher on conscientiousness, meaning carefulness and dependability, in the game version, which may be a quirk of the format. The performance ratings were self reported, the sample was small, and the authors themselves concluded that the game needs more research before it is used in real selection.
The fair summary is that a well built game can measure reasoning with group pass rates that look acceptable and a candidate experience people like. What is missing in this evidence is the link between game scores and how people actually perform at work. Until that link is shown for the specific game you are considering, treat the game as a promising measure of a known ability, not as proof of future performance.
How Well Can Machines Score Written Answers?
Many assessments ask candidates to write: an answer to a scenario, a reply to a customer complaint, an explanation of how they would handle a problem. Scoring these by hand is slow and expensive, which is why automated scoring is attractive.
One 2023 scoring study is among the strongest pieces of evidence in this field. Researchers used 7,905 expert scores of open ended written responses from virtual assessment center exercises, collected across three companies, three occupations and seven competencies. They compared three ways of teaching a machine to score them, from simple word counting to a modern transformer language model, a type of model that reads each word in light of the words around it. On answers the models had never seen, the best model agreed with the experts about as closely as trained experts agreed with each other after discussing their ratings. The authors also examined group differences and prediction, though the headline result is the agreement with experts. The data are proprietary, so outsiders cannot check them, and the study shows the machine copied expert judgment well rather than proving those judgments predict job success.
That study carries the core lesson for written text. The tool scored answers that candidates wrote in response to a job related task, against standards set by experts for that purpose. Tools that claim to read personality from resumes or online professional profiles work differently. They guess at traits from text that was never written to reveal them, and none of the evidence reviewed here shows such guesses predicting how people perform at work. The first is measurement. The second is guesswork wearing the clothes of measurement.
Can Large Language Models Score Interview Transcripts Fairly?
Large language models are systems trained on vast amounts of text that can read and write in fluent language. Because they can follow instructions without being trained on a specific task, employers are now asking them to rate interview answers directly.
One 2024 evaluation gave simulated one way video interview responses from 685 participants to two widely used chatbots built on large language models, without any extra training. The models predicted personality about as well as, or better than, a model built specifically for the task, and the reasons they gave could be traced back to the questionnaire items. The weaknesses were serious, though. Their accuracy varied across traits, their ratings were not consistent enough when the same material was rated again, and they showed some biases. The authors urged caution for high stakes uses such as hiring.
A 2025 preprint that has not been peer reviewed built a collection of 3,890 transcripts from human resources interviews and compared several language models with expert raters. The best models produced scores close to those of the experts. They often failed, however, to spot errors in what candidates said, and their feedback was vague. The authors concluded that the models were not ready to score interviews on their own and recommended keeping a person in the loop.
How you ask the model also matters. A 2026 working paper, not yet peer reviewed, asked a language model to rate simulated job applications of known quality. When the model rated applications one at a time, its ratings bunched toward the generous middle, the same leniency that afflicts human raters. When it compared five applications at once and was told roughly how scores should spread, its ratings separated better and tracked the true quality more closely.
Taken together, this research says language models can produce plausible interview scores quickly and cheaply. They also inherit familiar rater problems such as leniency, they can miss substance while sounding confident, and they can give different answers to the same material. None of these studies tested whether the scores predict job performance. For now, these models are better suited to helping trained interviewers than to replacing them.
Who Gets to See Your Job Advertisement?
The least visible tools in recruitment may also be the most consequential. Advertising platforms decide which users see a vacancy, and sourcing algorithms suggest which past applicants or profiles a recruiter should contact. If these systems skew who is reached, every later stage works with a skewed pool.
One advertising field study published in 2019 ran advertisements promoting careers in science, technology, engineering and math that were designed to be gender neutral. Fewer women than men saw them. The cause was not a hidden preference against women. Young women are a sought after and therefore expensive audience for advertisers, so a delivery system that tries to show an advertisement as cheaply as possible ends up showing it mostly to men. The researchers found the same pattern on other major platforms. Nobody had to intend discrimination for the outcome to look discriminatory.
A 2019 delivery audit went further by running real advertisements on a large social media platform with neutral targeting. Delivery still skewed along gender and racial lines for job and housing advertisements. Both the budget and the content of the advertisement shaped who saw it, because the platform used its own predictions of which users would find each advertisement relevant. In other words, an employer can choose an open audience and still have the platform narrow it.
A later audit addressed an obvious objection, which is that skew might simply reflect differences in who is qualified. The researchers paired advertisements for similar jobs at companies with different gender makeups, so qualification was held roughly constant. Advertisements on the large social media platform still skewed by gender beyond what qualifications could explain. On a professional networking platform, they found no such skew. Skew is therefore not an unavoidable feature of advertising algorithms. It depends on how a particular platform is built.
Sourcing tools raise similar questions. A 2026 preprint, which has not completed peer review, analyzed gender balance on a real recruitment platform. Candidate lists produced by the matching algorithm alone were less gender balanced than lists recruiters built through their own searches. The most balanced lists came from combining the two, and recruiters who first reviewed the algorithm's suggestions went on to view and contact a more balanced set of candidates. One preprint is not settled evidence, but it fits a pattern seen across this research: the outcome depends on how people and systems are combined, not on the system alone.
What Do AI Recruitment Tools Mean for You?
If you run recruitment, the most useful thing you can do is stop asking whether artificial intelligence works in hiring and start asking which decision a specific tool makes and what evidence supports it. A tool that books interviews deserves a different test from one that rejects candidates.
When a supplier tells you its tool is accurate, ask accurate at what. In most of the studies above, accuracy meant agreeing with human raters. That is worth knowing, but your real question is whether the people the tool favors do better in the job and whether every group has a fair chance of getting through. Very few of the studies reviewed here tested job performance at all.
Ask what the machine learned to copy. Video scoring models that learned from interviewer impressions reproduce interviewer impressions, including any bias interviewers carry. Personality readings from resumes and profiles failed basic consistency checks. Text scoring against expert standards for a job related task held up well. These are not equivalent products, even if the brochure uses the same words.
Look upstream as well as downstream. You can audit scores you can see. You cannot easily audit the candidates who never saw your advertisement. If your applicant pool looks less diverse than the local labor market, the cause may sit with the platform that delivered the job advertisement, not with your selection process.
Finally, keep people in the loop in a way that matters. The sourcing study found the best balance when recruiters and algorithms worked together. The research does not support handing whole decisions to machines. It supports using machines where they are shown to be consistent and keeping people responsible for the judgment.
Key Takeaways
- Recruitment software is a family of very different tools, and each one has to earn trust through its own evidence rather than borrowing credibility from the category.
- Agreement with human raters is the most common proof offered, yet it says nothing about whether chosen candidates perform better at work, and few studies have tested that.
- Scoring written answers to job related tasks against expert standards has the strongest support, while personality guesses drawn from resumes, profiles and casual conversation look shaky.
- Game based reasoning tests can show fair pass rates and a pleasant candidate experience, but their link to actual job performance remains largely unproven.
- Language models can rate interview answers quickly, but they drift toward generous middle scores, miss substantive errors and may rate the same answer differently twice.
- Job advertisements can reach a skewed audience through pricing and relevance predictions alone, so a fair selection process can still start from an unfair pool.
Implications for Practice
Start by drawing your hiring funnel and writing down every tool at every gate, from advertisement delivery to final scoring. Next to each tool, record the decision it makes and whether that decision can exclude a candidate. Tools that only schedule or inform carry far less risk than tools that rank, score or filter, and the depth of evidence you demand should match that difference.
For any tool that scores candidates, ask the supplier three questions grounded in the research above. What human judgment was the model trained to copy, interviewer impressions, self reports or expert standards for a defined task? Has it been tested on new people answering new questions, not just on the data it learned from? And has anyone checked whether its scores predict performance in jobs like yours and whether pass rates differ across groups? If the answers rest only on agreement with human raters, treat the tool as unproven for selection decisions.
Where speech is turned into text, test transcription accuracy separately for candidates with different accents, first languages and backgrounds. The transcription study showed that errors fall unevenly even when final scores turn out fair, and you cannot know which outcome you have without checking your own data.
Do not use tools that infer personality from resumes, online profiles or unstructured chat. The chatbot study showed weak links to real outcomes, and nothing reviewed here shows that personality read from such text predicts performance. If you need personality information, use a properly validated questionnaire.
If you use game based assessments, ask for evidence linking game scores to job performance in roles similar to yours, and monitor pass rates by group in your own applicant pool, since the published group results came from one region and a narrow range of jobs.
If you let a large language model rate interview answers, run the same answers through it more than once and compare the results. Give it clear scoring guidance, compare several candidates at a time rather than one by one, and have a trained interviewer review the scores before any decision is made.
For job advertisements, run paired test advertisements, track who actually sees each vacancy where the platform allows it, and use more than one channel. Since one platform skewed delivery and another did not, the choice of channel is itself a fairness decision.
With chatbots, budget recruiter time for writing scripts and following up on thin applications, connect the bot to your applicant tracking system, and tell candidates what information the bot does and does not need. If you introduce automated interviews, make sure the structure is the point: fixed questions and consistent follow up are what the field evidence credits, and those can also be built into human interviews.
