The evidence on AI tools for HR outside recruitment is mixed but clear in places. These tools do well on narrow, checkable jobs such as scoring written comments, estimating job requirements and supporting goal setting. They do badly where a confident wrong answer is costly, such as answering legal and policy questions, and their predictions about people need careful handling.
Most of the conversation about artificial intelligence in human resources is about hiring. Yet the bigger share of the work in a human resources department happens after people are hired: writing job descriptions, answering questions about leave and benefits, tracking who might leave, giving feedback, coaching and training. Software is now sold for all of these jobs, and four assumptions travel with it.
The first assumption is that a tool which writes fluent, confident text must know what it is talking about. The second is that more data about employees means better predictions about who will quit. The third is that employees will welcome feedback from a machine as long as the feedback is accurate. The fourth is that a chatbot coach or tutor is a cheap and weaker copy of a human one.
Each of these assumptions turns out to be partly wrong, and in different directions. Fluency and accuracy are separate things, and some of the best tested systems still invent answers. Turnover models work moderately well at the level of groups but carry legal and ethical traps at the level of individuals. Feedback from a machine can be better than feedback from a manager and still lower performance once people know where it came from. And in two careful trials, software coaching and tutoring held their own against people. The sections below look at what the research actually found, task by task, and where the gaps in that research still are.
Why Do So Many Leaders Assume One Tool Can Do Everything?
Part of the confusion comes from the word itself. "Artificial intelligence" covers two quite different families of software. The first family is machine learning, which means software that looks for patterns in past records and uses them to make predictions, such as which employees resemble those who resigned last year. The second family is large language models, which are programs trained on enormous amounts of text to predict the next word in a sentence. Large language models are the engines behind the chatbots that can draft a policy, summarize a grievance or answer a question in plain English.
These two families fail in different ways. A prediction model can be wrong about a person while being right about a group. A language model can be wrong about a fact while sounding completely sure of itself. When a vendor demonstration shows a chatbot writing a polished job description in 10 seconds, the polish says nothing about whether the content is correct, fair or suited to your organization. Leaders who have seen one impressive demonstration often extend that trust to every task the software claims to perform.
There is also a research gap that most buyers never see. Studies of software in hiring are fairly plentiful. Studies of software in the rest of human resources work are much thinner, and a good deal of what we know has to be borrowed from neighboring fields such as law, education and health. A 2026 mapping review of personalized coaching software screened 2,072 records and kept 42 peer reviewed papers. Most of those systems were built for lay users in health care, and workplace settings were barely represented at all.
That gap matters. It means a human resources leader who wants evidence for a coaching chatbot, a policy assistant or a turnover dashboard will often find studies from a hospital, a law library or a university physics course. Those studies are useful, and this article relies on several of them, but the honest position is that you are reasoning by analogy. The safest approach is to treat each tool as a hypothesis to be tested in your own setting, not a settled answer you can buy.
How Often Do AI Tools for HR Make Things Up When Answering Employee Questions?
One of the most popular uses of language models in human resources is the employee help desk: a chatbot that answers questions such as "How many days of compassionate leave do I get?" or "Can my manager change my shift without notice?" These questions often have a legal dimension. That makes research on legal question answering the closest available test of how such a tool might behave.
The key risk has a technical label: hallucination. In plain words, a hallucination is when a language model produces a statement that sounds authoritative but is false, or that is not supported by the sources it claims to rely on. It is not a rare glitch. It follows from how these models work, because they are built to produce likely sounding text, not to check facts.
A 2024 peer reviewed study asked general purpose language models specific, checkable questions about randomly chosen federal court cases in the United States. The models gave false answers in 58% of responses for the strongest model tested and 88% for an older open model. Two further findings are more worrying for anyone running an employee help desk. The models often failed to correct a user who asked a question built on a false assumption, and they could not reliably tell when they were producing a false answer. An employee who asks "Since I am entitled to 30 days of sick leave, how do I claim the rest?" may get a confident procedure built on an entitlement that does not exist.
Vendors responded to this problem with a design called retrieval, which means the system first searches a trusted library of material and then writes its answer from what it found. The idea is that tying answers to real sources should stop the model from inventing things. An evaluation published in 2025 put this claim to the test. The researchers sent 202 legal questions to specialist research tools sold by the two largest legal publishers, tools that were marketed as avoiding or even eliminating hallucinations. The tools invented content between 17% and 33% of the time. The share of answers that were both correct and properly supported by sources ranged from 65% for the best tool down to roughly 41% and 19% for the others. The specialist tools did better than a general purpose chatbot, which hallucinated more, but they were nowhere near the error free systems their marketing described.
What does this mean for a human resources chatbot? There are reasons to think an employee policy assistant could do better than a legal research tool. Your policy manual is far smaller and more consistent than the whole body of case law, and many employee questions are routine. But there are also reasons for concern. Policy manuals are often out of date, contradict collective agreements, or leave out the exceptions that matter most. I could not find a peer reviewed study that measured how often chatbots give wrong answers to real employee policy questions, so nobody can tell you the error rate with confidence. If your chatbot handles 5,000 questions a year and is wrong even 5% of the time, that is 250 employees acting on false information about their pay, leave or rights.
The pattern from the legal studies is consistent: retrieval reduces errors but does not remove them, and the errors that remain are the hardest kind to spot because they are wrapped in fluent, confident language and real looking citations.
Can Software Draft Job Descriptions and Job Advertisements You Can Trust?
Writing job descriptions is slow, and many organizations have hundreds that are years out of date. It is easy to see why drafting software is attractive. The evidence here splits into two parts: estimating what a job requires, and writing the text that describes it.
On the first part, the research is encouraging. Job analysis is the formal process of working out what knowledge, skills, abilities and interests a job demands, usually by asking panels of experienced people to rate long lists of requirements. It is expensive and tedious. A 2023 study trained machine learning models to estimate those ratings using nothing but the written job description and the list of tasks. The models learned from 963 occupations in a large national occupational database and were then tested on 229 jobs from a single large organization that the models had never seen. Across 126 different requirements, the machine estimates lined up closely with the ratings of expert panels, most closely for interests and skills and somewhat less closely for knowledge and abilities. The patterns also made sense: related requirements were predicted by related words in the task lists.
That is a meaningful result, but notice what kind of tool it was. It did not write anything. It read carefully written task statements and produced numerical estimates, which experts could then check. It was a draft for a job analyst, not a replacement for one.
The second part, generating the text itself, is where the problems show up. One 2022 study asked a large language model to write job advertisements and compared them with real advertisements for bias and realism. The generated advertisements carried gender bias. Adding instructions in the prompt asking for inclusive, diverse language made no meaningful improvement. What did help was further training of the model on real advertisements that had been screened for bias, which most human resources teams cannot do themselves.
A related line of work examined reference letters, which are close cousins of performance summaries and promotion recommendations. Research presented in 2023 found clear gender bias in letters written by language models, both in the style of language and in the content. Letters about women leaned toward describing warmth and pleasantness, while letters about men leaned toward describing leadership and achievement. The researchers also found that when the models invented details, the invented material tended to amplify the bias. Their conclusion was that such letters should not be used without careful human review.
There is one more limitation that the studies did not test directly but that follows from how these models work. A language model writes the most typical version of a text. Ask it for a job description for a "finance manager" and you will get the average finance manager role found in its training material, not the one in your organization, with its particular reporting lines, decision rights and scope. For job evaluation, where small differences in decision making authority can move a job between grades, the typical version can be actively misleading.
Can a Machine Tell You Who Is About to Quit?
Turnover prediction is the most studied use of machine learning in people analytics, the practice of using employee data to inform decisions. The basic method is simple. You take records of past employees, mark who left and who stayed, and let the software find the patterns that separate the two groups. This is called supervised learning, because the software learns from cases where the outcome is already known.
A 2024 systematic review gathered 52 peer reviewed studies of machine learning for turnover prediction published between 2012 and early 2023. More than 20 different techniques had been tried. Nearly all the studies, 50 of the 52, used supervised learning. Across the studies, salary and overtime came out as the most important factors for predicting who would leave. That finding is worth pausing on: the strongest signals were ones any competent human resources manager already watches, not hidden patterns only a machine could see.
Studies inside real organizations show what these models can and cannot do. A 2025 study followed 2,932 employees of a Northern European telecommunications multinational, at its Norwegian headquarters and a Danish subsidiary, and recorded who had left two years after a survey. About 8.6% of permanent employees had left voluntarily. The single strongest predictor was which country a person worked in, and the risk profiles then looked different in each country. In Norway, risk was tied to things such as low job variety. In Denmark, it was tied to job freedom and to how much employees valued job rotation. The model ranked a person who left above a person who stayed roughly three times in four, which the authors themselves called moderate accuracy. They stressed that it came from one firm and should support, not replace, human judgment.
Another study from 2023 used ordinary human resources records from two community mental health centers in the United States. A more flexible technique did better than a simpler one, and the better model ranked leavers above stayers more than 80% of the time. The most useful predictors included years of service, wage, hours worked, age, job position, training hours and marital status.
That last list should make any human resources professional uneasy. Age and marital status are protected or sensitive characteristics in many countries. A model that flags people partly because of their age or marital status, and a manager who then treats those people differently, can produce exactly the kind of unequal treatment that employment law forbids. A 2019 commentary in a leading industrial and organizational psychology journal raised precisely this problem: turnover models can create disparate treatment, which means treating people differently because of their group, and adverse impact, which means a practice that falls harder on one group even when nobody intends it.
The same journal carried a 2019 focal article that set out the practical decisions in building these models: how to plan the study, how to combine turnover theory with the messy data organizations actually hold, which analytical methods to use and how to communicate the results. Its central message was that the psychology of why people leave should guide which variables go into the model, rather than throwing every available field at the software.
There is also a simple arithmetic problem that vendors rarely explain. When only about one employee in 12 leaves in a given period, even a reasonably accurate model will flag many people who were never going to go. If a manager treats every flagged person as a flight risk, by withholding a stretch assignment or by starting to plan their replacement, the prediction can start to cause the very departure it forecast. These models are most defensible when used to spot groups and conditions, such as a team with heavy overtime and flat pay, and least defensible when used to label individuals.
What Happens When Feedback Comes From an Algorithm?
Feedback is where the research gets most interesting, because it shows that accuracy is not the whole story.
A 2021 field experiment tested what happens when a company uses artificial intelligence to give employees feedback on their performance. The researchers separated two effects. The first was a deployment effect: the software analyzed far more data than a manager could and produced feedback that was more accurate, more consistent and more relevant, which improved performance. The second was a disclosure effect: when employees were told that the feedback came from a machine, they formed negative views of it and their performance dropped. Both effects existed at the same time. The negative reaction to disclosure was weaker among employees who had been with the firm longer. The authors suggested a tiered approach in which experienced staff receive machine feedback while newcomers receive feedback from human managers.
A similar pattern appeared in a very different setting. A 2024 set of experiments looked at whether messages written by a language model could make people feel heard when they described a personal difficulty. The machine written responses made recipients feel more heard than responses written by people, and the software was better at detecting the emotions being described. But once recipients learned a response came from a machine, they felt less heard. Outside raters also judged that the software offered support with more restraint and fewer unrequested practical suggestions than people did. These were not workplace studies, but the lesson transfers directly to performance conversations, recognition messages and responses to employee concerns.
There is a second, quieter use of software in performance management that has stronger evidence behind it: reading what people write. Most organizations collect thousands of written comments in appraisals and surveys and then ignore them because nobody has time to read them all.
One 2021 study built automated scoring for specific dimensions of job performance from the narrative comments in appraisals. Across three samples, the automated scores agreed with independent human ratings of the same text and with the numerical ratings managers had given. The scores worked for both downward ratings, from supervisor to employee, and upward ratings, from employee to supervisor. But the quality depended on which technique was used, and the scoring transferred less well when the comments had been collected with different instructions.
A 2023 study extended this to employee surveys. The researchers built scoring tools for open ended comments covering 28 work attitudes and perceptions, such as views of pay, supervisors and workload. The tools produced two kinds of score: what a comment was about, and how positive or negative it was about that topic. The positivity scores were reliable and valid, especially when the comment was actually about the topic being measured. The practical lesson was that the question you ask employees shapes whether the software can read the answers well. Vague prompts such as "Any other comments?" produce text that is harder to score accurately than focused prompts about a specific topic.
Put together, these studies say something useful. Software can read and summarize feedback text at a scale and consistency no human team can match, and the scores it produces can be trusted when the questions are well designed. Software can also write feedback that is technically better than what many managers produce. The obstacle is human: people discount help when they know it came from a machine. That is not a reason to hide the source, which would be dishonest and would collapse trust when discovered. It is a reason to keep managers in the conversation, to use machine analysis to inform what they say rather than to replace them, and to explain to employees what the software does and does not do.
Can a Chatbot Coach or Teach as Well as a Person?
Coaching and learning are the areas where the evidence most clearly defies the expectation that software is a cheap and weaker copy.
A pair of randomized trials published in 2022 compared goal attainment, that is, how far people progressed toward goals they had set, under three conditions: human coaching, coaching by a chatbot, and no coaching. A randomized trial assigns people to groups by chance, which is the strongest way to test whether an intervention causes a result. The first trial randomly assigned 210 people to six monthly one hour sessions with a human coach or to a control group. The second trial randomly assigned 268 people to a coaching chatbot or a control group. Each ran for 10 months with monthly measurements, and 327 participants provided data at every point. Both kinds of coaching beat their control groups. The chatbot and the human coaches never differed meaningfully from each other at any point, and the size of the benefit was almost identical by the end. People who used the chatbot more often made larger gains, though that comparison was not randomized and may simply reflect more motivated users.
The limitations matter. The participants were university students, not employees. All the measures were self reported. And goal attainment was the only outcome measured. The study tells us that a well designed chatbot can help people stay focused on goals they have chosen. It does not tell us whether a chatbot can handle the messier work of executive coaching, such as helping a leader see how their behavior affects a team or working through a conflict with a peer.
Learning shows a similar pattern. A 2025 randomized trial at a large American university compared a tutor powered by a large language model with in class active learning, a teaching method that already beats traditional lectures. The study used a crossover design, meaning each student experienced both methods on different topics, and 194 students were eligible for analysis. Students who learned from the software tutor at home gained more than twice as much as those in the active class, measured by the middle score on tests before and after the lesson. The typical student using the tutor finished in 49 minutes, less than the hour assumed for the class. Students also reported feeling more engaged and motivated.
That is a striking result, but it comes with real caveats. It was one course at one institution covering three topics. The tutor was carefully built, with expert written instructions, step by step solutions prepared in advance and teaching videos made by the instructors. The lead author developed the tutoring software. And the test was of first exposure to new material, not of complex reasoning or applying skills on the job. A corporate learning team that buys a generic chatbot and points it at a slide deck should not expect the same results.
The mapping review mentioned earlier adds a warning about the coaching software market in general. It found that personalization, the feature most products advertise, was rarely based on theory or expert input and was evaluated less often than coaching outcomes, usually without independent assessment. In other words, the claim that a product adapts to each user is usually the least tested part of it.
The honest reading of this evidence is that software coaching and tutoring can work well for structured goals and well defined learning content, especially when the tool is designed with care by people who understand the subject. The evidence for complex workplace development, the kind that changes how a manager leads, is close to absent.
What Do AI Tools for HR Mean for Your Team?
If you lead a human resources function, the research points to a simple sorting rule. Ask of every proposed tool two questions: how bad is a confident wrong answer here, and how easily can a person check the output before it reaches an employee?
Where errors are cheap and checking is easy, you can move quickly. Using software to read thousands of survey comments, to produce a first estimate of job requirements for an analyst to review, or to support employees working toward goals they have set are all uses with reasonable evidence and limited downside.
Where errors are costly and hard to spot, you should slow down. A chatbot that answers employee questions about pay, leave and rights will sometimes be wrong, and its mistakes will look exactly like its correct answers. If you use one, decide in advance which questions it must route to a person, test it against a set of questions with known answers before launch, and keep testing it after every policy change.
Where the output is a judgment about a person, such as a turnover flag or a performance score, you should be most careful of all. You will be held responsible for what the model does, including any unequal treatment it produces, whether or not you understand how it works. Use these tools to find patterns and conditions that you can fix, not to label individuals.
And wherever software touches feedback, remember that your people will react to the source as well as the content. Your managers remain the people employees trust to deliver judgments about their work, even when software helps prepare those judgments.
Key Takeaways
- Specialist question answering systems that search trusted sources before replying still invented content in roughly one answer in six to one in three in a careful test, so any employee help chatbot needs routing rules and ongoing checks.
- Software can estimate the requirements of a job from well written task lists with accuracy close to expert panels, but generated job advertisements and recommendation letters carry gender slants that polite prompting does not remove.
- Turnover models usually reach moderate accuracy, rely heavily on signals such as pay and overtime that managers can already see, and can drift into unlawful treatment when sensitive characteristics enter the model.
- Machine feedback can be more accurate than manager feedback and still lower performance once employees know its source, with newer staff reacting most strongly.
- Automated reading of appraisal and survey comments has solid support, provided the questions asked of employees are specific enough to produce relevant answers.
- Two randomized trials found a coaching chatbot matched human coaches on goal progress, and a software tutor beat active classroom learning, but both studied students and narrow outcomes.
Implications for Practice
Before launching an employee question chatbot, build a test set of at least 100 real questions with answers agreed by your employment lawyer and your policy owners, including questions that contain false assumptions. Run the chatbot against the test set and record how often it is wrong, how often it fails to correct a false premise, and how often it cites a policy clause that does not say what it claims. The legal research findings show these are the failure types to expect. Set a rule that any question involving dismissal, discipline, discrimination, pay disputes or medical matters goes straight to a person, and rerun the test whenever a policy changes.
When using software to draft job descriptions or advertisements, treat the output as raw material. Start from your own task lists rather than a job title, because the job analysis study shows that accuracy comes from detailed task statements. Then have an editor check the draft against a written list of gendered and exclusionary language, since asking the software to be inclusive did not fix the bias in the job advertisement study. For job descriptions used in grading, have a trained analyst confirm decision making authority and scope line by line, because the typical role the software describes may not be your role.
If you build or buy a turnover model, remove protected characteristics and obvious stand ins for them from the inputs, and test whether the people it flags differ by sex, age or other protected group. Use the results to find conditions that drive leaving, such as overtime, pay position and job variety, and act on those conditions at the team level. Do not give managers individual risk scores, because with low base rates most flagged people will stay, and the flag itself can change how they are treated.
When software helps prepare performance feedback, let managers deliver it and own it, and tell employees plainly how the software was used. Pay extra attention to new joiners, who reacted most negatively to machine feedback in the field experiment. For surveys and appraisals, rewrite open ended prompts so each one asks about a specific topic, which will make automated analysis of the comments far more accurate.
For coaching and learning, pilot software with structured goals and well defined content first, and measure results against a comparison group that did not use the tool. Ask vendors for independent evaluations of their personalization claims, since the mapping review found this is the least tested feature. Keep human coaches for complex leadership development, where the evidence for software is thin.
