Here is a question most HR teams never ask: does our applicant tracking system actually help us pick the best people? The honest answer, based on decades of research into personnel selection, is no. It does not. An ATS tracks applications. It sorts resumes. It stores candidate data. But it does not predict who will perform well on the job. And that gap between tracking and selecting is where organisations lose their best hires.
Nearly 99% of Fortune 500 companies use an ATS. The technology has become so common that many hiring teams treat it as the selection system itself. They set keyword filters, rank resumes by match score, and assume that the highest-scoring candidate is the best. The research says otherwise.
This article examines why the confusion between tracking and selecting creates a real cost for organisations, what the science of personnel selection actually tells us about predicting job performance, and what HR leaders can do to close the gap.
What an ATS Actually Does (And Does Not Do)
An ATS is a database with workflow tools bolted on. At its core, it collects applications, parses resumes into searchable fields, and moves candidates through stages of a hiring pipeline. It can post jobs to multiple boards, send automated emails, schedule interviews, and generate compliance reports. These are useful administrative functions.
But here is the problem: none of these functions measures a candidate’s ability to do the job. An ATS decides who moves forward based on keyword matching, not job relevant competencies. It checks whether your resume says “project management” when the job description says “project management.” It does not know whether you can actually manage a project.
The Harvard Business School and Accenture study on “Hidden Workers” surveyed over 2,250 executives and found that 88% of employers believe their ATS filters out qualified candidates who do not match the exact criteria in the job description. Think about that number. The people running these systems know the systems are rejecting good candidates, and they keep using them anyway.
A 2025 study on ATS and AI matching found that traditional keyword based ATS systems miss qualified candidates who use different terminology. Someone who writes “client relations” instead of “customer success” gets filtered out, even though the skills overlap almost completely. The researchers found that newer AI based approaches using transformer models outperformed traditional ATS keyword matching by up to 15.85% in matching accuracy.
The root cause is simple: an ATS was never designed to be a selection instrument. It was designed to manage volume. And when you use a volume management tool as your selection tool, you get exactly what you should expect. Efficiency without effectiveness.
What the Science of Selection Actually Says
Personnel selection is one of the most thoroughly researched topics in organisational psychology. We have over a century of data on what predicts job performance and what does not. The findings are clear and, for most hiring managers, deeply uncomfortable.
Schmidt and Hunter published what became one of the most influential papers in the history of industrial psychology in 1998. They synthesised 85 years of research and ranked 19 selection methods by how well each one predicts job performance. General mental ability tests came out on top with a validity coefficient of r = 0.51. Structured interviews came in at r = 0.51 as well. Work sample tests scored r = 0.54.
And the methods that most ATS systems rely on? Years of education showed a validity of r = 0.10. Years of job experience came in at r = 0.18. In plain terms, the two things that keyword matching prizes most, education credentials and experience, are among the weakest predictors of how someone will actually perform on the job.
In 2022, Sackett, Zhang, Berry, and Lievens published an updated meta analysis in the Journal of Applied Psychology that revised many of Schmidt and Hunter’s estimates downward after correcting for statistical overcorrections in earlier studies. But the relative rankings held. Structured interviews emerged as the single strongest predictor of job performance with a revised validity of r = 0.42. Cognitive ability came in at r = 0.31. Job knowledge tests showed r = 0.40.
The message from both the original and updated research is the same: the best predictors of job performance are structured, job relevant assessments. Keyword matching on a resume is not one of them.
Related: The Best Employee Recruitment and Selection Methods Every Manager Need to Know
The Hidden Cost of Treating ATS as a Selection System
When organisations confuse tracking with selection, the costs manifest in several ways. Some are obvious. Others are harder to see but just as damaging.
The first cost is rejected talent. The Harvard Business School study estimated that more than 10 million workers in the US alone fall into the category of “hidden workers”, people who are qualified but get filtered out by automated screening before a human ever reviews their application. Nearly half of US employers use employment gap filters that automatically exclude anyone with more than six months between jobs. This eliminates caregivers, veterans transitioning from military service, people who took time off for health reasons, and many other capable workers.
The second cost is false confidence. When a hiring manager sees a shortlist from the ATS, they assume those candidates are the best available. But the shortlist is just the candidates whose resumes had the right words in the right places. The actual best candidate may have been eliminated three stages ago because their resume said “Bachelor of Arts” instead of “BA” and the parser did not match them.
The third cost is the validity gap. Selection systems are supposed to predict future performance. An ATS predicts nothing. It sorts. It ranks. But ranking resumes by keyword density is not the same as measuring whether someone can do the job. The Society for Industrial and Organizational Psychology has noted that the predictors at the top of the validity rankings are job specific assessments like structured interviews, job knowledge tests, work sample tests, and empirically keyed biodata. None of these are things an ATS can do.
The fourth cost is reduced diversity. When you set rigid filters for degrees, years of experience, specific job titles, and continuous employment history, you systematically exclude the very populations that diversity initiatives are supposed to reach. Berry, Lievens, Zhang, and Sackett (2024) found that excluding cognitive ability tests from selection batteries had little to no effect on validity but substantially decreased adverse impact. The lesson here is that the selection tools we choose shape the diversity of our workforce, and relying on crude ATS filters is one of the worst choices we can make.
Tracking Versus Selecting: Know the Difference
The confusion between tracking and selecting is not just a semantic problem. It reflects a fundamental misunderstanding of what hiring technology should do at each stage of the process.
Tracking is an administrative function. It answers questions like: how many people applied? Where are they in the pipeline? Have we responded to them? Are we meeting compliance requirements? These are important operational questions, and an ATS handles them well.
Selecting is a measurement function. It answers a completely different question: among the people who applied, who is most likely to succeed in this role? Answering that question requires psychometrically sound tools, ones that have been validated against actual job performance criteria. Things like structured interviews, cognitive ability assessments, work sample tests, situational judgment tests, and personality measures designed for selection.
Most organisations blend these two functions without realising it. They use the ATS to filter candidates (a selection decision), then use the ATS shortlist to decide whom to interview (another selection decision), and only then apply an actual validated selection method like a structured interview. By that point, the candidate pool has already been narrowed by a tool with no predictive validity for job performance.
Related: Recruitment vs Selection: Understanding the Key Differences
What Organisations Should Do Instead
Fixing this problem does not mean throwing out your ATS. It means using it for what it is good at and adding real selection science on top.
First, stop using the ATS as a screening tool for selection decisions. Use it to collect applications, manage workflows, and track candidates through the process. But do not let keyword match scores determine who advances and who gets rejected. The research is clear: keyword matching has no established relationship with job performance.
Second, introduce validated selection tools early in the process. Structured interviews, cognitive ability tests, and situational judgment tests all have strong evidence behind them. Sackett and colleagues found that combinations of these tools produce validity levels comparable to what we expected from the older, more optimistic estimates. Using multiple validated tools together gives you a far better prediction of job performance than any resume parsing algorithm.
Related: Recruitment Technology: What You Need to Know
Third, audit your ATS filters. Go through every filter your system uses and ask: is there evidence that this criterion predicts job performance? If the answer is no, remove it. Degree requirements, continuous employment filters, exact job title matches — these are proxies that often screen out qualified people while adding no predictive value.
Fourth, review your job descriptions. The Harvard Business School study found that job descriptions are rarely updated and often contain inflated or outdated requirements. Bloated job descriptions feed bloated ATS filters, which creates a self inflicted talent shortage. Strip your job descriptions down to what actually matters for performance in the role, then build your selection process around measuring those things.
Fifth, measure what matters. Track quality of hire, not just time to fill. If your ATS helps you fill positions faster but the people you hire do not perform well, you have not gained anything. The real measure of a hiring system’s value is whether it predicts on the job success.
Related: The Importance of Pre-employment Testing in Modern Recruitment
The Science Is Clear. The Practice Has Not Caught Up.
We have more than a century of research on personnel selection. The findings are consistent across thousands of studies. Structured, job relevant assessments predict performance. Resume keywords do not. Yet the hiring technology that dominates the market was built around resume keywords. The mismatch between what science says and what organisations do is one of the biggest unforced errors in human resources today.
An ATS is a fine tool for managing applications. It is a terrible tool for choosing people. Organisations that understand this difference and build their selection process accordingly will consistently hire better talent. Those that keep treating their tracking system as a selection system will keep wondering why their new hires do not work out.
The gap between tracking and selecting is not a technology problem. It is a knowledge problem. And the knowledge to fix it has been available for decades.
