Key takeaways
- Duplicates create repeat outreach, which is the problem candidates notice first.
- Match exact keys such as email and phone first, then review fuzzy matches by hand.
- Keep one outreach history per person, and check it before any new contact.
- Audit a sample of records each month, because duplicates come back.
In this post
A strong engineer applies through a job board in March. A recruiter at the agency sources the same person from a professional network in April, using a different email address. In May, a second recruiter sends a message about a different role. The candidate replies with one line: they have already been contacted about this. The agency now has two records, and a reputation problem that started with a spreadsheet.
How common are duplicates?
We could not find an independent survey that gives an industry-wide rate of duplicate candidate records. The one hard figure we found is a vendor case study. A recruiting consultancy that sells deduplication reported a database of about 34,000 candidates, with roughly 6,800 flagged as probable duplicates. That is about one record in five, and it is one example, not a sample. Fewer than 2,000 of the flagged records were exact matches on name and email, so most needed fuzzy matching.
Candidate records in one vendor case study
- Candidate records in the database34,000
- Flagged as probable duplicates6,800
Source: 4spot Consulting case study (see Sources). A single vendor example, not a survey.
Why duplicates form
Duplicates are usually created by normal work, not by carelessness. A candidate applies with a personal email and later uses a work address. A sourcing tool imports a profile that already exists under a slightly different name spelling. A resume parser creates a new record because the name field was read differently. Several recruiters or agencies each keep their own list, and nobody owns the master record.
Signs you have this problem
- Candidates mention they were contacted about the same role twice.
- Searching for a name returns two or more records for the same person.
- Recruiters ask in a meeting whether someone is already in the system.
- Outreach is sent from a sourcing list that was not checked against the ATS.
- The ATS has no report of duplicate records.
How to measure your own duplicates
Export a sample of 200 candidate records. Group them by email address and by normalized phone number, and count the exact duplicates. Then review a smaller set of likely pairs with matching names and employers, and count the true duplicates. Finally, count how many candidates received more than one outreach in the last 90 days. That last number is the one candidates feel.
The tools that help, and the decisions they leave to you
OpenCATS is an open-source applicant tracking system for recruiting agencies and internal HR teams. pyresparser is a Python tool that extracts skills, experience and contact details from resumes using spaCy and NLTK. flair is an NLP framework that teams use to build entity recognition for candidate documents. Together they cover the record, the parsing and the text extraction.
Deciding that two records are the same person, which record to keep, and who may contact that person next is a policy decision. Tools can suggest matches, but a recruiter should confirm the merge.
A deduplication pass that keeps one record per person
- 01Normalize email addresses and phone numbers
- 02Merge exact matches on email or phone
- 03Score fuzzy matches on name and current employer
- 04A recruiter confirms or rejects each probable match
- 05Keep one outreach history and check it before any new contact
The steps, in order.
Best practice: one person, one record, one history
- Normalize emails and phone numbers at the point of entry, not later.
- Match exact keys first, and send fuzzy matches to a person before merging.
- Check the outreach history before any message, and block contact for a set period after the last one.
- Audit a sample of records each month, because duplicates come back after every import.
- Keep candidate data handling within the privacy rules that apply to your candidates, and check with your legal adviser where the rules differ by country.
How Asauvia helps
We build the deduplication pass, the outreach check and the monthly audit for recruiting teams, and connect them to the ATS and sourcing tools you already use. A first engagement usually starts with the 200-record sample above, so you can see the duplicate rate and the repeat-outreach count before anything is merged.
Automated merging of candidate records needs clear rules and a person to confirm the uncertain cases. We agree those rules with your recruiting leads before anything runs.
- #Recruiting
- #ATS
- #TalentAcquisition
- #RecruitingOps
- #CandidateData
Share this post