How to evaluate a recruitment AI product

Twenty questions for any vendor

What it does

1. Which specific task does the AI complete end to end? Where does a human take over?
2. What data does it search or learn from? Is it licensed, public, or scraped?
3. What is the model trained or tuned to predict? (For ranking: what counts as a "good" candidate?)
4. Can it explain each individual output, in words a hiring manager would accept?
5. Which channels does it use to contact candidates, and how is deliverability managed?

Quality and safety

6. What is the measured reply rate / match agreement / scheduling time for customers like us?
7. Have you run a bias audit? Can we see the summary and the methodology?
8. How are factual errors in generated text prevented or caught?
9. What happens when a candidate asks to be removed or asks for human review?
10. What do candidates see? Can we read the exact messages or questions before go-live?

Data and compliance

11. Does our data train models used for other customers? Can we opt out?
12. Where is data stored; what are the retention and deletion rules?
13. Which laws have you designed for (NYC LL144, EU AI Act, GDPR)? Show the documentation.
14. Do you have SOC 2 or ISO 27001? Can we see the report?

Commercials and fit

15. What is the pricing model: per seat, per contact, per hire, flat? What happens at twice our volume?
16. Is there a free tier or trial where we can measure results before paying?
17. Which ATS and calendars does it integrate with natively?
18. How long is implementation, and who does the work?
19. What does a customer who churned say about why?
20. What is on the roadmap for the next two quarters, and what was shipped in the last two?

A four-week pilot

1. Week 0: pick one metric Interested replies per week for a sourcing agent; time to scheduled interview for conversational AI; scorecard completion for interview intelligence.
2. Week 1: baseline Measure the current process on the same roles for one week.
3. Weeks 2–4: run the tool On two or three live roles, with the same recruiter, same brief quality.
4. Review Compare the metric, read a sample of candidate-facing output, ask the hiring manager to rate shortlist relevance, and total the cost including recruiter time.

Red flags

• Cannot say what the model predicts or what data it uses.
• No way to read candidate-facing messages before they go out.
• Claims of "bias-free" AI (no such thing; ask for audit results instead).
• Describes LinkedIn automation as a feature; it violates LinkedIn's terms and puts your accounts at risk. Reputable sourcing agents reach candidates by email and SMS.
• Pricing that only makes sense on a three-year contract.