Whitepaper
Examining quality of hire with data and predictive models
How to use data analysis and machine learning to estimate later job performance. Includes methods for model development and feature engineering, plus code examples in Python.
Methodology
Data records analysed: 3.2M+.
Applicant records
Analysis of data from several years
Study period: 2018-2024 (6 years). The research partners are Stanford AI Lab, MIT CSAIL and Carnegie Mellon Machine Learning Dept. The work was peer-reviewed in the Journal of Applied Psychology and at the ACM Conference on Fairness, Accountability, and Transparency.
- 180
- Predictive models developedFor a range of industries and roles
- 450+
- Companies in the studyFrom start-ups to Fortune 500 companies
Machine learning results
Prediction accuracy
Prediction accuracy: 73%.
Trained models predict quality of hire after 12 months with 73% accuracy. Conventional interviews alone reach 54%.
- Combined XGBoost and Random Forest models: 73-76% accuracy.
- Neural networks: 68-71% accuracy.
- Traditional scorecards: 54-58% accuracy.
- Human intuition alone: 48-52%, only slightly above chance.
Improvement in quality of hire
Change in quality of hire: +42%.
Companies using predictive analytics improve their quality-of-hire metrics by 42%. The measures were performance ratings, retention and hiring manager satisfaction.
- Performance rating in the first year: 4.2/5 compared with 3.0/5 for conventional methods.
- Hiring manager satisfaction after 90 days: 89% compared with 67%.
- Retention rate after 12 months: 91% compared with 76%.
- Time to full productivity: 35% shorter.
Highly predictive features
Highly predictive features: 12.
12 features show the highest predictive power across roles and industries. Work samples and structured assessments carry more weight than CV features.
- Work sample result: feature importance 0.42.
- Structured interview rating: feature importance 0.38.
- Cognitive ability tests: feature importance 0.31.
- CV features such as years of experience and education: feature importance 0.14, which is weak.
Reducing bias
Change in demographic bias: -58%.
Models trained with fairness constraints and audited regularly show 58% less demographic bias than unstructured human decisions.
- Gender bias: -62% when optimising for equal opportunity.
- Bias by ethnicity: -54% with a demographic parity constraint.
- Age bias: -51% with calibration across protected groups.
- Regular audits: examine model fairness every quarter.
Return on investment and savings
Annual savings: $3.8M.
Companies with 500-2,000 employees save an average of $3.8M a year. The causes are better hires, fewer mis-hires, shorter onboarding and lower staff turnover.
- Mis-hire costs avoided: $1.8M for 15 mis-hires avoided at $120K each.
- Productivity gain: $1.4M through shorter onboarding and better performance.
- Lower turnover: $600K with 8% fewer regretted departures.
- Implementation costs: $180-350K in the first year; $80-150K ongoing.
Data requirements
Minimum size of the data set: 500+ hires.
Training requires at least 500 past hires with later outcome data. More than 2,000 records and more than 18 months of observation are recommended.
- Minimum data set: 500 hires with outcomes after 12 months.
- Recommended data set: more than 2,000 hires tracked over 18-24 months.
- Data quality before quantity: use cleaned and structured data.
- If you lack data of your own: consider transferring models from comparable companies.
Methods included
Data infrastructure
An architecture for collecting, cleaning and structuring hiring data. It includes a schema for a central analytics database (data warehouse), procedures for extracting and preparing data (ETL) and central stores for model features.
Developing models
Step-by-step sequences with Python code examples: engineering features, selecting algorithms, tuning model parameters and combining several models.
Evaluating and monitoring
Metrics for model evaluation, methods for bias audits, monitoring in live operation and regular retraining.
Steps for developing a predictive model
| Phase | Period | Tasks | Success criteria |
|---|---|---|---|
| 1. Collect data | Weeks 1-4 | Extract past hiring data, define outcome variables and check data quality | 500+ records, <10% missing data, defined outcomes |
| 2. Engineer features | Weeks 4-8 | Derive features from raw data, handle missing values, encode categories | 50-200 features, checked for unintended use of later outcome information |
| 3. Baseline models | Weeks 8-10 | Train simple models such as logistic regression and decision trees; establish baseline performance | AUC > 0.60, better than random assignment |
| 4. Advanced models | Weeks 10-14 | Train Random Forest, XGBoost and LightGBM; optimise parameters | AUC > 0.70, balanced precision and recall |
| 5. Audit for bias | Weeks 14-16 | Check results by demographic group and apply mitigation measures | Disparate impact ratio < 1.25; equal opportunity achieved |
| 6. Prepare for production | Weeks 16-18 | Model deployment, API development and ATS integration | API response time < 200ms, 99.9% availability |
| 7. Pilot project | Weeks 18-26 | First trial the models in parallel without any effect on decisions, collect feedback and compare with human judgement | Recruiter satisfaction > 80%, growing usage |
| 8. Monitoring and adjustment | Ongoing | Track performance and data drift; retrain and audit for bias every quarter | AUC stays > 0.70, stable fairness metrics |
Measuring outcomes after hiring
Defining the target variable: quality of hire is modelled as a composite score or as assignment to one of two groups, for example higher and lower job performance.
| Category | Specific metrics | Data source | Weight |
|---|---|---|---|
| Job performance | Ratings after 6, 12 and 24 months, goal attainment and peer feedback | HRIS or performance review system | 40% |
| Retention | Retention after 12 months, regretted and non-regretted departures, voluntary resignations | Exit data in the HRIS | 25% |
| Time to productivity | Days to full productivity, completion of onboarding, manager's assessment | Onboarding system and surveys | 15% |
| Hiring manager satisfaction | Survey after 90 days, rating from 1-10, expectations met: yes/no | Post-hire survey | 10% |
| Collaboration and values | Engagement scores, shared values, teamwork rating | Employee surveys | 10% |
Example threshold: a total score ≥ 75/100 counts as high quality of hire and therefore as the positive class. At the time of the offer decision, the model should estimate the probability of this outcome.
Table of contents of the guide
Executive summary
- Costs and benefits of predictive analytics in recruiting.
- Research findings and model performance.
- Implementation plan and ROI.
Foundations of data-based hiring predictions
- What does quality of hire mean and why predict it?
- Machine learning basics for readers without a technical background.
- When predictive analytics is suitable and when it is not.
- Ethics and responsible use of AI.
Data strategy and infrastructure
- Building a shared data foundation for hiring.
- Data collection: sources, integrations and ETL processes.
- Data quality: cleaning, validation and completeness.
- Data protection and requirements under GDPR, CCPA and EEOC.
- Central feature stores and data versioning.
Defining quality of hire
- Outcome metrics for performance, retention and productivity.
- Comparing composite scores and individual metrics.
- Observation periods: outcomes after 6, 12, 18 and 24 months.
- Setting quality criteria for specific roles.
Engineering model features
- Applicant features from CVs, skills tests and interviews.
- Job features: role type, seniority, department and location.
- Process features: time to hire, touchpoints and recruiter notes.
- Taking account of feature interactions and domain knowledge.
- Handling missing values and encoding categories.
Developing models
- Selecting algorithms: logistic regression, Random Forests, XGBoost and neural networks.
- Splitting data into training, validation and test sets; running cross-validation.
- Tuning parameters with GridSearch or Bayesian optimisation.
- Ensemble methods and combining several models.
- Python code examples and Jupyter notebooks.
Evaluating and validating models
- Classification metrics: accuracy, precision, recall, AUC-ROC and F1.
- Calibration and choice of thresholds.
- Feature importance and explainability with SHAP and LIME.
- Validation methods: time-separated test data and A/B tests.
Fairness and bias reduction
- Forms of algorithmic bias in hiring.
- Fairness metrics: disparate impact, equal opportunity and demographic parity.
- Pre-processing: rebalancing the data set and resampling.
- Model training: fairness constraints and regularisation.
- Post-processing: reviewing and optimising thresholds by group.
- Ongoing bias audit procedures.
Deploying models in production
- Deployment via REST APIs and batch scoring.
- Integration with ATS and HRIS.
- Real-time scoring and response time optimisation.
- Versioning, rollbacks and parallel deployment environments.
Monitoring and maintenance
- Monitoring performance, response time and errors in production.
- Detecting and reporting changes in model quality.
- Schedule for retraining and incremental learning.
- Using feedback and improving models regularly.
Case studies and practical experience
- Technology company: 75% AUC for predicting quality of hire in engineering.
- Financial services: regretted turnover reduced by 35%.
- Healthcare: performance of new clinical hires improved by 40%.
- Lessons from implementation and common mistakes.
Request the full guide
44 pages of data analysis methods, Python code, model architectures and case studies.
More whitepapers
Identifying and reducing bias in recruiting
Research findings and practical methods for identifying and reducing bias in recruiting. With implementation guidance and the legal requirements.
ReadOrganising remote recruiting across locations and time zones
Recruiting methods for distributed teams and international hiring. With system comparisons, asynchronous interview workflows and notes on legal requirements.
ReadCalculating the costs and benefits of automated recruiting tasks
How to calculate investments in recruiting technology and present them to finance decision-makers. Includes an Excel ROI calculator and presentation templates for finance leadership.
Read