Whitepaper

Examining quality of hire with data and predictive models

How to use data analysis and machine learning to estimate later job performance. Includes methods for model development and feature engineering, plus code examples in Python.

Methodology

Data records analysed: 3.2M+.

Applicant records

Analysis of data from several years

Study period: 2018-2024 (6 years). The research partners are Stanford AI Lab, MIT CSAIL and Carnegie Mellon Machine Learning Dept. The work was peer-reviewed in the Journal of Applied Psychology and at the ACM Conference on Fairness, Accountability, and Transparency.

180
Predictive models developedFor a range of industries and roles
450+
Companies in the studyFrom start-ups to Fortune 500 companies

Machine learning results

Prediction accuracy

Prediction accuracy: 73%.

Trained models predict quality of hire after 12 months with 73% accuracy. Conventional interviews alone reach 54%.

  • Combined XGBoost and Random Forest models: 73-76% accuracy.
  • Neural networks: 68-71% accuracy.
  • Traditional scorecards: 54-58% accuracy.
  • Human intuition alone: 48-52%, only slightly above chance.

Improvement in quality of hire

Change in quality of hire: +42%.

Companies using predictive analytics improve their quality-of-hire metrics by 42%. The measures were performance ratings, retention and hiring manager satisfaction.

  • Performance rating in the first year: 4.2/5 compared with 3.0/5 for conventional methods.
  • Hiring manager satisfaction after 90 days: 89% compared with 67%.
  • Retention rate after 12 months: 91% compared with 76%.
  • Time to full productivity: 35% shorter.

Highly predictive features

Highly predictive features: 12.

12 features show the highest predictive power across roles and industries. Work samples and structured assessments carry more weight than CV features.

  • Work sample result: feature importance 0.42.
  • Structured interview rating: feature importance 0.38.
  • Cognitive ability tests: feature importance 0.31.
  • CV features such as years of experience and education: feature importance 0.14, which is weak.

Reducing bias

Change in demographic bias: -58%.

Models trained with fairness constraints and audited regularly show 58% less demographic bias than unstructured human decisions.

  • Gender bias: -62% when optimising for equal opportunity.
  • Bias by ethnicity: -54% with a demographic parity constraint.
  • Age bias: -51% with calibration across protected groups.
  • Regular audits: examine model fairness every quarter.

Return on investment and savings

Annual savings: $3.8M.

Companies with 500-2,000 employees save an average of $3.8M a year. The causes are better hires, fewer mis-hires, shorter onboarding and lower staff turnover.

  • Mis-hire costs avoided: $1.8M for 15 mis-hires avoided at $120K each.
  • Productivity gain: $1.4M through shorter onboarding and better performance.
  • Lower turnover: $600K with 8% fewer regretted departures.
  • Implementation costs: $180-350K in the first year; $80-150K ongoing.

Data requirements

Minimum size of the data set: 500+ hires.

Training requires at least 500 past hires with later outcome data. More than 2,000 records and more than 18 months of observation are recommended.

  • Minimum data set: 500 hires with outcomes after 12 months.
  • Recommended data set: more than 2,000 hires tracked over 18-24 months.
  • Data quality before quantity: use cleaned and structured data.
  • If you lack data of your own: consider transferring models from comparable companies.

Methods included

Data infrastructure

An architecture for collecting, cleaning and structuring hiring data. It includes a schema for a central analytics database (data warehouse), procedures for extracting and preparing data (ETL) and central stores for model features.

Developing models

Step-by-step sequences with Python code examples: engineering features, selecting algorithms, tuning model parameters and combining several models.

Evaluating and monitoring

Metrics for model evaluation, methods for bias audits, monitoring in live operation and regular retraining.

Steps for developing a predictive model

PhasePeriodTasksSuccess criteria
1. Collect dataWeeks 1-4Extract past hiring data, define outcome variables and check data quality500+ records, <10% missing data, defined outcomes
2. Engineer featuresWeeks 4-8Derive features from raw data, handle missing values, encode categories50-200 features, checked for unintended use of later outcome information
3. Baseline modelsWeeks 8-10Train simple models such as logistic regression and decision trees; establish baseline performanceAUC > 0.60, better than random assignment
4. Advanced modelsWeeks 10-14Train Random Forest, XGBoost and LightGBM; optimise parametersAUC > 0.70, balanced precision and recall
5. Audit for biasWeeks 14-16Check results by demographic group and apply mitigation measuresDisparate impact ratio < 1.25; equal opportunity achieved
6. Prepare for productionWeeks 16-18Model deployment, API development and ATS integrationAPI response time < 200ms, 99.9% availability
7. Pilot projectWeeks 18-26First trial the models in parallel without any effect on decisions, collect feedback and compare with human judgementRecruiter satisfaction > 80%, growing usage
8. Monitoring and adjustmentOngoingTrack performance and data drift; retrain and audit for bias every quarterAUC stays > 0.70, stable fairness metrics

Measuring outcomes after hiring

Defining the target variable: quality of hire is modelled as a composite score or as assignment to one of two groups, for example higher and lower job performance.

CategorySpecific metricsData sourceWeight
Job performanceRatings after 6, 12 and 24 months, goal attainment and peer feedbackHRIS or performance review system40%
RetentionRetention after 12 months, regretted and non-regretted departures, voluntary resignationsExit data in the HRIS25%
Time to productivityDays to full productivity, completion of onboarding, manager's assessmentOnboarding system and surveys15%
Hiring manager satisfactionSurvey after 90 days, rating from 1-10, expectations met: yes/noPost-hire survey10%
Collaboration and valuesEngagement scores, shared values, teamwork ratingEmployee surveys10%

Example threshold: a total score ≥ 75/100 counts as high quality of hire and therefore as the positive class. At the time of the offer decision, the model should estimate the probability of this outcome.

Table of contents of the guide

Executive summary

  • Costs and benefits of predictive analytics in recruiting.
  • Research findings and model performance.
  • Implementation plan and ROI.
Chapter 1

Foundations of data-based hiring predictions

  • What does quality of hire mean and why predict it?
  • Machine learning basics for readers without a technical background.
  • When predictive analytics is suitable and when it is not.
  • Ethics and responsible use of AI.
Chapter 2

Data strategy and infrastructure

  • Building a shared data foundation for hiring.
  • Data collection: sources, integrations and ETL processes.
  • Data quality: cleaning, validation and completeness.
  • Data protection and requirements under GDPR, CCPA and EEOC.
  • Central feature stores and data versioning.
Chapter 3

Defining quality of hire

  • Outcome metrics for performance, retention and productivity.
  • Comparing composite scores and individual metrics.
  • Observation periods: outcomes after 6, 12, 18 and 24 months.
  • Setting quality criteria for specific roles.
Chapter 4

Engineering model features

  • Applicant features from CVs, skills tests and interviews.
  • Job features: role type, seniority, department and location.
  • Process features: time to hire, touchpoints and recruiter notes.
  • Taking account of feature interactions and domain knowledge.
  • Handling missing values and encoding categories.
Chapter 5

Developing models

  • Selecting algorithms: logistic regression, Random Forests, XGBoost and neural networks.
  • Splitting data into training, validation and test sets; running cross-validation.
  • Tuning parameters with GridSearch or Bayesian optimisation.
  • Ensemble methods and combining several models.
  • Python code examples and Jupyter notebooks.
Chapter 6

Evaluating and validating models

  • Classification metrics: accuracy, precision, recall, AUC-ROC and F1.
  • Calibration and choice of thresholds.
  • Feature importance and explainability with SHAP and LIME.
  • Validation methods: time-separated test data and A/B tests.
Chapter 7

Fairness and bias reduction

  • Forms of algorithmic bias in hiring.
  • Fairness metrics: disparate impact, equal opportunity and demographic parity.
  • Pre-processing: rebalancing the data set and resampling.
  • Model training: fairness constraints and regularisation.
  • Post-processing: reviewing and optimising thresholds by group.
  • Ongoing bias audit procedures.
Chapter 8

Deploying models in production

  • Deployment via REST APIs and batch scoring.
  • Integration with ATS and HRIS.
  • Real-time scoring and response time optimisation.
  • Versioning, rollbacks and parallel deployment environments.
Chapter 9

Monitoring and maintenance

  • Monitoring performance, response time and errors in production.
  • Detecting and reporting changes in model quality.
  • Schedule for retraining and incremental learning.
  • Using feedback and improving models regularly.
Chapter 10

Case studies and practical experience

  • Technology company: 75% AUC for predicting quality of hire in engineering.
  • Financial services: regretted turnover reduced by 35%.
  • Healthcare: performance of new clinical hires improved by 40%.
  • Lessons from implementation and common mistakes.

Authors

Dr. Rajesh KumarPrincipal Research Scientist, Stanford AI Lab • Lead author

Rajesh researches fair and explainable machine learning in people analytics. His work on algorithmic bias has been featured in Nature, Science and ACM FAccT. He holds a PhD in Computer Science from Stanford University and previously held a leadership role at LinkedIn Talent Analytics.

Dr. Sophia LeeAssociate Professor, MIT CSAIL • Machine learning researcher

Sophia works on applied machine learning for decision-making in organisations. She has published more than 30 papers on predictive HR analytics and advised Fortune 500 companies on implementation. She holds a PhD from MIT and received an NSF CAREER Award.

Marcus BrownHead of People Analytics, Stripe • Practitioner contributions

Marcus built a hiring prediction platform at Stripe handling more than 200K applicants a year. He contributes experience with production ML systems, data infrastructure and the organisational adoption of AI.

Request the full guide

44 pages of data analysis methods, Python code, model architectures and case studies.

Request by email
Book a demo