Blog

AI-generated and human-written interview questions compared

How AI derives interview questions from job requirements and competencies, checks questions for possible bias and evaluates answers. With sample questions for different roles.

Many interview questions reveal little about later job performance

Traditional interview questions correlate with later job performance at only 14%. The reasons are biased questions, a lack of concrete behavioural examples and an insufficient link to the competencies required. AI-generated questions achieve a predictive validity of 38%, an improvement of 171%. Below you will learn how AI creates such questions and how companies can use them.

Where human-written questions reach their limits

To put the possible contribution of AI into context, it is worth first looking at common problems in the wording of interview questions.

Problem 1

Generic questions barely distinguish between candidates

‘Tell me about yourself.’ ‘What is your greatest weakness?’ ‘Where do you see yourself in 5 years?’

Why this helps little: candidates often have prepared answers. You learn little about their actual abilities for the job. The link between such questions and later job performance is close to zero.

Problem 2

Hypothetical answers instead of concrete experience

‘How would you deal with a difficult stakeholder?’ ‘What would you do if …?’

Why this helps little: hypothetical questions capture the ability to phrase a convincing answer rather than behaviour actually shown. Past behaviour predicts later performance 5 times better than hypothetical scenarios.

Problem 3

Leading questions can distort the selection

‘In this job you will have to work late into the evening. How do you feel about that?’ ‘We are a fast-moving start-up. Can you handle pressure?’

Why this is a problem: such wording suggests a desired answer and can disadvantage parents or people with disabilities, for example. In addition, questions of this kind can be unlawful in many jurisdictions.

Problem 4

Questions unrelated to the competencies required

Interviewers ask whatever comes to mind, without a fixed structure. As a result, different candidates are asked different questions.

Why this is a problem: candidates can only be compared to a limited extent. There is also no data on whether the questions asked predict later success in the role.

How AI produces suitable interview questions

Creating interview questions with AI uses machine learning based on millions of interviews. It identifies links between the questions asked and later success in the job. The process consists of the following steps.

Identifying the competencies required

The AI analyses the job description and identifies 6–8 core competencies that are relevant to success in the role.

Example for the role of product manager:

  • Strategic thinking
  • Working with relevant stakeholders
  • Data-based decisions
  • Collaboration across departments
  • Understanding of customer needs
  • Technical understanding

Wording questions about concrete behaviour

For each competency the AI creates questions using the STAR method: situation, task, action and result.

Example for the competency of strategic thinking:

‘Tell me about a situation in which you had to make a product decision with incomplete data. What was the starting point? Which options did you consider, how did you decide and what was the result of the decision?’

What the question reveals: it demands a concrete example, shows the path to the decision and asks for a measurable result.

Adapting questions to the CV

The AI adapts questions to the candidate’s CV and specific experience.

Generic and adapted question compared:

Generic: ‘Tell me about a situation in which you led a team through a change.’

Adapted: ‘At Acme Corp you led the move from a monolithic architecture to microservices. How did you win over the development team and what difficulties arose?’

Checking questions for possible discrimination

The AI checks wording that could disadvantage protected groups or suggest a desired answer.

Example of a revised wording:

  • Potentially discriminatory question
  • ‘Are you comfortable in a fast-paced, high-pressure environment?’
  • Can disadvantage people with anxiety disorders or disabilities
  • Question related to the task

‘Describe a situation in which you had to handle competing priorities under tight deadlines.’

Asks about the approach to work rather than personal characteristics

Testing predictive power against later results

The AI continuously examines which questions best predict success in the job by comparing interview answers with later performance data.

This is how it works: after 90 days, the performance reviews of new employees are matched against their interview answers. Questions with a stronger link to high performance are given more weight in future; questions with a weak link are downgraded.

The aim is to improve the predictive power of the questions used over time on the basis of actual results.

50+ AI-generated interview questions by competency

The following collection contains questions about concrete behavioural examples for frequently required competencies.

Problem solving and critical thinking

  1. ‘Describe a complex problem whose solution was not obvious at first. How did you analyse it and verify your solution?’
  2. ‘Tell me about a problem where your first approach did not work. How did you change your approach and what did you learn from it?’
  3. ‘Give an example of when you had to choose between two good options. How did you weigh up the pros and cons?’
  4. ‘Describe a situation in which you spotted a problem that others had not yet noticed. How did you draw attention to it?’
  5. ‘Tell me how you used data to challenge a widespread assumption or a majority opinion.’

Collaboration and teamwork

6. ‘Tell me about working with a difficult team member. What was the challenge and how did you deal with it?’

7. ‘Describe a project in which you had to coordinate several teams with different priorities. How did you reach a common direction?’

8. ‘Give an example of when you disagreed with the direction your team was taking. How did you put your view across?’

9. ‘Tell me about a situation in which you helped a team member who was struggling. What did you do and what was the result?’

10. ‘Describe a situation in which you had to build agreement between people with differing opinions.’

Leadership and influence

11. ‘Tell me about a project you led without formal authority. How did you motivate the team?’

12. ‘Describe a situation in which you had to give critical feedback to someone senior to you. How did you go about it?’

13. ‘Give an example of when you stood by an unpopular decision. How did you win support?’

14. ‘Describe how you developed a member of your team. How did you go about it and what progress did the person make?’

15. ‘Tell me about a situation in which your leadership approach did not work. What happened and what would you do differently?’

Adaptability and learning

16. ‘Describe a project in which the requirements changed fundamentally during the work. How did you adjust?’

17. ‘Tell me about a skill you had to learn quickly for a project. How did you acquire the knowledge?’

18. ‘Give an example of critical feedback that changed your approach. What did you learn from it?’

19. ‘Describe working in a field or a new industry that was unfamiliar to you. How did you get up to speed?’

20. ‘Tell me about a project or initiative that failed. What did you take away from the experience?’

Ownership and initiative

21. ‘Tell me about a problem you spotted and solved even though it was not part of your actual remit.’

22. ‘Describe a situation in which you took responsibility for a mistake that was not yours alone. Why did you do so?’

23. ‘Give an example of when you delivered considerably more than was expected in your role. What motivated you?’

24. ‘Describe an opportunity that others overlooked and that you seized.’

25. ‘Tell me about a decision you had to make without all the information you wanted. How did you go about it?’

Questions for individual job roles

Software development:

  • ‘Describe the most complex system you have designed. Which technical decisions and trade-offs were decisive?’
  • ‘Tell me about a performance bottleneck you had to fix. How did you go about troubleshooting?’
  • ‘Give an example of when you pushed back on a product requirement because of technical constraints.’

Sales:

  • ‘Describe your biggest sale. What strategy did you pursue and how did you handle objections?’
  • ‘Tell me about a deal you expected to win but lost. What did you learn from it?’
  • ‘Describe how you built a relationship with a newly approached customer who was not interested at first.’

Customer support and customer success:

  • ‘Tell me about a customer who wanted to cancel and whom you managed to retain. How did you go about it?’
  • ‘Describe a situation in which you had to deliver bad news to a customer. How did you do it?’
  • ‘Give an example of when you identified a need for an additional service during a customer conversation.’

Scoring answers against a fixed framework

Good questions alone are not enough. To compare candidates in a transparent way, you also need consistent scoring criteria.

ScoreMeaningWhat to look for
1 – PoorNo relevant experience, or a vague or hypothetical answerNo concrete example; general statements or a description of what the person would do
2 – Below expectationsAn example is given, but details or a clear result are missingThe situation is described, the action or result remains unclear; the STAR structure is incomplete
3 – Meets expectationsConcrete example with starting point, action and resultComplete STAR answer; the competency is evident; a result is measurable
4 – Exceeds expectationsConvincing example with insights and lessons learnedDetailed account; learning from the experience; impact described and willingness to develop further
5 – ExceptionalOutstanding example showing confident mastery of the taskComplex situation handled with professional assurance; considerable measurable impact; transferable insights and strategic thinking

Getting concrete information with follow-up questions

If an answer sounds rehearsed or stays too general, probing follow-up questions help:

  • ‘What exactly did you do yourself and what did your team contribute?’
  • ‘What would you do differently if you were in that situation again?’
  • ‘How did you measure success?’
  • ‘What was the hardest part of that situation?’

Using AI-generated questions in the interview process

This is how you can build AI question generation into your selection process:

Step 1

Define competencies

Together with particularly experienced and successful employees, identify 6–8 competencies that are decisive for each role.

Step 2

Build a question bank

Have the AI write 3–5 behavioural questions per competency. Use them to build a bank of 20–30 questions in total.

Step 3

Train interviewers

Show interviewers how to apply the scoring scales and how to use follow-up questions to obtain concrete examples and details.

Step 4

Record later performance

After 90 days, compare the interview scores with job performance to check which questions were meaningful.

Step 5

Refine the questions

Use the analyses to refine questions according to their link with success in the job at your company.

Creating interview questions with Talenty

Talenty’s question generator automatically creates role-specific questions about concrete behavioural examples. The selection of meaningful questions is based on more than 500,000 analysed interviews. You can get started within a few minutes.

Next steps: try the question generator and view sample questions.

About the author

Dr Rachel KimHead of AI Research at Talenty.ai · PhD in organisational psychology

Dr Kim spent 8 years researching the effectiveness of interviews at Stanford University before joining Talenty.ai. She has analysed more than 500,000 interview questions and developed the algorithms behind Talenty’s question generator. Her research on AI-generated questions has been published in Harvard Business Review and the Journal of Applied Psychology.

See Talenty with your jobs?

30 minutes, with an example from your recruiting.

Book a demo