acceptodds
Under review as a conference paper at ICLR 2027

InterviewPlayground: A Simulation Environment for Evaluating AI Interviewers

Abstract

Increasingly, AI interviewers are being developed to elicit open-ended responses in applications like market research, public polling, preference elicitation, and social science research. However, evaluating AI interviewers is challenging because they function in extended, multi-turn interactions where they must adapt to participant behaviors. To address this need, we develop InterviewPlayground, a simulation environment for evaluating AI interviewers using simulated study participants whose behaviors are grounded in social theory. Simulated studies in InterviewPlayground produce an InterviewReportCard, which assesses the performance of AI interviewers using a suite of validated measures. To test whether our simulation-based evaluations predict performance with human participants, we conduct 15 real qualitative studies with five AI interviewers, three interview topics, and 450 human participants and compare them to simulated studies in InterviewPlayground. We find that AI interviewer performance in InterviewPlayground predicts performance in human studies with an average Pearson correlation of 0.86 across 12 measures, and the simulated interactions from InterviewPlayground reproduce key findings from behavioral analysis of AI interviewers in the human studies. Together, these findings support the validity of InterviewPlayground in assessing AI interviewer performance and examining potential failure modes. Our work contributes a simulation environment for AI interviewers supported with empirical validation, and more broadly, a roadmap for future work to develop validated, simulation-based evaluations of conversational AI systems.

Primary area:
datasets and benchmarks
Keywords:
AI Interviewers, User Simulation, Agent Evaluation, Ecological Validity, Conversational AI, Cognitive Modeling, Qualitative Research

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YbPvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

this paper introduces InterviewPlayground, a simulation environment that compare models using 6 literature grounded behavioral traits (knowledge, understanding, reflexivity, memory, verbosity, disclosure) and a 200-item episodic memory bank (194 everyday background memories and 6 insights) retried through noisy cosine similarity.

Further introduced InterviewReportCard for eval, validated by 5 ai interviewer across 3 domains.

the authors recruited 450 human participants using Listen Labs to conduct AI-led interviews.

I appreciate that 15 real human studies were conducted to test Sim2Real predictive validity.

InterviewPlayground's simulated studies can predict quantitative evaluations of AI interviewers in human studies with an average correlation of 0.86 across 12 measures. 10 of them were strong; and understandable with comfort level and overall lower.

Critical strengths
  • the authors tested 5 distinct interviewer harness (interviewgpt, llmroleplay, minitalk, sparkme, storysage) across 3 domains against real human studies gives concrete evidence on which metrics transfer well and which are bad.

  • mixing 5 insights into 194 distractor memories with trait-dependent gaussian retrival noise is clever

  • accessible benchmark after switching to qwen 3.5 4b

Critical weaknesses
  1. the r=0.86 correlation was computed by pooling all studies, we need to report also per-topic correlations

  2. conversation turns, average response length, relevant response volume seems to be directly fit from the 15 human validation studies as the test set?

Questions for authors
  1. can you report the per-topic correlations across the 15 studies?

  2. what were the Sim2Real correslations before the post-hoc adjustment to target-population trait distributions?

  3. would you clarify a) why m.sensitive and sensitive_count are unused? the exact human participant counts per cell in asian_american_politics?

  4. can we run llm judge on the human transcripts as well and see what is the correlation between the llm judge and human self-reports on the human transcript alone?

  5. for failure analysis, I would like to see percentage of how often it occurs (e.g., repeat questions) how often does the uncomfortable of discussing sensitive topics occur?

2 comments

Thank you for your thoughtful review and feedback! We are glad to hear that you liked our validation of Sim2Real predictive validity, concrete evidence on which metrics transfer well, the design of the memory retrieval in our cognitive model, and the accessibility of our benchmark. Below we address your questions and concerns:

Reporting Per-Topic Correlations
“the r=0.86 correlation was computed by pooling all studies, we need to report also per-topic correlations”

This is a good point! We provide these per-topic correlations in the comment below this one and have added them to the paper’s appendix. We note that because of the smaller samples when broken down by topic, the confidence intervals are wider than the confidence intervals for the full set of datapoints.

Clarification on Conversation Turns, Average Response Length, and Relevant Response Volume
“conversation turns, average response length, relevant response volume seems to be directly fit from the 15 human validation studies as the test set?”
We would like to clarify that this is not the case. We do not directly fit the conversation turns, average response length, or relevant response volume directly from the human studies. These are measures in InterviewReportCard that are used to characterize the performance of the AI interviewers. They are measured based on the interactions of participants (simulated or real) with the interviewers and are not adjusted post-hoc.

Clarification on the adjustment of trait distributions
“what were the Sim2Real correslations before the post-hoc adjustment to target-population trait distributions?”

After reviewing the paper, we see that the wording of “post-hoc adjustment” that we used was confusing. We were referring to the fact that the 3 presets had their trait distributions set based on the self-reported traits of the human populations that they’re simulating. However, this wording did not accurately represent that. We would like to emphasize two points to make this more clear:

  1. There were no correlations “before the post-hoc adjustment” because we could not run the simulation without setting these trait values.
  2. By design, the traits were set before any simulations were run and any comparisons to humans were made. No adjustments were made after the initial comparison, and the correlations reported in the paper are the first and only results found.

We see that the original language in the paper was confusing, and we have updated it to make it more clear.

Question About Unused Variables and Table Counts
“would you clarify a) why m.sensitive and sensitive_count are unused? the exact human participant counts per cell in asian_american_politics?”

Thank you for spotting this. M.sensitive and sensitive_count are two unused variables in our codebase that were initially part of a different implementation of the simulator that was replaced with the current version. We will remove them from the code.

For your question about participant counts per cell in asian_american_politics: we are not sure what table you’re referring to. If you could clarify, then we are happy to provide this information.

Question About LLM Judge on Human Transcripts
“can we run llm judge on the human transcripts as well and see what is the correlation between the llm judge and human self-reports on the human transcript alone?”

We ran the LLM judges for comfort and overall experience on the 450 human transcripts and compared them to the self-reports for the same transcripts. We find that comfort level has a Pearson correlation of 0.396 and overall experience has a Pearson correlation of 0.503, which is lower than the correlation between the simulation and self-reports. We believe this difference may be due to the phenomena that we observe in 5.3 Experiment 2, where we find that when simulated participants encounter uncomfortable questions, they tend to express this verbally, but human participants often do not have any direct expression, instead opting to only report in the post-survey.

Additional Numbers for Failure Analysis in Experiment 2
“for failure analysis, I would like to see percentage of how often it occurs (e.g., repeat questions) how often does the uncomfortable of discussing sensitive topics occur?”

For 80 transcripts with low experience ratings that we analyzed to understand causes of poor experience, we find that 22 of the 40 human transcripts (55%) and 29 of the 40 simulated transcripts (72.5%) had clear instances of repeated questioning. For the 80 transcripts with low comfort ratings that we analyzed to understand causes of discomfort, we find that in 7 of 40 human transcripts (17.5%) and 17 of 40 simulated transcripts (42.5%) the participants directly indicate that they are not comfortable discussing a topic.

Table of Per-Topic Correlations

MeasureAsian American PoliticsObesity & Weight ManagementGenAI Knowledge WorkAll Studies
r95% CIr95% CIr95% CIr95% CI
Relevant Response Volume0.99[0.79, 1.00]0.70[-0.48, 0.98]0.88[0.01, 0.99]0.74[0.37, 0.91]
Interview Guide Coverage1.00[0.94, 1.00]0.96[0.55, 1.00]0.97[0.62, 1.00]0.95[0.85, 0.98]
Novel Responses0.98[0.68, 1.00]0.87[-0.05, 0.99]0.99[0.86, 1.00]0.90[0.71, 0.97]
Coherence0.94[0.38, 1.00]0.99[0.88, 1.00]0.89[0.03, 0.99]0.95[0.84, 0.98]
Adaptiveness0.96[0.53, 1.00]0.91[0.16, 0.99]0.76[-0.36, 0.98]0.92[0.78, 0.97]
Leading Questions0.99[0.92, 1.00]0.97[0.61, 1.00]1.00[0.99, 1.00]0.98[0.93, 0.99]
Unclear Questions0.91[0.12, 0.99]0.89[0.06, 0.99]0.90[0.07, 0.99]0.85[0.60, 0.95]
Support and Rapport0.99[0.92, 1.00]1.00[0.94, 1.00]1.00[1.00, 1.00]1.00[0.99, 1.00]
Overall Experience0.72[-0.44, 0.98]0.78[-0.33, 0.98]0.76[-0.36, 0.98]0.67[0.25, 0.88]
Comfort Level0.70[-0.47, 0.98]0.75[-0.39, 0.98]0.67[-0.52, 0.98]0.60[0.13, 0.85]
Average Turns0.86[-0.08, 0.99]0.95[0.43, 1.00]1.00[1.00, 1.00]0.94[0.84, 0.98]
Average Response Length0.95[0.44, 1.00]0.80[-0.27, 0.99]0.91[0.12, 0.99]0.85[0.61, 0.95]
Reviewer_V9XJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2(−1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces InterviewPlaygroun,d a simulation environment for evaluating AI interviewers using simulated study participants. This simulation environment has been tested with actual humans showing average correlation to predicting performance in human studies about 0.86.

Critical strengths

This work provides a simulation environment which can be used for multiple important problems.

The validation of the simulation environment seems convincing since the pearson correlation is high and the number of human participants seems sufficient.

Critical weaknesses

High correlation could be shown because of other confounding factors rather the interviewers' actual characteristics.

Components of the InterviewPlayground system seems to lack of sufficient ablation studies to understand what actually is important.

Questions for authors

Please answer for Critical Weaknesses.

Minor editorial comment : Please check the ICLR template.

1 comment

Thank you for your feedback! We are glad to hear that you believe our simulation environment can be used for multiple important problems and has convincing human validation. We address your concerns and ask for additional clarification below:

Confounding Factors
“High correlation could be shown because of other confounding factors rather the interviewers' actual characteristics.”

In experiment 1, we find high correlations across 10 different measures covering different aspects of the participant responses, interviewer behavior, and conversation length. In addition to the quantitative evidence from experiment 1, we present qualitative evidence in experiment 2 demonstrating how different interviewer behaviors influence evaluation measures such as participant experience, interview guide coverage, and relevant response volume. We also highlight cases where our simulation underperforms (e.g., not accurately reproducing reactions to discomfort). If you believe that there are important confounding factors in our results that are not addressed in the paper, we would appreciate it if you could specify them, so we can address them appropriately.

Additional Ablation Studies
“Components of the InterviewPlayground system seems to lack of sufficient ablation studies to understand what actually is important.”

We agree that additional ablation studies can help support and clarify results, but we note that they do incur significant cost. We currently estimate that running a single ablation with the 5 interviewers would take 10 hours and cost about $330. Based on this consideration and your feedback, we have planned an ablation that clarifies the usefulness of the cognitive model by running the studies using simulated participants without the 6 traits, and we will provide results when this experiment is finished. However, as we are working on this experiment, we would appreciate it if you could clarify if this ablation would address your concerns, and if not, what ablation you would like to see to, considering that we are limited in the number that we can feasibly run.

Comment on ICLR Template
“Minor editorial comment : Please check the ICLR template.”

Thank you for this suggestion. We reviewed the template and have updated our table captions. If you were referring to a different aspect of the template, please let us know.

Reviewer_v6dbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper present InterviewPlayground, a simulated environment for testing AI interview systems. This is timely work given the expansion of AI interview systems across a range of domains (the authors focus more on qualitative survey usage, but AI systems are also increasingly being used for job interviews and related). So, I think the range of research and practitioners who would benefit from InterviewPlayground is quite broad and large.

The work is thorough in that it simulates qualitative research comprehensively through demographic personas, behavioral traits, and other dimensions like verbosity. So, it's nicely grounded in social science measurement dimensions. The resulting report card is also nicely comprehensive, including interviewer behavior and participant experience dimensions. It's always hard to address every dimension and thus the extent of generalization always remains somewhat in question, but I do think the dimensions used for the human eval comparisons (which were quite robust) capture the "head of curve" dimensions.

Finally, the paper presents some very useful dimensions that InterviewPlayground can uncover, such as causes of poor performance and how to best elicit information (section 5.3).

Critical strengths
  1. Substantial human validation. 450 actual human participants were used in the human comparison studies.
  2. Somewhat general comment, but I really like that the breakdown of how a participant would respond into the dimensions chosen - traits, memory constructions, etc. That's another way of saying this is nicely motivated by social science measurement.
  3. Human-LLM agreement was high (appendix c). Having done a fair amount of this, human-LLM agreement on complex dimensions like these is hard to get to that .6 - .7 range (and you can see that in the human-human agreement), so these look good and are reassuring.
  4. Good robustness checks with different models.
  5. Practical value: identifying dimensions that, for instance, can best elicit a response, is super useful.
Critical weaknesses
  1. Maybe a few minor things around operationalization of metrics. For instance, novelty of response is relative to the interviewer guide not genuine novelty. Other metrics may have some confounds, like counts of rapport and leading questions may depend on interview length.
Questions for authors
  1. I liked the practical value of the experiement 2 (section 5.3), especially around techniques for eliciting responses. Are there experiments that could be run, maybe in follow up, to better quantify the effects of these? For instance, you could include, not include, and then vary the degree of role playing to see the effect on the amount and nature of the responses.
1 comment

Thank you for your thoughtful review! We are encouraged by your assessment that our work is timely, would benefit a large range of researchers and practitioners, is well-grounded in social science measurement, has substantial human validation, and provides practical value. Below we address your questions and concerns:

Clarification on Novel Responses
“novelty of response is relative to the interviewer guide not genuine novelty”

You rightly point out that there can be different notions of novelty. Our measure captures novelty relative to the interview guide rather than novelty relative to a broader scientific knowledge base. We have updated the InterviewReportCard section to make this distinction more clear.

Confounding Between Count-based Metrics and Interview Length
“counts of rapport and leading questions may depend on interview length”

Though there is a relationship between the counts of leading questions and support and rapport building statements, InterviewPlayground’s simulations are time-restricted, which controls the length of the interviews to give all interviewers the same time-limit. We believe that the counts of support and rapport statements and leading questions provide interpretable insight into how interviewers choose to use that limited time.

Follow-up Work on Evaluating Interviewing Techniques
“I liked the practical value of the experiement 2 (section 5.3), especially around techniques for eliciting responses. Are there experiments that could be run, maybe in follow up, to better quantify the effects of these? For instance, you could include, not include, and then vary the degree of role playing to see the effect on the amount and nature of the responses.”

This is a good point! We believe InterviewPlayground will enable these types of follow-up studies by allowing for controlled testing of different interview techniques such as the degree of role-playing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.