- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
this paper introduces InterviewPlayground, a simulation environment that compare models using 6 literature grounded behavioral traits (knowledge, understanding, reflexivity, memory, verbosity, disclosure) and a 200-item episodic memory bank (194 everyday background memories and 6 insights) retried through noisy cosine similarity.
Further introduced InterviewReportCard for eval, validated by 5 ai interviewer across 3 domains.
the authors recruited 450 human participants using Listen Labs to conduct AI-led interviews.
I appreciate that 15 real human studies were conducted to test Sim2Real predictive validity.
InterviewPlayground's simulated studies can predict quantitative evaluations of AI interviewers in human studies with an average correlation of 0.86 across 12 measures. 10 of them were strong; and understandable with comfort level and overall lower.
Critical strengths
-
the authors tested 5 distinct interviewer harness (interviewgpt, llmroleplay, minitalk, sparkme, storysage) across 3 domains against real human studies gives concrete evidence on which metrics transfer well and which are bad.
-
mixing 5 insights into 194 distractor memories with trait-dependent gaussian retrival noise is clever
-
accessible benchmark after switching to qwen 3.5 4b
Critical weaknesses
-
the r=0.86 correlation was computed by pooling all studies, we need to report also per-topic correlations
-
conversation turns, average response length, relevant response volume seems to be directly fit from the 15 human validation studies as the test set?
Questions for authors
-
can you report the per-topic correlations across the 15 studies?
-
what were the Sim2Real correslations before the post-hoc adjustment to target-population trait distributions?
-
would you clarify a) why m.sensitive and sensitive_count are unused? the exact human participant counts per cell in asian_american_politics?
-
can we run llm judge on the human transcripts as well and see what is the correlation between the llm judge and human self-reports on the human transcript alone?
-
for failure analysis, I would like to see percentage of how often it occurs (e.g., repeat questions) how often does the uncomfortable of discussing sensitive topics occur?
2 comments
Table of Per-Topic Correlations
| Measure | Asian American Politics | Obesity & Weight Management | GenAI Knowledge Work | All Studies | ||||
|---|---|---|---|---|---|---|---|---|
| r | 95% CI | r | 95% CI | r | 95% CI | r | 95% CI | |
| Relevant Response Volume | 0.99 | [0.79, 1.00] | 0.70 | [-0.48, 0.98] | 0.88 | [0.01, 0.99] | 0.74 | [0.37, 0.91] |
| Interview Guide Coverage | 1.00 | [0.94, 1.00] | 0.96 | [0.55, 1.00] | 0.97 | [0.62, 1.00] | 0.95 | [0.85, 0.98] |
| Novel Responses | 0.98 | [0.68, 1.00] | 0.87 | [-0.05, 0.99] | 0.99 | [0.86, 1.00] | 0.90 | [0.71, 0.97] |
| Coherence | 0.94 | [0.38, 1.00] | 0.99 | [0.88, 1.00] | 0.89 | [0.03, 0.99] | 0.95 | [0.84, 0.98] |
| Adaptiveness | 0.96 | [0.53, 1.00] | 0.91 | [0.16, 0.99] | 0.76 | [-0.36, 0.98] | 0.92 | [0.78, 0.97] |
| Leading Questions | 0.99 | [0.92, 1.00] | 0.97 | [0.61, 1.00] | 1.00 | [0.99, 1.00] | 0.98 | [0.93, 0.99] |
| Unclear Questions | 0.91 | [0.12, 0.99] | 0.89 | [0.06, 0.99] | 0.90 | [0.07, 0.99] | 0.85 | [0.60, 0.95] |
| Support and Rapport | 0.99 | [0.92, 1.00] | 1.00 | [0.94, 1.00] | 1.00 | [1.00, 1.00] | 1.00 | [0.99, 1.00] |
| Overall Experience | 0.72 | [-0.44, 0.98] | 0.78 | [-0.33, 0.98] | 0.76 | [-0.36, 0.98] | 0.67 | [0.25, 0.88] |
| Comfort Level | 0.70 | [-0.47, 0.98] | 0.75 | [-0.39, 0.98] | 0.67 | [-0.52, 0.98] | 0.60 | [0.13, 0.85] |
| Average Turns | 0.86 | [-0.08, 0.99] | 0.95 | [0.43, 1.00] | 1.00 | [1.00, 1.00] | 0.94 | [0.84, 0.98] |
| Average Response Length | 0.95 | [0.44, 1.00] | 0.80 | [-0.27, 0.99] | 0.91 | [0.12, 0.99] | 0.85 | [0.61, 0.95] |
Thank you for your thoughtful review and feedback! We are glad to hear that you liked our validation of Sim2Real predictive validity, concrete evidence on which metrics transfer well, the design of the memory retrieval in our cognitive model, and the accessibility of our benchmark. Below we address your questions and concerns:
Reporting Per-Topic Correlations
“the r=0.86 correlation was computed by pooling all studies, we need to report also per-topic correlations”
This is a good point! We provide these per-topic correlations in the comment below this one and have added them to the paper’s appendix. We note that because of the smaller samples when broken down by topic, the confidence intervals are wider than the confidence intervals for the full set of datapoints.
Clarification on Conversation Turns, Average Response Length, and Relevant Response Volume
“conversation turns, average response length, relevant response volume seems to be directly fit from the 15 human validation studies as the test set?”
We would like to clarify that this is not the case. We do not directly fit the conversation turns, average response length, or relevant response volume directly from the human studies. These are measures in InterviewReportCard that are used to characterize the performance of the AI interviewers. They are measured based on the interactions of participants (simulated or real) with the interviewers and are not adjusted post-hoc.
Clarification on the adjustment of trait distributions
“what were the Sim2Real correslations before the post-hoc adjustment to target-population trait distributions?”
After reviewing the paper, we see that the wording of “post-hoc adjustment” that we used was confusing. We were referring to the fact that the 3 presets had their trait distributions set based on the self-reported traits of the human populations that they’re simulating. However, this wording did not accurately represent that. We would like to emphasize two points to make this more clear:
We see that the original language in the paper was confusing, and we have updated it to make it more clear.
Question About Unused Variables and Table Counts
“would you clarify a) why m.sensitive and sensitive_count are unused? the exact human participant counts per cell in asian_american_politics?”
Thank you for spotting this. M.sensitive and sensitive_count are two unused variables in our codebase that were initially part of a different implementation of the simulator that was replaced with the current version. We will remove them from the code.
For your question about participant counts per cell in asian_american_politics: we are not sure what table you’re referring to. If you could clarify, then we are happy to provide this information.
Question About LLM Judge on Human Transcripts
“can we run llm judge on the human transcripts as well and see what is the correlation between the llm judge and human self-reports on the human transcript alone?”
We ran the LLM judges for comfort and overall experience on the 450 human transcripts and compared them to the self-reports for the same transcripts. We find that comfort level has a Pearson correlation of 0.396 and overall experience has a Pearson correlation of 0.503, which is lower than the correlation between the simulation and self-reports. We believe this difference may be due to the phenomena that we observe in 5.3 Experiment 2, where we find that when simulated participants encounter uncomfortable questions, they tend to express this verbally, but human participants often do not have any direct expression, instead opting to only report in the post-survey.
Additional Numbers for Failure Analysis in Experiment 2
“for failure analysis, I would like to see percentage of how often it occurs (e.g., repeat questions) how often does the uncomfortable of discussing sensitive topics occur?”
For 80 transcripts with low experience ratings that we analyzed to understand causes of poor experience, we find that 22 of the 40 human transcripts (55%) and 29 of the 40 simulated transcripts (72.5%) had clear instances of repeated questioning. For the 80 transcripts with low comfort ratings that we analyzed to understand causes of discomfort, we find that in 7 of 40 human transcripts (17.5%) and 17 of 40 simulated transcripts (42.5%) the participants directly indicate that they are not comfortable discussing a topic.