- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a benchmark for evaluating whether LLM agents can detect belief-reality mismatch in user-agent interactions and whether agents translate such inferences into better downstream solutions rather than literally executing the flawed request.
They find that explicit mental-state elicitation improves Solution quality beyond generic CoT; mismatched mental-state info hurts; Belief accuracy is more "stubborn" than Solution quality as reasoning strength increases (dimension decoupling); and RL training with belief/profile-specific rewards improves Solution scores across domains with manageable variance.
Critical strengths
Most "does the model understand mental states" papers jump straight to synthetic constructions. Here, the SWE-Chat annotation study (Appendix A) independently establishes that the belief-reality mismatch problem is frequent and consequential in real agent transcripts, and that current agents handle it poorly (19.1% success). This gives the subsequent benchmark a motivational anchor that is empirically justified, not just intuitive.
The authors go beyond releasing a static benchmark: human-LLM judge agreement, a 5-run variance study for the RL model, causal interventions with mismatched-information controls (Figure 3), and transfer to two independent, previously published multi-agent benchmarks (HiddenBench, AgentWebBench) that were not designed for this paper. This triangulation substantially raises confidence in the paper.
Critical weaknesses
-
W1 The benchmark's ground truth is generated and graded by LLMs in a closed loop. The "ground truth" belief and profile in BASIS are generated by an LLM, checked for quality by another LLM, and scored during evaluation by a third LLM judge. The RL reward function then optimizes directly against those same three scored dimensions. No step in this pipeline is validated against real human belief data. So when the RL-trained model scores higher, the paper cannot rule out that it learned to match the judge's preferred answer style, rather than genuinely better mental-state reasoning.
-
W2 The judge model is also one of the models being evaluated. Qwen3.8-27B is used as the automated judge in Table 3, but it is also one of the six models being scored, and it performs especially well on some dimensions (e.g., Profile in SWE). LLM judges are known to favor outputs similar to their own style (a point the paper itself cites, Zheng et al., 2023). The authors do not cross-check Qwen3.8-27B's scores with a different judge to rule out this bias.
Questions for authors
-
Can you report inter-rater agreement/sample size details for the SWE-Chat annotation study in Appendix A (number of independent annotators per trajectory, agreement statistics), separate from the Table 2 agreement study for the main benchmark? Currently it reads as single-annotator labels.
-
Do you have any evidence (even a small-scale study) that the LLM-synthesized beliefs/profiles in BASIS resemble beliefs real users actually hold in comparable mismatch situations, beyond being "inferable" and "realistic" per the validation rubric?
-
Was Qwen3.8-27B's score in Table 3 cross-checked with a different judge model to rule out self-preference bias, given it serves as both judge and evaluated model?
2 comments
Question 1: SWE-Chat Annotation Details
Thank you for requesting further details about the SWE-Chat annotation study in Appendix A. The initial study was annotated by one annotator. To assess its reliability, we subsequently obtained independent labels from a second annotator on the same trajectories and calculated percentage agreement and Cohen's kappa for each of the four annotation dimensions.
Table 3: Inter-annotator agreement in the SWE-Chat study.
| Dimension | Agreement (%) | Cohen's κ |
|---|---|---|
| Instruction quality | 90.0% | 0.843 |
| Mental state | 89.4% | 0.818 |
| Literal execution | 95.7% | 0.912 |
| Agent gap handling | 89.4% | 0.813 |
Across the four dimensions, agreement ranges from 89.4% to 95.7%, and Cohen's κ ranges from 0.813 to 0.912. In particular, the Mental state category achieves 89.4% agreement, supporting the reproducibility of judgments about whether instruction gaps relate to users' limited observations or task-relevant mental states. Instruction quality was annotated on all 100 trajectories; the other three dimensions were assessed on the 47 trajectories identified as containing instruction gaps. We will report these additional annotation details and agreement statistics in Appendix A.
Thank you again for your constructive feedback. We hope these clarifications and additional analyses address your concerns.
Thanks for your review. We respond to each point below.
Weakness 1 & Question 2: Realism of Generated Beliefs and Profiles
We appreciate your concern about whether the synthesized beliefs and profiles reflect plausible user states. Although LLMs are used in our data construction and evaluation pipeline, the scenarios are not generated without task grounding. As illustrated in Figure 2 and detailed in Section 4.1, we derive the underlying task states and task-relevant profiles from existing benchmark datasets, then synthesize the observations, beliefs, and requests that create a belief–reality mismatch. For example, Profile in Preference scenarios build on existing preference data, mismatch in Culture scenarios build on cultural interpretation tasks, and SWE scenarios are grounded in software-engineering problems and their reference solutions like gold patches.
We also conduct human evaluation of the final generated instances, in addition to LLM-based validation. As reported in Table 1 and Appendix B.3, human annotators assess the realism and consistency of those generated items. These evaluations provide evidence that the generated scenarios are generally plausible and internally consistent rather than relying solely on LLM judgments. Figure 1 and the detailed examples in Appendix F further illustrate the intended mismatch settings.
Weakness 2 & Question 3: Potential Bias from Using Qwen3.8-27B as a Judge
Thank you for pointing out the potential self-preference bias. To investigate judge dependence, we additionally use DeepSeek as an independent judge and compare its scores with those assigned by Qwen3.8-27B. We evaluate responses produced by both Qwen3.8-27B and Gemini3.8-Flash across all four domains, examining the Pearson correlation between the two judges' instance-level scores for Belief, Profile, and Solution.
Across all paired settings, the overall Pearson correlations between the two judges for Qwen3.8-27B responses are 0.668 (Belief), 0.718 (Profile), and 0.791 (Solution). For Gemini3.8-Flash responses, the corresponding correlations are 0.687, 0.713, and 0.754. Thus, the correlations are broadly comparable for the two evaluated response sources, without a clear systematic advantage in correlation for Qwen's own outputs. These results provide an additional cross-judge consistency check.