- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a pipeline for evaluating the accuracy and consistency of the beliefs that LLM agents hold about environment states and task progress in partially observable environments. An authoring LLM generates proposition templates and evaluation rules in four categories, namely World, Epistemic, Affordance, and Decision, which are grounded turn by turn in the agents' actual trajectories. Reference labels are computed with deterministic LLM-written rules in ALFWorld and WebShop and with a Bayesian observer in the authors' Noisy Cabinets environment, and are discretized into true, false, and unknown. The model's self-reported belief is read from the next-token probabilities of the answer letters after appending a question to the same prefix. Across six model conditions, the paper measures permutation, negation, sequential, and rephrasing consistency as well as accuracy, and claims that Affordance and Decision accuracy are positively associated with task success and that a response-tendency model explains why negation consistency and accuracy decline as the context grows.
Critical strengths
S1. The paper addresses an important evaluation gap. Prior work has mostly examined whether elicited confidence agrees with behavior and has paid less attention to whether the beliefs are actually correct. This paper queries beliefs at exactly the same prefix of the agent's own trajectory, compares them with references, and measures several consistency relations at the same time. This is a useful direction for diagnosing agent failures.
S2. The measurement design handles Unknown responses carefully. By including an Unknown option and showing how consistency and accuracy change under different treatments of Unknown answers, the paper reveals that the measurements are strongly affected by the tendency to answer Unknown. This is a useful lesson for future work on belief evaluation.
S3. Noisy Cabinets separates world truth from what can be inferred from observations. Building an environment with observation noise to distinguish world truth from references inferable from the interaction history is a serious attempt to define what the correct answer should be in belief evaluation.
Critical weaknesses
W1. The reference labels are not matched to the information available to the model, which limits what accuracy means. As the paper itself states, the observer uses information that cannot be known from the interaction history alone, such as the numerical probabilities of stochastic events and the set of possible objects. When the model answers Unknown while the reference is true or false, the answer may therefore reflect legitimate uncertainty rather than an error. The current evaluation cannot separate such cases from overly cautious answering, and the results are highly sensitive to how they are treated: AF-RL's accuracy, for example, ranges from 34.82% to 83.07% depending on whether they are counted. The authors acknowledge the observer's information advantage, but an information-matched observer that uses only the information available to the model is needed to make the accuracy numbers interpretable.
W2. There is insufficient evidence that the elicited beliefs are the ones that drive the agent's decisions. As the authors' commonsense-assertion experiment shows, the measurements change substantially merely by changing the instruction, which suggests that they partly reflect how the question is interpreted. More importantly, the paper does not check whether the reported beliefs are consistent with the agent's own behavior. A natural test would be whether the agent judges the action it actually selected to be feasible, or how often it selects actions it reports as infeasible. Without such a link, it is difficult to treat the measurements as the agent's operative beliefs rather than as answers to a separate question.
W3. The main conclusions are stated more strongly than the evidence supports. The association between accuracy and task success is a comparison among 18 model-environment conditions with R² values of 0.168 and 0.075, yet the paper describes it as "strongly predictive." The robustness checks that omit individual models or environments are helpful, but each model is evaluated on its own trajectories, so context length and the visited states differ across conditions and remain potential confounders. The response-tendency analysis shows useful statistical regularities, including the transfer of the dependence structure to held-out tasks, but the decomposition itself is an identity and does not establish the claimed fixed-response mechanism or explain why the error rates grow with context. The consistency relations themselves follow prior work such as Elazar et al., BECEL, and BeliefBank, so the contribution lies in the measurement framework rather than in new findings.
Questions for authors
Q1. How does accuracy change if the reference observer is restricted to the information available to the model? Among the current mismatches between Unknown reports and definite references, what fraction would an information-matched observer also label Unknown?
Q2. Are the self-reported beliefs consistent with the agent's own actions? For example, how often does the agent judge the action it actually selected to be feasible, and how often does it select actions it reports as infeasible?
Q3. Does the association between accuracy and task success hold in an episode-level analysis that controls for model fixed effects and context length, or in a setting where all models answer at the same prefixes?
3 comments
Thank you for the thorough response and the additional analyses. They address most of my concerns, and I have raised my score to 3. I encourage the authors to include these analyses in the revised paper.
Thank you for your follow-up and for recognizing our rebuttal efforts. We truly appreciate your time and the positive update. We have already include the new analysis, tables and figures into our revised manuscript and will further integrate all of them into the future camera-ready version.
We sincerely thank the reviewer for the thorough, constructive review and for recognizing the same-prefix design, Unknown handling, and Noisy Cabinets.
W1 & Q1: an information-matched observer
For most of our data the references exactly match the model's information. In deterministic ALFWorld and WebShop, the rules read only the model's prompt and assign U when evidence is missing (Section 3.3). Only NC has extra priors information, and there we built the restricted observer you suggest, which relabels all five posterior-based families as U (Appendix E.1, Table 15; %):
On your first question, accuracy hardly changes: Overall moves by at most 2.5 points and T/F-only by at most 0.5, and the ranking holds. On your second, 85.8% of TU/FU occur in ALFWorld + WebShop, where there is no information gap, and the restricted observer would also label only 6.8% of all TU/FU Unknown. As for AF-RL, it answers Unknown on 70.1% of the T/F propositions that its own prompt settles. Dropping NC's TU/FU lifts its accuracy only from 34.82% to 38.19%, so the gap to 83.07% reflects its tendency to answer Unknown, not the observer's priors.
W2 & Q2: elicited beliefs and the agent's own actions
As proposed, we add a test to match each prefix's Affordance report against the action then taken (Section 4.3, Table 4; Appendix E.5, %):
Models judge the action they then take feasible in 70.0–98.9% of cases, and only 0.0–1.3% of the actions they report infeasible are then taken.
On the commonsense experiment, the two prompts differ in the evidence the model may use. Under the interaction-only prompt, U is the correct answer to a fact absent from the interaction, so the change shows models following the stated scope. All our evaluations use one fixed prompt, under which reports match actions.
W3 (a): claim strength
By "strongly predictive" we meant that the association is statistically robust (95% CIs > 0 in Figure 3; Table 16), not that R2 is high. We modify our results and it now says "general correlates of task success."
W3 (b) & Q3: confounders, episode level, and same prefixes
W3 (c): decomposition and the response-tendency model
Our claim is the response-tendency model: the errors that grow with context attach to statement polarity. Appendix F.6 (Tables 22–24) fits fact error (flips all reports on ϕ), response tendency (one polarity), and random error (per reply) to accuracy changes alone, predicting first-to-last-bin agreement changes (pp, Table 23):
A fact error model predicts no change, a random error model predicts permutation and rephrasing drops that do not occur, only the response tendency model we made fits the data. And we agree that why the errors grow with context is a separate causal question that needs interventions (Appendix H).
W3 (d): contribution
The consistency relations do follow prior work (Section 4.1); we use them as instruments. However, three findings are new: Unknown tendencies confound cross-model comparisons (Section 4.2), Affordance and Decision accuracy are general correlates of task success (Section 4.3), and response tendencies link context-length accuracy drops to negation inconsistency and transfer to held-out tasks (Section 5).
We thank the reviewer again and welcome further discussion.