- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a framework for diagnosing how LLM agents report action-induced screen changes in ARC-AGI-3 games with unknown rules. It constructs game-specific ground-truth events from game code, environment transitions, and displayed frames, then compares these events with the free-form Reasoning Logs used for action selection. The evaluation separates unreported events, entity/content errors, and unsupported claims. Across four models and 20 games, the paper finds substantial variation in reporting accuracy across models and change types. It then resumes agents from matched states and histories, with or without a one-time textual description of a selected ground-truth change. The intervention improves target-report accuracy for Qwen3.8-Max and Opus5; it increases aggregate level attainment for Qwen3.8-Max but not for Opus5.
Critical strengths
1.The RQ1 diagnosis is carefully operationalized and substantially more informative than task-success-only evaluation. Ground-truth events are extracted deterministically from game code, state transitions, and the frames presented to the agent, independently of the agent's logs. The paper evaluates 400 runs across four models and 20 games, distinguishes omission, entity, content, and unsupported-claim errors, and reports robustness to a second LLM judge and aggregation choices. This produces useful evidence that reporting failures vary strongly by model and mechanic.
2.The RQ2 design is a meaningful attempt to test downstream relevance rather than treating every reporting error as causal. Baseline and intervention continuations resume from the same state and history and use the same 15-action budget; the selected change is supplied only once and excludes strategic labels, rules, and correct actions. The paper measures both a target-report manipulation check and later level attainment, and reports null or mixed cases rather than claiming universal improvement. This is a valuable methodological direction for diagnosing agent failures.
Critical weaknesses
1.The RQ2 intervention does not isolate the informational content of the supplied change from the attention induced by naming a particular object and change. Section 3.4 explicitly notes that the comparison estimates the combined effect of supplying information and directing attention to it. The Qwen3.8-Max attainment increase is also limited and sensitive to the unit of analysis: it rises from 6/200 to 17/200 continuations, with a run-level stratified test of p=0.011 but a game-level sign-flip sensitivity test of p=0.078; removing the most influential game yields p=0.066. Opus5 has no aggregate attainment gain. Thus, the results support that one-time targeted messages can help Qwen3.8-Max at some selected points, but do not yet establish that correcting perception itself, rather than targeted attention or prompt augmentation, is the cause.
2.The framework evaluates written Reasoning Logs rather than an agent's internal perceptual state. As the paper acknowledges, an omitted report may reflect reporting choice, state comparison, memory, or use of the information rather than a failure to perceive the screen change. In addition, semantic scoring remains partly dependent on LLM judges: the blinded human validation uses one evaluator and 200 event-side items, with exact-tuple agreement of 80.0% with Sol and 83.5% with GLM. These checks are valuable, but they do not establish that the reported metrics are a complete or model-independent measure of perception. The claims should remain scoped to action-selection reports in textual-grid ARC-AGI-3 settings.
Questions for authors
1.Can the authors add an attention- and length-matched control for RQ2, such as an equally specific but task-irrelevant visible-change message, or a mismatched change description? This would test whether the Qwen3.8-Max attainment result depends on the semantic correctness of the supplied event rather than on directing attention to a salient object.
2.Target selection is based on an error in one original RQ1 trajectory, but the error does not always recur in the resumed baseline continuations; the paper notes that this occurs at several Opus5 checkpoints. Please report the intervention effect restricted to selected points where baseline continuations reproduce the target-report error, or otherwise clarify how conclusions about correcting a diagnosed bottleneck should be interpreted when the baseline report is already correct.
3.Please clarify whether the Cochran-Mantel-Haenszel test was the pre-specified primary inference for Qwen3.8-Max attainment, and provide an inference that treats games as the primary generalization unit or a hierarchical analysis. The current run-level result and game-level sensitivity analysis lead to different significance conclusions.
4.Can the authors expand the validation of semantic scoring with multiple independent human evaluators, especially for intervention-target reports and claim-side labels? This would help quantify how much of the reported error profile depends on the chosen LLM judge and codebook scope.
4 comments
Response to Reviewer bM14 (2/3)
W1. "The Qwen3.8-Max attainment increase is also limited and sensitive to the unit of analysis" [...] "Opus5 has no aggregate attainment gain."
Q3. "Please clarify whether the Cochran-Mantel-Haenszel test was the pre-specified primary inference for Qwen3.8-Max attainment, and provide an inference that treats games as the primary generalization unit or a hierarchical analysis."
Pre-specification. Yes. The game-stratified CMH test was fixed as the primary attainment analysis before the comparisons were run. The selection rule was fixed before any new run for the common-rule cohort was launched (Appendix I.1). The game-level sign-flip test is a sensitivity analysis.
Revised results. Attainment now increases in both models (our general response gives the counts and explains how they changed from the submission). For Qwen3.8-Max, the values you cite are now p = 0.019 (CMH), p = 0.094 (game-level sign-flip), and p = 0.106 (excluding the game with the largest increase) (Section 4.3).
Hierarchical and game-level analyses. Appendix I.2 adds the hierarchical analysis you ask for, and a game-level test over the two models.
| Analysis | Qwen3.8-Max | Opus5 |
|---|---|---|
| Game-stratified CMH test (primary): p; common odds ratio (95% interval) | 0.019; 3.27 (1.17–9.17) | 0.031; 2.41 (1.08–5.39) |
| Logistic model with a random intercept for each game: odds ratio (95% interval); Wald p; likelihood-ratio p | 3.31 (1.18–9.31); 0.023; 0.016 | 2.41 (1.07–5.40); 0.033; 0.029 |
A sign-flip test on the per-game differences in attaining-run counts, summed over the two models across the 20 games, gives p = 0.011. This combined analysis evaluates the summed two-model effect at the game level.
The increase in attainment therefore holds for both models when games are modeled as a random effect, with odds ratios close to those of the primary analysis.
W2. "The framework evaluates written Reasoning Logs rather than an agent's internal perceptual state." [...] "The claims should remain scoped to action-selection reports in textual-grid ARC-AGI-3 settings."
We agree with this scope, and we revised the paper to state it early.
- Scope of the claims. The abstract and Introduction now state that we evaluate what agents write, not their internal states, and that screens are presented as textual integer grids. Sections 3.1 and 4.1 already state these. Section 5 states that the evidence is limited to the studied games, observation format, models, and continuation budget. We do not claim a complete or model-independent measure of perception.
- Reliability of the evaluation. The main results are reported under two judges from different model families (Table 2). Sol was fixed as the primary judge before the human validation (Section 4.2). We also expanded the human validation to three raters. Appendix F.3 states: "Raters B and C each labeled the same 50 RQ1 items shared with Rater A and 50 RQ2 intervention-target items (24 baseline and 26 intervention)." On Rater A's full 200-item RQ1 sample, exact-tuple agreement is 81.0% for the primary judge (κ = 0.661) and 84.5% for the second judge (κ = 0.712) (Table 40).
Response to Reviewer bM14 (3/3)
Q4. "Can the authors expand the validation of semantic scoring with multiple independent human evaluators, especially for intervention-target reports and claim-side labels?"
We expanded the event-side validation to three human evaluators (Appendix F.3). Rater A labeled 200 RQ1 items. Raters B and C each labeled the same 50 RQ1 items shared with A and the same 50 RQ2 intervention-target items (24 baseline and 26 intervention). Each set covers 20 games. Agreement on the same items is summarized below; exact-tuple agreement requires agreement on reported, entity, and content together.
| Comparison | RQ1: 50 shared items | RQ2: 50 shared items |
|---|---|---|
| Human–human | 85.3% | 74.0% |
| Primary judge (Sol)–human | 90.0% | 72.0% |
| Second judge (GLM)–human | 88.0% | 62.0% |
| Sol–GLM | 92.0% | 82.0% |
Values average the applicable pairs: humans A, B, and C in RQ1, and B and C in RQ2. In RQ2, the primary judge was close to human–human agreement in point estimate, while the second judge was lower. The Sol–human minus human–human differences were +4.7 percentage points in RQ1 (95% item-bootstrap interval [+1.3, +8.7]) and −2.0 in RQ2 [−15.0, +10.0]; the corresponding GLM differences were +2.7 [−3.3, +8.0] and −12.0 [−26.0, +2.0]. Table 42 and Figure 24 report the individual pairwise results, Cohen's κ, the reported/entity/content components, and uncertainty. Table 40 retains the separate analysis of Rater A's full 200-item sample.
These comparisons assess semantic scoring of the sampled reports. Level attainment is determined separately from environment transition records, not from either judge's semantic labels (Section 3.4).
In summary, the validation coverage is as follows:
| Your point | What we report |
|---|---|
| Intervention-target reports | Every first report in RQ2 is scored by both judges, and the results are given for both (Tables 2 and 54). Raters B and C each labeled the same 50 intervention-target items; their exact-tuple agreement was 74.0% (Appendix F.3). |
| Claim-side labels | There are no human labels. At the observation level, the two judges assigned matching counts for each claim-side category in 82.7% of the 17,657 RQ1 observations (κ = 0.74; new analysis). |
| Codebook scope | 774 of the 12,480 change claims (6.2%) were classified as outside the codebook scope (primary judge, Tables 44 and 45). The codebooks and the judges' evaluation prompts are included in the released materials. |
Q2. "Please report the intervention effect restricted to selected points where baseline continuations reproduce the target-report error, or otherwise clarify how conclusions about correcting a diagnosed bottleneck should be interpreted when the baseline report is already correct."
To examine whether the intervention remains effective at points where the baseline error recurs, we excluded points where the baseline target report was correct in at least nine of ten continuations (new analysis; primary judge; baseline → intervention):
| Model | Points kept | Target-report accuracy | Attainment |
|---|---|---|---|
| Qwen3.8-Max | 20 of 20 | 71/200 → 150/200 | 6/200 → 16/200 |
| Opus5 | 12 of 20 | 25/120 → 80/120 | 24/120 → 26/120 |
For Qwen3.8-Max, all 20 points were retained, so the results are unchanged. For Opus5, target-report accuracy improved substantially, while the increase in attainment was small. Section 3.4 defines how these two outcomes are read: "An improvement supports the diagnosis that the information deficit limited progress at that point; if the report improves but progress does not, correcting the report was not sufficient there." Thus, although its effect on attainment is limited for Opus5 in this restricted analysis, the intervention improves reporting accuracy in both models and raises attainment for Qwen3.8-Max.
Thanks for your response. I think if the results are updated, the persuasiveness of this paper would decrease significantly; therefore, I will maintain my score.
Response to Reviewer bM14 (1/3)
Thank you for the review. We summarize our main responses below. First, a control message that identifies the same object without describing its change did not raise Qwen3.8-Max attainment, while the change description did (Q1). Furthermore, the increase in attainment holds for both models in a hierarchical analysis with games as a random effect (Q3). We also expanded the human validation to three raters, including intervention-target reports (Q4), and report the restricted analysis you asked for (Q2). Finally, we adopted the scope you proposed (W2). Note that these answers use the revised RQ2 results, which differ from the submission; our general response explains why they were updated and what changed in the conclusions.
W1. "The RQ2 intervention does not isolate the informational content of the supplied change from the attention induced by naming a particular object and change."
Q1. "Can the authors add an attention- and length-matched control for RQ2, such as an equally specific but task-irrelevant visible-change message, or a mismatched change description?"
We ran a control with the aim you describe (Target-only; new analysis). Its message identifies the same target object by its size and position and does not describe the change. It differs from the two messages you propose: it identifies the same object but omits the change description, rather than replacing it with an irrelevant or mismatched one, and it is not length-matched to the intervention message. Within the rebuttal period, we ran this control for Qwen3.8-Max at all 20 checkpoints, from the same states and histories and with the same generation settings.
Game-stratified CMH test on attainment: baseline vs. Target-only, p = 0.743; Target-only vs. intervention, p = 0.008. Observed attainment was 5/200 with object identification alone, compared with 6/200 without a message and 16/200 with the change description. Thus, the change-describing message outperformed the object-only control. This supports a contribution of the information itself, beyond directing attention to the object. However, it does not fully separate the two, because the control is not length-matched and was run for one model.