acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing Perceptual Bottlenecks of LLM Agents in Interactive Games with Unknown Rules

Abstract

We introduce a framework for evaluating LLM agents' perception through their action-selection reports on ARC-AGI-3, a benchmark of interactive games with unknown rules, controls, and goals. It automatically labels screen changes and uses an LLM judge to compare them with agents' reports, identifying omissions, inaccurate descriptions, and unsupported claims. Across four models, errors vary by model and change type; even explicitly reported changes are sometimes misdescribed. To test whether detected errors hinder progress, we resume execution from identical states and histories, with and without a message describing a selected unreported change. The intervention improves both reporting accuracy for the selected changes and overall level attainment in Qwen3.8-Max, highlighting the importance of accurate perception for reasoning and planning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, perception bottleneck, hallucination, diagnostic evaluation, targeted intervention, games, ARC-AGI-3

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_bM14OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a framework for diagnosing how LLM agents report action-induced screen changes in ARC-AGI-3 games with unknown rules. It constructs game-specific ground-truth events from game code, environment transitions, and displayed frames, then compares these events with the free-form Reasoning Logs used for action selection. The evaluation separates unreported events, entity/content errors, and unsupported claims. Across four models and 20 games, the paper finds substantial variation in reporting accuracy across models and change types. It then resumes agents from matched states and histories, with or without a one-time textual description of a selected ground-truth change. The intervention improves target-report accuracy for Qwen3.8-Max and Opus5; it increases aggregate level attainment for Qwen3.8-Max but not for Opus5.

Critical strengths

1.The RQ1 diagnosis is carefully operationalized and substantially more informative than task-success-only evaluation. Ground-truth events are extracted deterministically from game code, state transitions, and the frames presented to the agent, independently of the agent's logs. The paper evaluates 400 runs across four models and 20 games, distinguishes omission, entity, content, and unsupported-claim errors, and reports robustness to a second LLM judge and aggregation choices. This produces useful evidence that reporting failures vary strongly by model and mechanic.

2.The RQ2 design is a meaningful attempt to test downstream relevance rather than treating every reporting error as causal. Baseline and intervention continuations resume from the same state and history and use the same 15-action budget; the selected change is supplied only once and excludes strategic labels, rules, and correct actions. The paper measures both a target-report manipulation check and later level attainment, and reports null or mixed cases rather than claiming universal improvement. This is a valuable methodological direction for diagnosing agent failures.

Critical weaknesses

1.The RQ2 intervention does not isolate the informational content of the supplied change from the attention induced by naming a particular object and change. Section 3.4 explicitly notes that the comparison estimates the combined effect of supplying information and directing attention to it. The Qwen3.8-Max attainment increase is also limited and sensitive to the unit of analysis: it rises from 6/200 to 17/200 continuations, with a run-level stratified test of p=0.011 but a game-level sign-flip sensitivity test of p=0.078; removing the most influential game yields p=0.066. Opus5 has no aggregate attainment gain. Thus, the results support that one-time targeted messages can help Qwen3.8-Max at some selected points, but do not yet establish that correcting perception itself, rather than targeted attention or prompt augmentation, is the cause.

2.The framework evaluates written Reasoning Logs rather than an agent's internal perceptual state. As the paper acknowledges, an omitted report may reflect reporting choice, state comparison, memory, or use of the information rather than a failure to perceive the screen change. In addition, semantic scoring remains partly dependent on LLM judges: the blinded human validation uses one evaluator and 200 event-side items, with exact-tuple agreement of 80.0% with Sol and 83.5% with GLM. These checks are valuable, but they do not establish that the reported metrics are a complete or model-independent measure of perception. The claims should remain scoped to action-selection reports in textual-grid ARC-AGI-3 settings.

Questions for authors

1.Can the authors add an attention- and length-matched control for RQ2, such as an equally specific but task-irrelevant visible-change message, or a mismatched change description? This would test whether the Qwen3.8-Max attainment result depends on the semantic correctness of the supplied event rather than on directing attention to a salient object.

2.Target selection is based on an error in one original RQ1 trajectory, but the error does not always recur in the resumed baseline continuations; the paper notes that this occurs at several Opus5 checkpoints. Please report the intervention effect restricted to selected points where baseline continuations reproduce the target-report error, or otherwise clarify how conclusions about correcting a diagnosed bottleneck should be interpreted when the baseline report is already correct.

3.Please clarify whether the Cochran-Mantel-Haenszel test was the pre-specified primary inference for Qwen3.8-Max attainment, and provide an inference that treats games as the primary generalization unit or a hierarchical analysis. The current run-level result and game-level sensitivity analysis lead to different significance conclusions.

4.Can the authors expand the validation of semantic scoring with multiple independent human evaluators, especially for intervention-target reports and claim-side labels? This would help quantify how much of the reported error profile depends on the chosen LLM judge and codebook scope.

4 comments
AuthorsOpenReview ↗

Response to Reviewer bM14 (1/3)

Thank you for the review. We summarize our main responses below. First, a control message that identifies the same object without describing its change did not raise Qwen3.8-Max attainment, while the change description did (Q1). Furthermore, the increase in attainment holds for both models in a hierarchical analysis with games as a random effect (Q3). We also expanded the human validation to three raters, including intervention-target reports (Q4), and report the restricted analysis you asked for (Q2). Finally, we adopted the scope you proposed (W2). Note that these answers use the revised RQ2 results, which differ from the submission; our general response explains why they were updated and what changed in the conclusions.

W1. "The RQ2 intervention does not isolate the informational content of the supplied change from the attention induced by naming a particular object and change."

Q1. "Can the authors add an attention- and length-matched control for RQ2, such as an equally specific but task-irrelevant visible-change message, or a mismatched change description?"

We ran a control with the aim you describe (Target-only; new analysis). Its message identifies the same target object by its size and position and does not describe the change. It differs from the two messages you propose: it identifies the same object but omits the change description, rather than replacing it with an irrelevant or mismatched one, and it is not length-matched to the intervention message. Within the rebuttal period, we ran this control for Qwen3.8-Max at all 20 checkpoints, from the same states and histories and with the same generation settings.

Qwen3.8-Max, 200 continuations per conditionAttainmentTarget report, primary judge (GPT-5.6 Sol)Target report, second judge (GLM-5.2)
Baseline (no message)6/20071/20061/200
Target-only (object identified, change not described)5/20093/20093/200
Intervention (object identified, change described)16/200150/200152/200

Game-stratified CMH test on attainment: baseline vs. Target-only, p = 0.743; Target-only vs. intervention, p = 0.008. Observed attainment was 5/200 with object identification alone, compared with 6/200 without a message and 16/200 with the change description. Thus, the change-describing message outperformed the object-only control. This supports a contribution of the information itself, beyond directing attention to the object. However, it does not fully separate the two, because the control is not length-matched and was run for one model.

AuthorsOpenReview ↗

Response to Reviewer bM14 (2/3)

W1. "The Qwen3.8-Max attainment increase is also limited and sensitive to the unit of analysis" [...] "Opus5 has no aggregate attainment gain."

Q3. "Please clarify whether the Cochran-Mantel-Haenszel test was the pre-specified primary inference for Qwen3.8-Max attainment, and provide an inference that treats games as the primary generalization unit or a hierarchical analysis."

Pre-specification. Yes. The game-stratified CMH test was fixed as the primary attainment analysis before the comparisons were run. The selection rule was fixed before any new run for the common-rule cohort was launched (Appendix I.1). The game-level sign-flip test is a sensitivity analysis.

Revised results. Attainment now increases in both models (our general response gives the counts and explains how they changed from the submission). For Qwen3.8-Max, the values you cite are now p = 0.019 (CMH), p = 0.094 (game-level sign-flip), and p = 0.106 (excluding the game with the largest increase) (Section 4.3).

Hierarchical and game-level analyses. Appendix I.2 adds the hierarchical analysis you ask for, and a game-level test over the two models.

AnalysisQwen3.8-MaxOpus5
Game-stratified CMH test (primary): p; common odds ratio (95% interval)0.019; 3.27 (1.17–9.17)0.031; 2.41 (1.08–5.39)
Logistic model with a random intercept for each game: odds ratio (95% interval); Wald p; likelihood-ratio p3.31 (1.18–9.31); 0.023; 0.0162.41 (1.07–5.40); 0.033; 0.029

A sign-flip test on the per-game differences in attaining-run counts, summed over the two models across the 20 games, gives p = 0.011. This combined analysis evaluates the summed two-model effect at the game level.

The increase in attainment therefore holds for both models when games are modeled as a random effect, with odds ratios close to those of the primary analysis.

W2. "The framework evaluates written Reasoning Logs rather than an agent's internal perceptual state." [...] "The claims should remain scoped to action-selection reports in textual-grid ARC-AGI-3 settings."

We agree with this scope, and we revised the paper to state it early.

  • Scope of the claims. The abstract and Introduction now state that we evaluate what agents write, not their internal states, and that screens are presented as textual integer grids. Sections 3.1 and 4.1 already state these. Section 5 states that the evidence is limited to the studied games, observation format, models, and continuation budget. We do not claim a complete or model-independent measure of perception.
  • Reliability of the evaluation. The main results are reported under two judges from different model families (Table 2). Sol was fixed as the primary judge before the human validation (Section 4.2). We also expanded the human validation to three raters. Appendix F.3 states: "Raters B and C each labeled the same 50 RQ1 items shared with Rater A and 50 RQ2 intervention-target items (24 baseline and 26 intervention)." On Rater A's full 200-item RQ1 sample, exact-tuple agreement is 81.0% for the primary judge (κ = 0.661) and 84.5% for the second judge (κ = 0.712) (Table 40).
AuthorsOpenReview ↗

Response to Reviewer bM14 (3/3)

Q4. "Can the authors expand the validation of semantic scoring with multiple independent human evaluators, especially for intervention-target reports and claim-side labels?"

We expanded the event-side validation to three human evaluators (Appendix F.3). Rater A labeled 200 RQ1 items. Raters B and C each labeled the same 50 RQ1 items shared with A and the same 50 RQ2 intervention-target items (24 baseline and 26 intervention). Each set covers 20 games. Agreement on the same items is summarized below; exact-tuple agreement requires agreement on reported, entity, and content together.

ComparisonRQ1: 50 shared itemsRQ2: 50 shared items
Human–human85.3%74.0%
Primary judge (Sol)–human90.0%72.0%
Second judge (GLM)–human88.0%62.0%
Sol–GLM92.0%82.0%

Values average the applicable pairs: humans A, B, and C in RQ1, and B and C in RQ2. In RQ2, the primary judge was close to human–human agreement in point estimate, while the second judge was lower. The Sol–human minus human–human differences were +4.7 percentage points in RQ1 (95% item-bootstrap interval [+1.3, +8.7]) and −2.0 in RQ2 [−15.0, +10.0]; the corresponding GLM differences were +2.7 [−3.3, +8.0] and −12.0 [−26.0, +2.0]. Table 42 and Figure 24 report the individual pairwise results, Cohen's κ, the reported/entity/content components, and uncertainty. Table 40 retains the separate analysis of Rater A's full 200-item sample.

These comparisons assess semantic scoring of the sampled reports. Level attainment is determined separately from environment transition records, not from either judge's semantic labels (Section 3.4).

In summary, the validation coverage is as follows:

Your pointWhat we report
Intervention-target reportsEvery first report in RQ2 is scored by both judges, and the results are given for both (Tables 2 and 54). Raters B and C each labeled the same 50 intervention-target items; their exact-tuple agreement was 74.0% (Appendix F.3).
Claim-side labelsThere are no human labels. At the observation level, the two judges assigned matching counts for each claim-side category in 82.7% of the 17,657 RQ1 observations (κ = 0.74; new analysis).
Codebook scope774 of the 12,480 change claims (6.2%) were classified as outside the codebook scope (primary judge, Tables 44 and 45). The codebooks and the judges' evaluation prompts are included in the released materials.

Q2. "Please report the intervention effect restricted to selected points where baseline continuations reproduce the target-report error, or otherwise clarify how conclusions about correcting a diagnosed bottleneck should be interpreted when the baseline report is already correct."

To examine whether the intervention remains effective at points where the baseline error recurs, we excluded points where the baseline target report was correct in at least nine of ten continuations (new analysis; primary judge; baseline → intervention):

ModelPoints keptTarget-report accuracyAttainment
Qwen3.8-Max20 of 2071/200 → 150/2006/200 → 16/200
Opus512 of 2025/120 → 80/12024/120 → 26/120

For Qwen3.8-Max, all 20 points were retained, so the results are unchanged. For Opus5, target-report accuracy improved substantially, while the increase in attainment was small. Section 3.4 defines how these two outcomes are read: "An improvement supports the diagnosis that the information deficit limited progress at that point; if the report improves but progress does not, correcting the report was not sufficient there." Thus, although its effect on attainment is limited for Opus5 in this restricted analysis, the intervention improves reporting accuracy in both models and raises attainment for Qwen3.8-Max.

Thanks for your response. I think if the results are updated, the persuasiveness of this paper would decrease significantly; therefore, I will maintain my score.

Reviewer_mcBnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates perceptual failures in LLM agents playing unfamiliar grid games from the ARC-AGI-3 demo set. Because task failure alone cannot distinguish poor planning from a failure to perceive visible screen changes, the authors introduce a two-stage evaluation framework. First, they extract ground-truth screen changes directly from game transitions and use model judges to evaluate how accurately agents' free-form reasoning logs capture each event (RQ1). Second, they resume execution from diagnosed failure checkpoints, supply a neutral description of the missed event, and measure level attainment across a 15-action budget (RQ2). Across four evaluated models, reporting accuracy varies widely by change type. Supplying the missed change roughly doubles target reporting accuracy for both Qwen3.8-Max and Claude Opus 5, but translates into higher level attainment only for Qwen. The authors conclude that repairing a local perceptual report is not always sufficient to resolve execution bottlenecks, and that task outcomes alone cannot isolate where an agent failed.

Critical strengths

The primary strength of the work is moving beyond passive error logging to counterfactual intervention. Rather than stopping at taxonomy counts from an automated judge, the authors restart runs from identical checkpoints with minimal hints to test whether an omission actually blocked downstream task progress. This offers a practical experimental template for causal error attribution in agent benchmarks. The empirical reporting is also unusually transparent. The authors pre-specified selection rules, evaluated agreement across two independent judge models and a human annotator, and openly report sensitivity checks where headline claims lose significance, such as when Qwen's attainment gain weakens under a game-level sign-flip test. They also analyze why Opus showed no aggregate gain instead of minimizing the neutral outcome.

Critical weaknesses

The baseline resampling design for RQ2 introduces an important confound. In the intervention runs, the baseline regenerates its observation log from scratch rather than resuming from the logged failure. Because checkpoints were selected from any failure across five previous runs, strong models like Opus already report the target change correctly in nearly all resampled baseline trials. In those cases, the experiment evaluates an unneeded prompt on an intact agent rather than the repair of a genuine blind spot. Conflating silent omissions with active false descriptions further obscures whether the intervention restored progress or simply reminded the agent of an obvious detail. In addition, single-event cues appear to induce tunnel vision during multi-event transitions. When an action causes multiple visual changes, describing only one event can mislead the agent. In the trace for LS20, an upward move both shifted the player block and rotated a corner display. The baseline model recognized the player move but missed the rotation. When the intervention explicitly supplied the rotation, Opus asserted that the move did nothing except rotate the display, discarding its previous correct perception of player movement. Supplying a partial cue appears to create a false pragmatic impression that no other changes occurred, which can corrupt the agent's world model and explain why attainment dropped in several games.

Questions for authors
  1. What do attainment results look like if you restrict the analysis to checkpoints where the resampled baseline actually fails to perceive the event? Why resample the log from scratch rather than resuming actions directly from the logged failure?

  2. In games like RE86 and TN36, why did providing the ground-truth change cause target report accuracy to drop compared to baseline? Did the models omit the change because it was already treated as given context?

  3. Across the intervention trials, how often did supplying a single target change degrade reporting accuracy for other, simultaneous screen events?

2 comments
AuthorsOpenReview ↗

Response to Reviewer mcBn (1/2)

Thank you for the review. We first summarize our main responses. In RQ2, both the baseline and intervention conditions resume from the same saved state and history as the logged failure and regenerate the report there, so they differ only in the intervention message (W, Q1). To further address the concern about the baseline, we also report an analysis restricted to points where the baseline error recurs (Q1). Moreover, target-report accuracy and attainment were higher under intervention for both omissions and false descriptions, while reporting accuracy for other changes that occurred at the same observation as the selected target did not decrease in aggregate (Q3). Note that these answers use the revised RQ2 results, which differ from the submission; our general response explains why they were updated and what changed in the conclusions.

Detailed responses follow.

W. "the baseline regenerates its observation log from scratch rather than resuming from the logged failure."

Q1. "Why resample the log from scratch rather than resuming actions directly from the logged failure?"

Design. We do not regenerate the observation log from scratch: the logs preceding the logged failure are kept unchanged. Both the baseline and intervention conditions resume from the same saved state and history as the logged failure, and both generate only the report for that observation again. The input to the baseline is identical to the input at the logged failure. We regenerate this report in both conditions for two reasons.

  1. Reports can vary between generations from the same state, so a report can become correct through regeneration alone. If the baseline kept the logged erroneous report while the intervention generated a new one, improvement simply due to regeneration could be mistakenly attributed to the intervention message.
  2. One of our two evaluation outcomes is whether the supplied change is correctly reflected in the report used for action selection. The message must therefore be supplied before the report is written. If the logged report were kept in both conditions and the message added afterwards, this outcome could not be measured.

Regeneration is therefore common to both conditions, and the only difference is the presence or absence of the intervention message.

Q1. "What do attainment results look like if you restrict the analysis to checkpoints where the resampled baseline actually fails to perceive the event?"

For Opus5, the baseline target report was fully correct in 101 of 200 continuations (50.5%, primary judge). At 8 of the 20 selected points (6 with the second judge), it was correct in at least nine of ten continuations (Section 4.3). Excluding those points (new analysis; primary judge; baseline → intervention):

ModelPoints keptTarget-report accuracyAttainment
Qwen3.8-Max20 of 2071/200 → 150/2006/200 → 16/200
Opus512 of 2025/120 → 80/12024/120 → 26/120

For Qwen3.8-Max, all 20 points were retained, so the results are unchanged. For Opus5, target-report accuracy improved substantially, while the increase in attainment was small. Section 3.4 defines how these two outcomes are read: "An improvement supports the diagnosis that the information deficit limited progress at that point; if the report improves but progress does not, correcting the report was not sufficient there." Thus, although its effect on attainment is limited for Opus5 in this restricted analysis, the intervention improves reporting accuracy in both models and raises attainment for Qwen3.8-Max.

W. "Conflating silent omissions with active false descriptions further obscures whether the intervention restored progress or simply reminded the agent of an obvious detail."

To address this concern, we present two complementary sets of results.

First, we divided the 40 selected points into two groups (unreported and misdescribed) based on the errors in the original RQ1 reports (new analysis; primary judge; baseline → intervention).

Error at selectionModelPointsTarget-report accuracyAttainment
UnreportedQwen3.8-Max1648/160 → 121/1603/160 → 12/160
UnreportedOpus51673/160 → 114/16052/160 → 59/160
MisdescribedQwen3.8-Max423/40 → 29/403/40 → 4/40
MisdescribedOpus5428/40 → 34/403/40 → 7/40

Target-report accuracy and attainment were higher under intervention in both groups.

Second, to examine whether the message only reminded the agent of an obvious detail, we tested a control message that identified the target object without describing its change. For Qwen3.8-Max, observed attainment was 6/200 under baseline, 5/200 with this control message, and 16/200 with the change description (new analysis; see our response to Reviewer bM14, Q1).

AuthorsOpenReview ↗

Response to Reviewer mcBn (2/2)

W. "single-event cues appear to induce tunnel vision during multi-event transitions." [...] "which can corrupt the agent's world model and explain why attainment dropped in several games."

Q3. "how often did supplying a single target change degrade reporting accuracy for other, simultaneous screen events?"

We agree that supplying a message about one change may cause the agent to overlook other changes from the same transition, as suggested by the LS20 trace you describe. To examine how often this occurs, we scored the changes other than the target in the first report after resumption (new analysis). 32 of the 40 selected points have such changes.

Qwen3.8-MaxOpus5
Selected points with other simultaneous changes14 of 2018 of 20
Other changes fully correct, primary judge (baseline → intervention)201/480 → 225/480416/610 → 418/610
Other changes fully correct, second judge188/480 → 227/478378/610 → 397/610
Points where this rate was lower / higher / equal under intervention, primary judge7 / 5 / 27 / 7 / 4
Points where this rate was lower / higher / equal under intervention, second judge5 / 5 / 46 / 10 / 2

Reporting accuracy for other changes was lower under intervention at some points and higher at others, as shown above. Regarding level attainment, Table 55 shows that it was higher under intervention in 6 of 20 games for each model, but lower in 1 game for Qwen3.8-Max and 2 games for Opus5. In each case, the number of successful continuations decreased by only one out of ten (Tables 57 and 58).

Overall, although reporting accuracy for other simultaneous changes was sometimes lower under intervention, pooled accuracy did not decrease for either model under either judge. The decreases in level attainment were also limited in both frequency and magnitude. Thus, while negative effects at individual points cannot be ruled out, the results do not show an overall decrease in reporting accuracy or level attainment under intervention.

Q2. "In games like RE86 and TN36, why did providing the ground-truth change cause target report accuracy to drop compared to baseline? Did the models omit the change because it was already treated as given context?"

We analyzed the reporting errors in RE86 and TN36 using the primary judge (GPT-5.6 Sol) and the second judge (GLM-5.2). In RE86, unreported target changes increased under the primary judge; in TN36, the change was reported, but its content was more often wrong. The table below shows the breakdown of the first reports after resumption across 10 continuations per condition (Opus5; new analysis).

Game, judgeBaseline: unreported / wrong content / fully correctIntervention: unreported / wrong content / fully correct
RE86, primary judge1 / 0 / 95 / 1 / 4
RE86, second judge7 / 0 / 36 / 0 / 4
TN36, primary judge0 / 1 / 90 / 6 / 4
TN36, second judge0 / 1 / 90 / 6 / 4

The results for each game can be summarized as follows:

  • RE86: with the primary judge, most of the decrease is in omissions: 5 of 10 intervention reports did not mention the target change (1 of 10 under baseline). With the second judge, there is no decrease (3/10 → 4/10).
  • TN36: the change was not omitted. All 10 intervention reports mention it, and both judges scored 6 of them as stating its content incorrectly.
  • Attainment did not decrease in these two games: 6/10 → 9/10 in RE86 and 2/10 → 3/10 in TN36 (Table 58).

These results show that the decrease in target-report accuracy cannot be explained solely by the omission of information already provided in the intervention message. Although the underlying causes remain unclear, level attainment did not decrease in either game.

Reviewer_gSB6OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce a perception evaluation framework for interactive environments with unknown rules. They target the questions of 1.) how well LLMs actually are able to perceive the visual changes that occur after their action, and 2.) whether or not the modality of how the LLM receives the observations from the interactive environment is a bottle neck for their performance. They claim as contributions a framework for evaluating the ability of LLMs to actually understand the observations given to them in a partially observable environment, an analysis of 5 models in how they perceive the effects of their actions, and an intervention framework for testing modality bottlenecks. All experiments are done on ARC-AGI3, and the authors report improvement in level attainment for Qwen3,8-max but not Claude Opus 5.5.

Critical strengths

I'd say top two things in the paper that I appreciated:

1.) The thoroughness of the cross validation of the judge reliability. A blind human reviewer plus two LLMs is a decent breadth of trial runs for making sure the criteria the LLM judges are given are reliable.

2.) The authors are very comprehensive with the results they report, even when the results aren't the most flattering to their claims.

Critical weaknesses

Some major weaknesses where I would say I want all of these addressed in some way before adjusting my score. I know we are asked only to provide 1 or 2, but I feel all of these would need to be addressed before raising my score into the range of acceptance:

1.) Prompts were not given up front. We cannot trust the validity of most, if not all, results without the ability to look at what the LLMs were given to see if the wording biased them one way or another. Beyond just concerns about how the 'reasoning log' was generated, this also undercuts 1.) in the strengths

2.) I would argue the target measure isn't even perception bottlenecks, and this undercuts both research question 1 and 2. Its more like 'how close is the generated summary (reasoning log) correctly describing the authors' codebook of visible changes'. (And again, this was not provided upfront). While the authors DO use softer language later on, ie section 3.1, as it stands the abstract and introduction heavily oversell the paper

3.) Perception (the LLM not understanding the symbolic render of the environment) and hallucination are not disentangled in how they could be effecting model performance.

4.) Experiments are done only in ARC-AGI-3. I could want to see these results replicated in some other symbolic environment.

5.) Zero engagement with text-based situated environments (Textworld, Alfworld, Scienceworld, etc) where the 'perception bottleneck' should more or less not exist according to the framing in the introduction.

Questions for authors

1.) Why were the prompts not provided upfront, especially given the shorter time frame for discussions compared to the initial review?

2.) Related to 3 in critical weaknesses, what proof do the authors have for failures in perception not being hallucination versus incorrect interpretations of the observations / environment state?

3.) What proof do the authors have of their framework generalizing to other environments?

3 comments
AuthorsOpenReview ↗

Response to Reviewer gSB6 (1/3)

Thank you for the review. We first summarize our main responses. We have made the prompts, codebooks, code, and logs available so that what the models and the judges were given can now be checked (W1, Q1). We have also clarified in the abstract and Introduction that we evaluate written reports. However, the paper does more than compare these reports with a reference record: it tests whether a diagnosed reporting error matters for progress by supplying that change, and attainment rose in both models (W2, W3). The evidence comes from ARC-AGI-3, where the same procedure was applied to 20 games, but replication in other symbolic environments remains future work (W4). Note that these answers use the revised RQ2 results, which differ from the submission; our general response explains why they were updated and what changed in the conclusions.

Detailed responses follow.

W1. "Prompts were not given up front. We cannot trust the validity of most, if not all, results without the ability to look at what the LLMs were given"

Q1. "Why were the prompts not provided upfront, especially given the shorter time frame for discussions compared to the initial review?"

The prompts, codebooks, code, and logs are available now, and we apologize for not providing them earlier.

In the paper. Appendix D.2 now reproduces the observation and action requests verbatim.

Released materials. The link to the anonymized repository is in a separate comment to the reviewers and area chairs, and its README describes where the materials are located. The materials cover the requests to the agent (same text for all four models), the messages actually sent in each RQ1 run, the judges' evaluation prompts and the per-game codebooks and judging rules for all 20 games, and the intervention messages for the 40 conditions.

The observation request was taken from the official ARC-AGI-3 agent implementation without modification. No system message or per-game description was provided. With the released materials, what each model and each judge was given can now be checked directly against the results.

AuthorsOpenReview ↗

Response to Reviewer gSB6 (2/3)

W2. "the target measure isn't even perception bottlenecks" [...] "the abstract and introduction heavily oversell the paper"

First, we agree on what is compared: the written report, and a record of visible changes whose scope is set by per-game codebooks. We revised the abstract and Introduction to say so. Both now state that we evaluate what agents write, not their internal states, as Section 3.1 does.

Second, to clarify why evaluating written reports is meaningful, we highlight three key points:

  1. The report is an input to action selection. It is the agent's response to the observation request, and the same text is passed to its action selection (Section 3.1). It is not a summary generated afterwards.
  2. The reference record is built without reading the agent's reports. It is extracted deterministically from environment records and the frames presented to the agent. Human input is limited to the per-game codebooks and judging rules (Section 3.2). Of the 12,480 change claims in the reports, 774 (6.2%) fell outside the codebook scope (primary judge, Tables 44 and 45). The codebooks and the judges' evaluation prompts are included in the released materials.
  3. Whether a reporting error is a bottleneck is tested, not assumed. Section 3.4 calls an unreported or misdescribed change a candidate bottleneck. RQ2 supplies that one change and follows execution. Attainment rose in both models (2.67-fold for Qwen3.8-Max, p = 0.019; 1.20-fold for Opus5, p = 0.031). For Qwen3.8-Max, observed attainment was 5/200 with a control message that identified the target object without describing its change, compared with 6/200 without a message and 16/200 with the change description (new analysis; see our response to Reviewer bM14, Q1).

Together, these properties let us identify candidate bottlenecks and test whether supplying the diagnosed changes improves progress. The higher aggregate attainment in both models suggests that information about unreported or misdescribed screen changes can be important for task progress.

W3. "Perception (the LLM not understanding the symbolic render of the environment) and hallucination are not disentangled in how they could be effecting model performance."

Q2. "what proof do the authors have for failures in perception not being hallucination versus incorrect interpretations of the observations / environment state?"

The scoring separates three kinds of error (unreported, misdescribed, and unsupported claim) as they appear in the reports. An unreported change is not a hallucination: a change occurred, and the report does not mention it. A misdescribed change is not a hallucinated change either: the change occurred, and it is reported with the wrong object or the wrong content. Hallucinated changes are counted separately, on the claim side, as unsupported claims.

Kind of errorWhat it isRQ1, primary judge, range over the four models
UnreportedA change occurred and is absent from the report19.6%–72.6% of changes
MisdescribedA change occurred and is reported with the wrong object or the wrong contentAmong reported changes, the object was correct for 95.1%–99.7% and the content for 79.4%–94.4%
Unsupported claim (includes hallucinated changes)The report claims a change with no corresponding ground-truth event within the evaluation scope7.1%–25.7% of the claims classified as linked or unsupported (second judge: 5.4%–17.5%)

These values are in Section 4.2.

Effect on performance, by kind of error (new analysis). RQ2 supplies only changes that actually occurred and were unreported or misdescribed. Split by the error in the RQ1 report used for selection (baseline → intervention, primary judge):

Error at selectionSelected pointsTarget-report accuracyAttainment
Unreported32121/320 → 235/32055/320 → 71/320
Misdescribed851/80 → 63/806/80 → 11/80

Unsupported claims. We did not intervene on unsupported claims, so their effect on attainment is not tested. In the first report after resumption, reports with an unsupported claim about the target mechanic were between 0 and 9 of 200 in every model, condition, and judge (Table 59).

AuthorsOpenReview ↗

Response to Reviewer gSB6 (3/3)

W4. "Experiments are done only in ARC-AGI-3. I could want to see these results replicated in some other symbolic environment."

Q3. "What proof do the authors have of their framework generalizing to other environments?"

We have evaluated the framework only on ARC-AGI-3, and we do not claim that the results hold in other environments. To assess the capabilities and limitations of current LLM agents, we need games that remain challenging for these models, such as those in ARC-AGI-3. Comparable environments are limited, and developing new ones requires substantial effort. However, although ARC-AGI-3 is a single benchmark, we applied the same procedure to 20 distinct games with different rules, controls, and goals. This provides evidence across diverse games within ARC-AGI-3, but does not establish generalization beyond it.

Accordingly, Section 5 states that the evidence is limited to the studied games, observation format, models, and continuation budget. Section 6 leaves other environments for future work. The revised abstract also reflects these limitations.

W5. "Zero engagement with text-based situated environments (Textworld, Alfworld, Scienceworld, etc) where the 'perception bottleneck' should more or less not exist according to the framing in the introduction."

We added TextWorld, ALFWorld, and ScienceWorld to Related Work and clarified the difference in the Introduction. These environments are relevant comparisons because they provide natural-language observations and action feedback. Our setting instead presents each screen only as a grid of integers, without a natural-language description of the change. The agent must determine what changed from the observations. This is the key distinction from text-based environments. Note that we do not assume that interpreting natural-language observations is necessarily easy.

Reviewer_zHpTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework for LLM evaluation that aims to understand the role of perception in agent behavior/performance. Specifically, using ARC-AGI-3 (interactive games with unknown rules) as a testbed, the paper asks if agents fail because of poor reasoning/planning or because they misperceive the environment and the impact of their actions. The framework employs an LLM judge to compare the ground truth (from the game rules, frames, etc.) and the agent's reasoning log. The judge flags missing, incorrect, or fabricated information in the reasoning log. Overall, the authors find significant rates of omission and misdescription across 4 models (Gemma4-31B, Qwen3.8-27B, Qwen3.8-Max, Opus 5) with unique error signatures for different models. The paper goes on to test the impact of these errors by intervening (adding a sentence that corrects the error). Intervention significantly improves the agent's report in both Qwen3.8-Max and Opus 5. In the former, it also improves the performance of the agent.

Critical strengths
  1. The work gives a well-designed experimental framework for isolating and unpacking the role of perception. In particular, the intervention and testing its effect from the saved state is a valuable way of getting at this.
  2. Following from 1, the authors show that, for Opus 5, the report accuracy increases significantly upon intervention, but there is no parallel improvement in level attainment. I think this finding is an informative one and is currently undersold. Read one way, this shows that we must exercise caution when attempting to study agents from chain-of-thought or self-reported reasoning.
Critical weaknesses
  1. The authors measure perception as what the agent chooses to write when asked to write a "few sentences". This still leaves a gap: an unreported change could be something the model perceived but did not deem relevant or something omitted for brevity.
  2. The abstract and contribution 3 seem to over-claim on the intervention findings. For example, the abstract mentions that intervention improves performance of Qwen3.8-Max but does not mention that the same is not true for Opus.
Questions for authors
  1. Have you tried giving the agent the full ground truth at every step? It seems like this sort of perception oracle would make it so that any remaining errors must be in reasoning/planning rather than in perception. This seems like a natural test if the goal is diagnosing the bottleneck.
  2. How do the rates of reporting errors change if the agent is asked to thoroughly report all observed changes rather than a few useful sentences? If they drop sharply, RQ1 mostly measures reporting policy rather than perception. It would also be very helpful if you included all prompts used in the appendix.
  3. In RQ2, could you select targets which are relevant for the game? That way, errors are actually impactful and can't be written off as the agent selectively reporting what it thinks is important.
3 comments
AuthorsOpenReview ↗

Response to Reviewer zHpT (1/3)

Thank you for the review. First, we agree that an unreported change alone does not show that the agent failed to perceive it, and the paper now states that we evaluate written reports. We therefore narrowed contribution 3 and the abstract accordingly (W2). However, omission by choice does not explain all reporting errors, as reports also deny or misdescribe changes that occurred, and errors remain even when the agent is explicitly asked to describe what changed (Q2). Moreover, the diagnosed errors matter for progress: supplying a diagnosed change improves both the report and level attainment in both models (W1, Q3). Note that these answers use the revised RQ2 results, which differ from the submission; our general response explains why they were updated and what changed in the conclusions.

Detailed responses follow.

W1. "an unreported change could be something the model perceived but did not deem relevant or something omitted for brevity."

We agree that this distinction matters for our claims, and that we score what the agent writes, not its internal state. Section 3.1 states this, and the abstract and Introduction now state it as well. Note that Section 3.1 defines perception as updating the agent's estimate of the environment state from the new observation, the preceding action, and retained information. The scored report is the agent's response to the observation request, not a summary written afterwards: at each step, this request presents the new frame before the action request, and the same text is then passed to action selection.

However, we cannot tell, for an individual unreported change, whether the agent perceived it. What we can show is that relevance or brevity alone does not explain the reporting errors. Four results support this; the first two concern incorrect statements, not omissions.

ResultValueSource
Reports that describe an object concretely and explicitly deny that it changedat least 14 scenes in 8 gamesAppendix B.3, Table 5
Selected RQ2 targets where the agent had described the change and got it wrong8 of 40; at these points, target report 51/80 → 63/80 and attainment 6/80 → 11/80 (baseline → intervention, primary judge)new analysis
Unreported rate for action-specific changes, with routine displays such as counters and bars set aside, matched on prior mention and occurrence order19.0%–73.6% across modelsAppendix H.1, Figure 35
Errors when the agent is explicitly asked to describe what changed6.8%–38.4% of changes unreported; 3.0%–11.2% reported with an entity or content error (across models)Q2 below (new analysis)

RQ2 tests the link to task progress. Section 3.4 treats an unreported or misdescribed change as a candidate bottleneck, which we test by supplying that change. Supplying one diagnosed change raised level attainment in both models (2.67-fold for Qwen3.8-Max, p = 0.019; 1.20-fold for Opus5, p = 0.031). In contrast, for Qwen3.8-Max, observed attainment was 5/200 with a control message that identified the target object without describing its change, compared with 6/200 without a message and 16/200 with the change description (p = 0.008 between the two messages; new analysis; see our response to Reviewer bM14, Q1). These results suggest that information about screen changes that is useful for task completion was not sufficiently reflected in the agent's reports or subsequent actions, as explicitly providing this information improved level attainment. Therefore, although our framework cannot directly identify failures in the agent's internal perception, it can help evaluate reporting errors related to perception that may affect task progress.

What this shows. The reports contain errors that omission by choice does not explain, and supplying a diagnosed change improved both target-report accuracy and level attainment in both models. This supports a bottleneck diagnosis (Section 3.4).

AuthorsOpenReview ↗

Response to Reviewer zHpT (2/3)

W2. "The abstract and contribution 3 seem to over-claim on the intervention findings."

We agree that the scope of these claims needed to be clearer: what is measured, and what the intervention results support. We revised both the scope of the claim and its empirical basis.

  • What was too strong. Contribution 3 claimed that the framework distinguishes perceptual deficits from downstream limitations.
  • How we limited the claim. Contribution 3 no longer makes that claim, and the revised abstract states explicitly that the framework evaluates what agents write, not their internal states.
  • What the revised results support. The intervention raises level attainment in both models (2.67-fold for Qwen3.8-Max, p = 0.019; 1.20-fold for Opus5, p = 0.031; game-stratified Cochran–Mantel–Haenszel test, Section 4.3 and Appendix I.2). This suggests that information about unreported or misdescribed screen changes can be important for task progress. The abstract and contribution 3 now state this aggregate result for both models.

Q2. "How do the rates of reporting errors change if the agent is asked to thoroughly report all observed changes rather than a few useful sentences?" [...] "It would also be very helpful if you included all prompts used in the appendix."

We ran a closely related condition (new analysis). We kept the default observation request and added one sentence to it: "Compare this frame to the previous frame and explicitly describe what changed (any movement direction, color change, or state toggle)." We ran one run per game for each of the four models (80 runs) and compared them with the paired RQ1 runs of the same models and games, aggregated in the same way as in RQ1.

ModelUnreported (%)Entity or content error (%)Fully correct (%)
Gemma4 31B67.2 → 35.16.8 → 9.325.9 → 55.6
Qwen3.8-27B53.7 → 38.411.5 → 11.234.8 → 50.3
Qwen3.8-Max48.5 → 20.810.2 → 10.141.3 → 69.2
Opus518.2 → 6.86.6 → 3.075.2 → 90.2

Paired RQ1 runs → with the added sentence; share of ground-truth changes; primary judge.

  • The aggregate unreported rate was lower with the added sentence in all four models.
  • Errors remained under the explicit request: 6.8%–38.4% of changes were still unreported, and 3.0%–11.2% were reported with an entity or content error.

What this shows, and what it does not. The lower unreported rates show that reporting depends on the observation request. The added sentence asks for both frame comparison and explicit description, so the decrease alone does not distinguish better use of the observations from a change in reporting policy. Errors remained under the explicit request, so brevity or relevance does not explain all reporting errors.

Why RQ1 uses the default request. RQ1 scores the report written under the default request because that report is the one passed to action selection. Under the default observation request, supplying one diagnosed change raised attainment (RQ2).

Prompts. Appendix D.2 now reproduces the observation and action requests verbatim. The observation request is the one in the official ARC-AGI-3 agent implementation, used without modification. The judges' evaluation prompts, the per-game codebooks, and the intervention messages are in the released materials (see the general response).

AuthorsOpenReview ↗

Response to Reviewer zHpT (3/3)

Q1. "Have you tried giving the agent the full ground truth at every step?"

No, we have not run this experiment. Providing full ground truth at every step would be a valuable extension, but it requires constructing and validating ground truth for the entire screen state, which our current dataset does not provide. Moreover, even if we provided our existing change-only ground truth at every step, we could not conclude that the remaining errors were due solely to reasoning and planning, for two reasons.

  1. Our framework focuses on changes in the environment. Our framework focuses on the perception of changes in the environment, as defined in Section 3.1. Providing full ground truth would require including unchanged parts of the screen, which would extend the evaluation beyond our framework's scope. On the other hand, providing all changes at every step would introduce multiple pieces of information simultaneously, making it difficult to isolate the effect of each diagnosed change in RQ2.
  2. A supplied change is not always reflected in the report. Under intervention, the first report after resumption was not fully correct about the selected change in 102 of the 400 continuations across the two models (25.5%, primary judge).

RQ2 instead supplies one diagnosed change, once. The difference in attainment between the two conditions can then be related to that change.

Q3. "In RQ2, could you select targets which are relevant for the game?"

Yes, we also selected targets based on their relevance to the game and reported the results as exploratory analyses in Appendices I.3–I.9. Specifically, we selected changes that could negatively affect subsequent action selection if not correctly perceived. Attainment improved with the intervention in all 15 comparisons shown in Figure 38, each with 10 continuations per condition (e.g., 3/10 → 6/10 for Qwen3.8-Max in TN36). However, five additional comparisons with fewer continuations showed no improvement (Appendix I.9). Because these targets were selected based on relevance, we report these results separately from the main analysis.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all four reviewers for their careful reviews. This comment covers what is common to our four responses: why the results were updated, the updated results, and what changed in the conclusions. Values in our responses are those of the revised paper, unless they are marked as new analyses.

1. Why we updated the results

After receiving the reviews, we re-examined the RQ2 runs and the evaluation procedure, and found two issues that affect the results.

  • Steps without a usable response (RQ2). Some RQ2 continuations contained steps where the model returned no usable response. We reran continuations so that all runs follow the protocol in the revised Appendix I.1, including its handling of such steps: ACTION5 is used when no recognized action is obtained, and the continuation proceeds through all 15 actions. The reruns changed attainment counts in both conditions.
  • Evaluation (RQ1 and RQ2). We corrected the ground truth and the evaluation prompts of some games and rescored. The RQ1 logs are those of the submission. With the corrected RQ1 scores, the same selection rule selected a different starting point in three of the 40 conditions (Tables 56–58). Both conditions were run at these starting points, with intervention messages generated from the corrected ground truth.

Both issues affect the values in both conditions, so the results below replace those of the submission. All RQ2 runs now follow one protocol (Appendix I.1), and their logs are released.

2. Updated RQ2 results

With these corrections, the intervention improves target-report accuracy and raises level attainment in both tested models.

RQ2, baseline → intervention (200 continuations per condition)SubmissionRevised paper
Level attainment, Qwen3.8-Max6/200 → 17/2006/200 → 16/200 (p = 0.019)
Level attainment, Opus566/200 → 66/20055/200 → 66/200 (p = 0.031)
Target-report accuracy, Qwen3.8-Max33.0% → 73.5%35.5% → 75.0%
Target-report accuracy, Opus548.5% → 74.0%50.5% → 74.0%

The p-values are from the game-stratified Cochran–Mantel–Haenszel test (Section 4.3, Appendix I.2). Target-report accuracy is with the primary judge. For Opus5, the per-game attainment counts changed in both conditions (Table 58). In our individual responses, we state the effect on attainment as the ratio of the attainment rate under intervention to that under baseline (Qwen3.8-Max 2.67; Opus5 1.20); the counts are in the table above.

3. What changed in the conclusions, and what did not

  • RQ1: the model ranking by fully correct rate is unchanged.
  • RQ2, Qwen3.8-Max: unchanged. The intervention raises target-report accuracy and attainment.
  • RQ2, Opus5: changed. The submission reported equal aggregate attainment under baseline and intervention. The revised results show higher attainment under intervention (p = 0.031).

4. Where the revised paper reflects the updated results

  • Abstract and contribution 3: we now report the RQ2 improvements for both models. For example, the last sentence of the abstract now reads: "The intervention improves both reporting accuracy for the selected changes and overall level attainment in Qwen3.8-Max and Opus5, indicating that inaccurate perception of screen changes, as reflected in these reports, can limit task progress."
  • Sections 4.3, 5, and 6, and the RQ2 tables: updated results.
  • Appendix I.1: handling of steps with no recognized action.
  • Appendix I.2: inference for Opus5.

Changes made in response to individual comments are described in our response to each reviewer.

5. Prompts, code, and logs

The requests to the agent, the judges' evaluation prompts, the per-game codebooks and judging rules, the intervention messages, the code, and the execution logs, including the runs of the two additional conditions reported in our responses, are available in an anonymized repository. Its link is in a separate comment to the reviewers and area chairs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.