- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ARCUS-X, a benchmark for diagnosing how reliably LLMs execute long sequences of explicitly specified state transitions. Models are given a procedurally generated toroidal-grid walk with labeled movement rules and must output every intermediate coordinate; four perturbation tiers relabel, invert, or rotate the rules. The benchmark reports trajectory-level metrics, response validity, generation overhead, and a deterministic failure taxonomy, and uses Adaptive Fracture Search (AFS) to sample horizons near where accuracy falls below 50%. Across 13 models, the paper reports three findings. First, these measurements diverge. Second, models differ in failure signatures and perturbation profiles. Third, in a matched-horizon analysis, accuracy often declines toward the end of a trajectory. The authors present these results as descriptive summaries of adaptively sampled observations rather than controlled comparisons.
Critical strengths
1. A transparent and reproducible evaluation setup.
The benchmark provides exact ground truth for each transition, supports deterministic task regeneration from fixed seeds, and preserves raw model outputs. It also distinguishes invalid responses from incorrect trajectories rather than simply treating both as zero scores. These choices make the benchmark easier to reproduce and audit, while allowing the potential issues discussed below to be investigated and corrected using the existing outputs.
2. The matched-horizon analysis of where errors occur within a trajectory is informative.
Table 6 compares all 13 models at the same horizons and splits each trajectory into early, middle, and late thirds. At z = 30, models with similar overall accuracy behave very differently:
- Kimi K2.5 stays nearly flat (69.5% → 66.8% → 65.9%);
- GLM-5 declines sharply (92.4% → 61.6% → 35.5%);
- Nemotron Super 3 falls from 76.1% to 3.8%.
A single trajectory-level accuracy hides these strongly position-dependent failures. To me, this is the clearest evidence for the paper's central claim. It also shows that matched-horizon comparisons are feasible within this framework.
Critical weaknesses
1. The main diagnostic results are not yet reliable enough to interpret.
Taxonomy vs. Exact Match. None is described as "a fully correct trajectory" (Appendix A.1.2). Yet after adjusting for validity, its share greatly exceeds the Exact Match rate for 8 of the 13 models. Mistral Large is about 44% vs. 1.1%, and Kimi K2.5 about 50% vs. 39.1%. For the other five models the two agree almost exactly (e.g., GPT-5-mini 42.4% vs. 42.5%), so the gap is not a difference in denominators.
The largest gaps occur in models with low horizon compliance. This suggests that trajectories with correct coordinates but the wrong length are labeled None, rather than Horizon failure as A.1.2 specifies. If so, Horizon would likely become Mistral Large's dominant failure category. That would contradict the abstract's statement that "transition-rule signatures are largest for Mistral Large 2512 (23.6%)."
Unmatched horizons. AFS samples different mixtures of horizons for each model and tier, as the paper acknowledges. Tier differences such as Kimi's Tier 1 vs. Tier 0 (75.8% vs. 63.0%) may therefore reflect which horizons were sampled rather than the perturbations themselves.
Since the failure signatures and tier profiles are the paper's main diagnostic outputs and are highlighted in the abstract, it is currently unclear how much of the paper's diagnosis would survive a corrected taxonomy and matched horizons.
2. The scientific contribution is not clearly established, partly because the claims are so heavily qualified.
Positioning. The paper does not cite two closely related lines of work.
- Sinha et al. (ICLR 2026, The Illusion of Diminishing Returns) already isolate execution by giving the model the plan and knowledge explicitly. They find that per-step accuracy degrades with task length, and they use a binary search over task length similar to AFS.
- Wu et al. (NAACL 2024, Reasoning or Reciting?) test a spatial task with swapped and rotated coordinate conventions, closely analogous to Tiers 2 and 3. They find that swapping north and south hurts least. ARCUS-X's pooled results show the same pattern: Tier 2 stays within about 6 points of Tier 0 for every model (Table 12).
The concurrent Complexity Ceiling Benchmark (ICML 2026 workshop), which also studies grid-based state tracking under depth scaling, is related as well, though I do not count it against this submission. The paper does not state what new finding ARCUS-X establishes beyond this work.
Caveats. I appreciate the authors' efforts to avoid overclaiming, but the repeated caveats leave the central claim little more specific than "sequential reliability is multi-faceted." The results are described as descriptive rather than causal, not a model ranking, not an intrinsic capacity, and not a general measure of long-horizon ability. Yet the abstract emphasizes cross-model signatures and tier differences that the main text then disclaims. For a benchmark paper, the contribution should lie in what the measurements reveal, not just the infrastructure. A few concrete, testable findings supported by matched-horizon evidence would help establish this contribution.
These concerns appear addressable. The taxonomy can be recomputed from stored outputs, and Table 6 demonstrates that matched-horizon comparisons are feasible. Until then, the central diagnostic claims remain insufficiently supported for acceptance.
Questions for authors
-
Taxonomy and Exact Match (reanalysis; W1). Could you explain why the share of probes labeled
Noneexceeds the Exact Match rate for several models, given thatNoneis described as a fully correct trajectory? For each model, it would be helpful to see a cross-tabulation of four quantities: the taxonomy label, Exact Match, whether the trajectory length matches the requested horizon, and validity. If correcting the classifier makesNoneagree with Exact Match and the main failure signatures remain largely unchanged, that would address much of my concern. If the signatures do change, I would appreciate updated versions of Tables 5 and 15 and of the abstract. -
Matched-horizon comparisons (additional analysis; W1). Could the perturbation-tier comparisons (Table 3) and the failure-taxonomy partition (Table 5) be reported at shared horizons, for example at z = 20 and 30 as in Table 6, or on a fixed horizon grid? Tier differences that persist at matched horizons would substantially strengthen the perturbation-tier contribution.
-
Treatment of invalid outputs (clarification; W1). Please state, for each table and analysis, whether invalid responses are excluded, counted as zero, or handled separately. For example, MAI-Thinking-1 has a fracture depth of 1 in all four tiers (Table 9), despite 98–100% accuracy on valid trajectories at z = 20 and 30 (Table 6). Does the post-hoc fracture analysis use unconditional step accuracy, while AFS uses accuracy conditional on validity?
-
Fallback assignments in the taxonomy (clarification; W1). Appendix A.1.2 states that State Tracking is also the fallback label for valid trajectories with an unmapped early divergence. What share of the State Tracking labels (e.g., 93.8% for Qwen 3 Coder and 84.3% for GLM-5) comes from this fallback rule?
-
New findings relative to prior work (clarification; W2). Which specific finding does ARCUS-X establish that would not be expected from Sinha et al. or Wu et al.? A short statement of the main new findings, each tied to a matched-horizon result, would make the contribution much clearer.
-
Secondary components (clarification; minor). Could you clarify the role of the gravity parameter g, which Section 3.4 says does not affect the prompt or tier? If it is unused, removing it might simplify the protocol. Similarly, could you clarify the intended role of CRI, given that the primary claims do not rely on it? If it mainly serves as a secondary summary, moving it to the appendix might help keep the focus on the decomposed measurements.
We thank the reviewer for their detailed and constructive assessment, particularly for identifying the taxonomy-exact match inconsistency. We agree that these concerns certainly warranted substantive attention.
For brevity, we summarize the major revisions:
1. Corrected the taxonomy-exact match discrepancy and updated the abstract, tables, and related analyses ( Appendix A.1)
2. Added taxonomy, Exact Match, length, and validity reconciliation and matched-horizon analyses with uncertainty estimates ( Appendix A.2)
3. Clarified invalid-response handlign and corrected the interpretation of MAI-Thinking-1's fracture results ( Appendix A.4)
4. Removed State Tracking fallback and added taxonomy validation and precedence sensitivity checks (Appendices A.1 and A.9)
5. Expanded Related Work section, clarified the contribution relative to Sinha et al. and Wu et al., and explained the secondary roles of gravity (inert) and CRI (secondary, first glance metric)
We address the concerns in full below:
1.Taxonomy and Exact Match
We agree that the discrepancy was a genuine inconsistency. The generator retains z+1 coordinates for z actions, while models must output exactly z coordinates, including the initial state. The taxonomy incorrectly compared outputs against the full reference rather than the requested z-coordinate trajectory.
We corrected this mismatch. Across 39,728 valid non-calibration observations, all 3,123 None labels now correspond exactly to requested-output Exact Matches, and vice versa. Remaining percentage differences arise from denominators: Exact Match includes all completed probes, while taxonomy proportions include only valid probes.
We recomputed the partitions, updated the abstract and tables, and added reconciliation checks. Some failure signatures changed substantially, notably Horizon Collapse, and unsupported conclusions were revised. The corrected results reinforce the importance of separately measuring trajectory correctness, horizon compliance, and validity.
2. Matched-horizon comparisons
We agree that pooled AFS results cannot establish controlled tier differences because horizon distributions may vary.
We added exact horizon comparisons at z=20 and 30, with coverage reporting, master-seed-block bootstrap intervals, and additional horizon-neighborhood sensitivity analyses ( see appendix A.2).All 52 model-tier combinations have observations at both horizons. At z=20, cohort pooled Step accuracy is 69.8% for Tier 0 versus 59.2% for Tier 3. At z=30, the corresponding values are 65.3% and 56.8%, respectively. We explicitly distinguish horizon matching from task pairing and qualify the pooled tier and taxonomy comparisons accordingly.
3. Invalid outputs and MAI-Thinking 1
We clarified the validity policy across metrics and analyses. Invalid responses are accounted for separately and excluded from valid-conditional trajectory accuracy and taxonomy proportions; Exact match uses all completed probes. Live AFS excludes operationally invalid responses before estimating batch accuracy. An all-invalid batch supplies insufficient evidence, not zero accuracy. Post-hoc fracture analysis likewise excludes invalid-only horizons and evaluates the sampled valid-response curve. For MAI-Thinking-1, the earlier fracture-depth interpretation was simply misleading. Appendix A.4 now documents invalid only observations at short horizons and high valid-conditional accuracy at z=20 and 30. On the corrected valid-only curves, Tiers 0, 2, and 3 have no observed fracture; Tier 1 has a right-censored terminal candidate at z=150, not a confirmed fracture.
4. State Tracking fallback
We removed the automatic State Tracking fallback. Across 39,728 valid observations, 13,633 now receive State tracking under explicit classification conditions, while four remain Unknown/Unmapped. Synthetic error-injection and precedence-sensitivity checks are reported in Appendices A.1 and A.9
5. Relative to prior work
We expanded section 2 to discuss Sinha et al., Wu et al., and related trajectory evaluations. ARCUS-X contributes an auditable joint decomposition of sequential execution into independently computable trajectory correctness, temporal continuity, exact agreement, horizon compliance, validity, and generation overhead under controlled rule perturbations. This is evidently shown by models with similar overall Step Accuracy, which can exhibit markedly different within-trajectory degradation and length compliance. The matched horizon temporal-position analysis demonstrates distinct early-to-late accuracy profiles, while decomposed measurements reveal differences hidden by aggregate correctness.
6. Gravity and CRI
The legacy gravity parameter g is inert and retained only for historical configuration reproducibility. CRI is a secondary researcher-weighted summary for first glance
We appreciate the reviewer's specific, in-depth, and actionable criticisms!