acceptodds
Under review as a conference paper at ICLR 2027

ARCUS-X: Adaptive Rule Compliance Under Sequential eXecution for Long-Horizon Language Model Evaluation

Abstract

Long-horizon language-model evaluations often reduce performance to final-answer accuracy, obscuring failures that accumulate across dependent sequential transitions. We introduce ARCUS-X, an auditable benchmark and evaluation protocol for decomposing deterministic sequential execution into trajectory correctness, continuity, rule binding, horizon control, response validity, and excess generation. ARCUS-X uses procedurally generated toroidal-grid tasks with exact per-transition ground truth, controlled lexical and transition perturbations, raw-stream preservation, and deterministic post-run diagnostics. Across 13 models, ARCUS-X exposes substantial separation between aggregate trajectory quality and observed failure signatures. In the adaptively sampled runs, GPT-5-mini attains 84.8% step accuracy, 42.5% exact match, and 84.5% continuity, but 25.9% generation efficiency and 11.2% invalid responses. The valid-error partitions differ sharply: state-tracking signatures account for 93.8% of Qwen 3 Coder 480B A35B probes and 84.3% of GLM-5 probes, transition-rule signatures are largest for Mistral Large 2512 (23.6%), and semantic-interpretation signatures are largest for Kimi K2.5 (23.2%). Sampled fracture behavior also varies by perturbation tier. Because AFS allocates horizons from observed performance, pooled aggregates describe the acquired support rather than a common uniform horizon distribution. ARCUS-X therefore provides a reproducible diagnostic record of explicit state-transition execution rather than a universal model ranking or measure of general long-horizon agency.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, LLM Evaluation, Benchmarking, Long-Horizon Reasoning, Sequential Execution, Model Robustness, AI Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9bBkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ARCUS-X, a benchmark for diagnosing how reliably LLMs execute long sequences of explicitly specified state transitions. Models are given a procedurally generated toroidal-grid walk with labeled movement rules and must output every intermediate coordinate; four perturbation tiers relabel, invert, or rotate the rules. The benchmark reports trajectory-level metrics, response validity, generation overhead, and a deterministic failure taxonomy, and uses Adaptive Fracture Search (AFS) to sample horizons near where accuracy falls below 50%. Across 13 models, the paper reports three findings. First, these measurements diverge. Second, models differ in failure signatures and perturbation profiles. Third, in a matched-horizon analysis, accuracy often declines toward the end of a trajectory. The authors present these results as descriptive summaries of adaptively sampled observations rather than controlled comparisons.

Critical strengths

1. A transparent and reproducible evaluation setup.

The benchmark provides exact ground truth for each transition, supports deterministic task regeneration from fixed seeds, and preserves raw model outputs. It also distinguishes invalid responses from incorrect trajectories rather than simply treating both as zero scores. These choices make the benchmark easier to reproduce and audit, while allowing the potential issues discussed below to be investigated and corrected using the existing outputs.

2. The matched-horizon analysis of where errors occur within a trajectory is informative.

Table 6 compares all 13 models at the same horizons and splits each trajectory into early, middle, and late thirds. At z = 30, models with similar overall accuracy behave very differently:

  • Kimi K2.5 stays nearly flat (69.5% → 66.8% → 65.9%);
  • GLM-5 declines sharply (92.4% → 61.6% → 35.5%);
  • Nemotron Super 3 falls from 76.1% to 3.8%.

A single trajectory-level accuracy hides these strongly position-dependent failures. To me, this is the clearest evidence for the paper's central claim. It also shows that matched-horizon comparisons are feasible within this framework.

Critical weaknesses

1. The main diagnostic results are not yet reliable enough to interpret.

Taxonomy vs. Exact Match. None is described as "a fully correct trajectory" (Appendix A.1.2). Yet after adjusting for validity, its share greatly exceeds the Exact Match rate for 8 of the 13 models. Mistral Large is about 44% vs. 1.1%, and Kimi K2.5 about 50% vs. 39.1%. For the other five models the two agree almost exactly (e.g., GPT-5-mini 42.4% vs. 42.5%), so the gap is not a difference in denominators.

The largest gaps occur in models with low horizon compliance. This suggests that trajectories with correct coordinates but the wrong length are labeled None, rather than Horizon failure as A.1.2 specifies. If so, Horizon would likely become Mistral Large's dominant failure category. That would contradict the abstract's statement that "transition-rule signatures are largest for Mistral Large 2512 (23.6%)."

Unmatched horizons. AFS samples different mixtures of horizons for each model and tier, as the paper acknowledges. Tier differences such as Kimi's Tier 1 vs. Tier 0 (75.8% vs. 63.0%) may therefore reflect which horizons were sampled rather than the perturbations themselves.

Since the failure signatures and tier profiles are the paper's main diagnostic outputs and are highlighted in the abstract, it is currently unclear how much of the paper's diagnosis would survive a corrected taxonomy and matched horizons.

2. The scientific contribution is not clearly established, partly because the claims are so heavily qualified.

Positioning. The paper does not cite two closely related lines of work.

  • Sinha et al. (ICLR 2026, The Illusion of Diminishing Returns) already isolate execution by giving the model the plan and knowledge explicitly. They find that per-step accuracy degrades with task length, and they use a binary search over task length similar to AFS.
  • Wu et al. (NAACL 2024, Reasoning or Reciting?) test a spatial task with swapped and rotated coordinate conventions, closely analogous to Tiers 2 and 3. They find that swapping north and south hurts least. ARCUS-X's pooled results show the same pattern: Tier 2 stays within about 6 points of Tier 0 for every model (Table 12).

The concurrent Complexity Ceiling Benchmark (ICML 2026 workshop), which also studies grid-based state tracking under depth scaling, is related as well, though I do not count it against this submission. The paper does not state what new finding ARCUS-X establishes beyond this work.

Caveats. I appreciate the authors' efforts to avoid overclaiming, but the repeated caveats leave the central claim little more specific than "sequential reliability is multi-faceted." The results are described as descriptive rather than causal, not a model ranking, not an intrinsic capacity, and not a general measure of long-horizon ability. Yet the abstract emphasizes cross-model signatures and tier differences that the main text then disclaims. For a benchmark paper, the contribution should lie in what the measurements reveal, not just the infrastructure. A few concrete, testable findings supported by matched-horizon evidence would help establish this contribution.

These concerns appear addressable. The taxonomy can be recomputed from stored outputs, and Table 6 demonstrates that matched-horizon comparisons are feasible. Until then, the central diagnostic claims remain insufficiently supported for acceptance.

Questions for authors
  1. Taxonomy and Exact Match (reanalysis; W1). Could you explain why the share of probes labeled None exceeds the Exact Match rate for several models, given that None is described as a fully correct trajectory? For each model, it would be helpful to see a cross-tabulation of four quantities: the taxonomy label, Exact Match, whether the trajectory length matches the requested horizon, and validity. If correcting the classifier makes None agree with Exact Match and the main failure signatures remain largely unchanged, that would address much of my concern. If the signatures do change, I would appreciate updated versions of Tables 5 and 15 and of the abstract.

  2. Matched-horizon comparisons (additional analysis; W1). Could the perturbation-tier comparisons (Table 3) and the failure-taxonomy partition (Table 5) be reported at shared horizons, for example at z = 20 and 30 as in Table 6, or on a fixed horizon grid? Tier differences that persist at matched horizons would substantially strengthen the perturbation-tier contribution.

  3. Treatment of invalid outputs (clarification; W1). Please state, for each table and analysis, whether invalid responses are excluded, counted as zero, or handled separately. For example, MAI-Thinking-1 has a fracture depth of 1 in all four tiers (Table 9), despite 98–100% accuracy on valid trajectories at z = 20 and 30 (Table 6). Does the post-hoc fracture analysis use unconditional step accuracy, while AFS uses accuracy conditional on validity?

  4. Fallback assignments in the taxonomy (clarification; W1). Appendix A.1.2 states that State Tracking is also the fallback label for valid trajectories with an unmapped early divergence. What share of the State Tracking labels (e.g., 93.8% for Qwen 3 Coder and 84.3% for GLM-5) comes from this fallback rule?

  5. New findings relative to prior work (clarification; W2). Which specific finding does ARCUS-X establish that would not be expected from Sinha et al. or Wu et al.? A short statement of the main new findings, each tied to a matched-horizon result, would make the contribution much clearer.

  6. Secondary components (clarification; minor). Could you clarify the role of the gravity parameter g, which Section 3.4 says does not affect the prompt or tier? If it is unused, removing it might simplify the protocol. Similarly, could you clarify the intended role of CRI, given that the primary claims do not rely on it? If it mainly serves as a secondary summary, moving it to the appendix might help keep the focus on the decomposed measurements.

1 comment

We thank the reviewer for their detailed and constructive assessment, particularly for identifying the taxonomy-exact match inconsistency. We agree that these concerns certainly warranted substantive attention.

For brevity, we summarize the major revisions:

1. Corrected the taxonomy-exact match discrepancy and updated the abstract, tables, and related analyses ( Appendix A.1)

2. Added taxonomy, Exact Match, length, and validity reconciliation and matched-horizon analyses with uncertainty estimates ( Appendix A.2)

3. Clarified invalid-response handlign and corrected the interpretation of MAI-Thinking-1's fracture results ( Appendix A.4)

4. Removed State Tracking fallback and added taxonomy validation and precedence sensitivity checks (Appendices A.1 and A.9)

5. Expanded Related Work section, clarified the contribution relative to Sinha et al. and Wu et al., and explained the secondary roles of gravity (inert) and CRI (secondary, first glance metric)

We address the concerns in full below:

1.Taxonomy and Exact Match

We agree that the discrepancy was a genuine inconsistency. The generator retains z+1 coordinates for z actions, while models must output exactly z coordinates, including the initial state. The taxonomy incorrectly compared outputs against the full reference rather than the requested z-coordinate trajectory.

We corrected this mismatch. Across 39,728 valid non-calibration observations, all 3,123 None labels now correspond exactly to requested-output Exact Matches, and vice versa. Remaining percentage differences arise from denominators: Exact Match includes all completed probes, while taxonomy proportions include only valid probes.

We recomputed the partitions, updated the abstract and tables, and added reconciliation checks. Some failure signatures changed substantially, notably Horizon Collapse, and unsupported conclusions were revised. The corrected results reinforce the importance of separately measuring trajectory correctness, horizon compliance, and validity.

2. Matched-horizon comparisons

We agree that pooled AFS results cannot establish controlled tier differences because horizon distributions may vary.

We added exact horizon comparisons at z=20 and 30, with coverage reporting, master-seed-block bootstrap intervals, and additional horizon-neighborhood sensitivity analyses ( see appendix A.2).All 52 model-tier combinations have observations at both horizons. At z=20, cohort pooled Step accuracy is 69.8% for Tier 0 versus 59.2% for Tier 3. At z=30, the corresponding values are 65.3% and 56.8%, respectively. We explicitly distinguish horizon matching from task pairing and qualify the pooled tier and taxonomy comparisons accordingly.

3. Invalid outputs and MAI-Thinking 1

We clarified the validity policy across metrics and analyses. Invalid responses are accounted for separately and excluded from valid-conditional trajectory accuracy and taxonomy proportions; Exact match uses all completed probes. Live AFS excludes operationally invalid responses before estimating batch accuracy. An all-invalid batch supplies insufficient evidence, not zero accuracy. Post-hoc fracture analysis likewise excludes invalid-only horizons and evaluates the sampled valid-response curve. For MAI-Thinking-1, the earlier fracture-depth interpretation was simply misleading. Appendix A.4 now documents invalid only observations at short horizons and high valid-conditional accuracy at z=20 and 30. On the corrected valid-only curves, Tiers 0, 2, and 3 have no observed fracture; Tier 1 has a right-censored terminal candidate at z=150, not a confirmed fracture.

4. State Tracking fallback

We removed the automatic State Tracking fallback. Across 39,728 valid observations, 13,633 now receive State tracking under explicit classification conditions, while four remain Unknown/Unmapped. Synthetic error-injection and precedence-sensitivity checks are reported in Appendices A.1 and A.9

5. Relative to prior work

We expanded section 2 to discuss Sinha et al., Wu et al., and related trajectory evaluations. ARCUS-X contributes an auditable joint decomposition of sequential execution into independently computable trajectory correctness, temporal continuity, exact agreement, horizon compliance, validity, and generation overhead under controlled rule perturbations. This is evidently shown by models with similar overall Step Accuracy, which can exhibit markedly different within-trajectory degradation and length compliance. The matched horizon temporal-position analysis demonstrates distinct early-to-late accuracy profiles, while decomposed measurements reveal differences hidden by aggregate correctness.

6. Gravity and CRI

The legacy gravity parameter gg is inert and retained only for historical configuration reproducibility. CRI is a secondary researcher-weighted summary for first glance

We appreciate the reviewer's specific, in-depth, and actionable criticisms!

Reviewer_Mf4gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ARCUS-X, an auditable benchmark for diagnosing failures in long sequences of dependent state transitions that final-answer accuracy can overlook. It uses procedurally generated, deterministic toroidal-grid tasks with exact ground-truth trajectories and controlled changes to action labels and transition rules. Its evaluation separates trajectory correctness, continuity, rule binding, execution-length control, response validity, and generation overhead, while adaptively sampling execution lengths to investigate failure behavior. Experiments across 13 models reveal that aggregate accuracy can conceal distinct failure patterns, motivating a multidimensional assessment of sequential reliability within this controlled setting.

Critical strengths
  1. The authors provide clear empirical evidence for evaluating reliability across multiple dimensions. Experiments across 13 models separately measure trajectory correctness, continuity, compliance with the requested number of execution steps, response validity, and generation overhead. For example, GPT-5-mini achieves 84.8% step accuracy, but its generation efficiency is only 25.9%, and 11.2% of its responses are invalid. These findings show why accuracy alone provides an incomplete picture of sequential execution reliability.

  2. The outcomes present relevantly strong auditability and reproducibility. Task generation from random seeds, independently validated reference trajectories, preserved raw responses, recorded experiment settings, and support for resuming interrupted runs make individual results traceable and easy to reanalyze. The evaluation also distinguishes operational failures, such as API or parsing errors, from execution mistakes in successfully parsed trajectories, making the diagnostic findings easier to interpret.

  3. Ground truth is computed, not judged. Every step in ARCUS-X has one correct answer derived mechanically from the transition rules (Sec. 3.3), independent of any model or annotator. Comparable benchmarks lack this: agent benchmarks (SWE-bench, WebArena, AgentBench) only observe final task success, confounded by tool use and environment noise, while trajectory benchmarks (AgentRewardBench, Plan-RewardBench, Chen et al. 2026) rely on human or LLM judges to label intermediate steps. ARCUS-X can therefore measure step accuracy, continuity, exact match and first divergence exactly, with no labeling noise, and seeded procedural generation (Sec. 3.2) supplies unlimited, contamination-free tasks whose horizon is set to exactly z steps.

Critical weaknesses
  1. Adaptive sampling confounds the main comparisons. AFS evaluates different execution lengths across models and perturbation tiers, so differences in pooled accuracy may partly reflect differences in sampled difficulty. The appendix’s comparisons at (z=20) and (z=30) help, but do not resolve this issue for the main aggregate results. The authors should evaluate all included models and perturbation tiers on matched underlying tasks at a common set of execution lengths, and report confidence intervals for these comparisons.

  2. The failure taxonomy lacks independent validation. Classification relies on heuristic rules applied in a fixed order: an incorrect initial coordinate is labeled a semantic failure, while some otherwise unexplained divergences default to state-tracking failure. These choices may influence the reported failure proportions, making it unclear how faithfully the categories capture the observed errors. The authors may need validate the labels using controlled error injection and expert annotations, and test whether the conclusions remain stable when the classification order changes.

  3. The failure taxonomy, the paper’s central contribution, is unvalidated and partly a catch-all. There is no human-labeled subset, no test that injects known failure types to check the classifier recovers them, and no agreement study, whereas AgentRewardBench and the Chen et al. attribution framework validate against expert annotations. State Tracking is the explicit fallback for any “valid, unmapped early divergence,” which likely explains why it absorbs 84–94% of errors for GLM-5, Qwen3-Coder, and Nemotron. “Semantic failure” fires when the model mis-copies an initial coordinate given verbatim in the prompt (a transcription error, not semantic interpretation), and “Horizon failure” merges two unrelated conditions. The category names promise cognitive attribution that the authors then disclaim, leaving labels with unclear meaning. One suggestion is that the author may want to validate the classifier on a human-annotated sample and on synthetic trajectories with injected failures of each type, report its precision/recall per category, and either rename the categories to describe the observable pattern (e.g., “persistent offset”) or remove the fallback so that unmapped errors stay in Unknown.

Questions for authors

Please see the question proposed in weaknesses section.

2 comments

We thank the reviewer for their constructive feedback and for recognizing ARCUS-X's value. We agree that AFS and taxonomy validity require careful interpretation.

For brevity, we summarize the major revisions:

1. Added exact-horizon comparisons at z = 20 and 30, coverage reporting, and bootstrap confidence intervals (Appendix A.2)

2. Clarified limitations of pooled comparisons over adaptively sampled horizons (sections 3.5, 4, 5.3)

3. Clarified taxonomy rules, and fallback to Unknown (Appendix A.1)

4. Synthetic error-injection checks and classification precedence sensitivity analyses (Appendices A.1 and A.9)

5. Clarified that taxonomy labels describe observable failure signatures, not independently established cognitive mechanisms

We address these in depth below:

1. Adaptive sampling and matched-horizon comparisons

We agree that AFS's nonuniform allocation of horizons complicates comparisons. We added exact-horizon analyses and uncertainty estimates. All 52 model-tier combinations have observations at both z= 20 and 30. Table 7 reports horizon coverage, and Table 8 gives conditional Step Accuracy and 95% Confidence intervals using master-seed-block bootstrap resampling. At z=20, cohort pooled Step Accuracy is 69.8% for Tier 0 versus 59.2% for Tier 3; at z=30, the values are 65.3% and 56.8%, respectively. These comparisons match the requested horizon but are not necessarily paired on identical underlying tasks. We explicitly distinguish horizon matching from task pairing and acknowledge that the original AFS design cannot establish controlled differences across all horizons. Accordingly, the main pooled results are now characterized as descriptive summaries of acquired observations.

2. Taxonomy validation and State Tracking fallback

We agree that deterministic diagnostic rules should not be interpreted as independently validated explanations of model cognitive abilities. We clarified the classification rules in Appendix A.1, removed the fallback to State Tracking, and moved it to Unknown/ Unmapped unless an explicit classification condition is satisfied. Across 39728 valid non-calibration observations, 13633 receive State Tracking failure under explicit conditions, while 4 are classified as Unknown/Unmapped. All 3123 valid None assignments correspond to exact requested output matches.

3. Error injection, precedence sensitivity and category interpretation

We added the suggested deterministic failure injection checks to examine whether the classifier recovers specific observable failure patterns.

We also tested alternative classification-precedence policies ( Appendix A.9). Horizon-first precedence reassigns 34.4% of primary labels and transition-first reassigns 33.6%. These findings demonstrate that taxonomy proportions can depend substantially on precedence choices.

We therefore explicitly interpret the taxonomy as a policy-dependent partition of observable trajectory behavior, not a uniquely identified account of latent cognitive causes.

For example, an initial-coordinate mismatch may receive the operational Semantic Interpretation label, but this does not establish whether its cause was transcription, instruction interpretation, or another mechanism. Likewise, Horizon Collapse describes output-length and trajectory patterns rather than one uniquely identified cognitive failure.

We acknowledge that synthetic consistency checks and precedence analyses do not replace expert annotated validation, or inter-annotator agreement. To address these limitations, we are initiating an independent annotation study to assess agreement with ARCUS-X's observable failure categories, including possibly ambiguous ones. Given the timeline for the rebuttal, we do not claim results from this ongoing study.

The primary trajectory measurements are computed directly against deterministic ground truth and do not depend on the taxonomy's classification precedence.

We appreciate the reviewer's detailed suggestions, and believe these changes substantially improve the interpretability and standing of ARCUS-X.

Dear reviewer Mf4g,

We wanted to follow up before the discussion period concludes. Our November 7 response details additional matched-horizon analyses, bootstrap confidence intervals, taxonomy error injection checks, and precedence-sensitivity results, with the corresponding manuscript edits. We would appreciate any further feedback on whether these additions and clarifications address your concerns.

We thank you for your careful and informative review!

Reviewer_9P4mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ARCUS-X to evaluate how reliably language models follow explicit rules over a sequence of state transitions. Models execute actions on a grid with wraparound boundaries and output the full trajectory. The benchmark measures step accuracy, continuity, length compliance, response validity, and generation overhead. It also changes action labels and transition rules, uses adaptive search to select sequence lengths, and report experimental results for 13 models.

Critical strengths
  • The paper constructs benchmark and design evaluation protocol for sequential state-transition execution reliability.
  • The controlled task provides exact ground truth for every step. This makes it possible to inspect where execution fails, rather than only checking the final answer.
  • The released code and raw responses are useful for examining the proposed benchmark.
Critical weaknesses
  • The paper emphasizes the need for process evaluation, but overlooks related long-context and memory studies, such as RULER, LongMemEval, and LoCoMo, making its additional contribution unclear.
  • The benchmark can report process failures, but it is unclear how these observations can guide model improvement. Large performance drops as the context length increases is already a known fact, see related works such as RULER, LongMemEval, LOCOMO. Controlling for these length effects and testing whether memory technique support reduce specific failures would further show the effectiveness and novelty of the proposed benchmark.
  • The benefit of adaptive sampling is unclear. The experiments do not show what it gains over testing models at predefined sequence lengths.
  • The task construction and evaluation process are not clearly explained. Section 3 lacks a complete example showing the task, model input, expected output, and evaluation. An overview figure would help readers follow the process.
  • Some unexplained early errors are labeled as state-tracking failures by default (see appendix A.1.2). It is unclear whether the reported failure performance reflect true differences between error types.
Questions for authors

Please see the previous Weaknesses section.

4 comments

We thank the reviewer for their constructive feedback and for recognizing the value of ARCUS-X's reproducibility, deterministic ground truth, and trajectory-level diagnostics.

For brevity, we include a summarized change log:

1. Relevant works described are implemented and positioned within Section 2, Related Works.

2. Practical utility and usage are described and addressed in Section 5.2, Practical Uses, as a diagnostic to help formulate and direct targeted investigations, among other things.

3. An End-to-End worked example of the pipeline, including the original prompt, model output, ground truth, and a worked example of the analysis, is provided at Appendix B, pg. 27-41. This example uses Ministral 14B 2512 at z=50z=50, across all 4 Tiers ( Tier 0 Baseline and Tiers 1-3).

4. AFS evaluated synthetically

5.Clarified taxonomy rules and fallback to Unknown.

We address the stated concerns in full below:

1. RULER, LongMemEval, and LoCoMo

We agree that these works are relevant and should be discussed more explicitly, and we thank you for bringing these to light.

RULER ( Hsieh et al., 2024) evaluates long-context capabilities through tasks that involve retrieval, multi-hop tracing, and aggregation. LongMemEval (Wu et al., 2025) evaluates long-term conversational memory, including information extraction, temporal reasoning, knowledge updates, and abstention. LoCoMo (Maharana et al., 2024) examines long-term conversational memory through multi-session interactions and associated reasoning tasks.

ARCUS-X addresses a narrower evaluation objective, distinct from the works stated above: the reliability of executing explicitly specified deterministic state transitions over dependent sequences. Rather than primarily testing information retrieval or memory across extended contexts, ARCUS-X evaluates each intermediate state against a computer ground truth.

Its contribution lies in jointly characterizing transition correctness, temporal continuity, exact trajectory agreement, horizon compliance, operational validity, and generational overhead under controlled rule perturbations.

We incorporated these references and clarify this distinction in the revised related works section.

2. Practical utility of the diagnostic framework

We agree that a practical interpretation of ARCUS-X's diagnostics deserves greater emphasis.

ARCUS-X is designed to distinguish failure behaviors that aggregate metrics, such as aggregated accuracy, cannot alone identify. These distinctions can inform targeted follow-up investigations: continuity and first divergence measurements to help localize where execution deteriorates; horizon compliance identifies incomplete or excessive trajectories; controlled rule perturbations can reveal sensitivity to action labels or transformed transition conventions.

The reported results show why this separation matters. Table 1 reports similar pooled Step Accuracy for GLM-5 (56.2%) and Mistral Large 2512 (56.0%), but substantially different Horizon Compliance (98.4% and 45.2%, respectively). This illustrates why trajectory correctness and output-length control should be examined separately. Hence, our claim is one of diagnostic utility, not demonstrated intervention effectiveness. We make this distinction and the potential applications clearer with the addition of Section 5.2

3. End-to-end worked example of the evaluation pipeline. (See Appendix B, pg. 27-41) We added this example to clarify how the deterministic task specification is translated into independently scored trajectory-level measurements. We have added all 4 tier examples to Appendix B at z=50 on Ministral 14B 2512.

4. AFS We agree that the original presentation did not adequately establish what adaptive sampling gains over predefined horizons. To address this, we added a controlled synthetic sampling-efficiency evaluation (Appendix A.10), comparing AFS to alternative adaptive sampling policies under common response budgets. At an 81-response budget, AFS localized 59.3% of true first threshold crossings within five horizons, compared with 46.5% for fixed-grid sampling, a paired difference of 12.8 percentage points (95% CI [10.1, 15.4]).

5. State tracking fallback We have clarified the decision rules in Appendix A.1. In the 13-model cohort, 13,633 of 39,728 valid non-calibration observations receive a State Tracking Failure label under an explicit classification condition. Four valid observations remain Unknown / Unmapped rather than being assigned a mechanism by default. We additionally report deterministic synthetic error-injection checks and precedence-sensitivity analyses. These establish implementation consistency and reveal dependence on classification policy, but do not constitute independent semantic validation.

We greatly appreciate the reviewer's suggestions and believe these clarifications and additions will strengthen the position and value of ARCUS-X.

Dear reviewer 9P4m,

We wanted to follow up as the discussion period ends. We have addressed your concerns through the manuscript revisions and additional analyses detailed in our November 7 response, including the AFS sampling-efficiency evaluation, expanded related work, and the requested end to end worked examples. We would greatly appreciate any further feedback on whether these revisions address your concerns!

Thank you again for your time and consideration

Thank you for your response. The authors state that the paper scope is narrower than the long-context and memory benchmarks (RULER, LongMemEval, LoCoMo), and that its contribution lies in jointly characterizing transition correctness, temporal continuity, exact trajectory agreement, horizon compliance, operational validity, and generation overhead. However, I still do not see why these measurements are important for LLMs than the measurements of other benchmarks, or how we can use them to improve LLMs. The paper limits its practical-use claim to diagnostic hypotheses and gives no concrete potential application in Section 5.2. No experiment tests whether any method reduces a specific failure type. Therefore, I think the contribution of this work is not sufficient in its current version. I will keep my original rating.

We thank the reviewer for emphasizing the importance of demonstrating practical utility beyond descriptive failure analysis.

To address this concern, we conducted a utility validation suite which is a new controlled intervention experiment using the original ARCUS-X task generator and an explicitly corrected (z+1)(z+1) output convention. This evaluates Ministral 14B 2512 on 400 tasks (1200 model responses) across 2 horizons and 2 perturbation tiers. Each task is evaluated under 3 paired inference-time conditions, an unmodified control, horizon verification, and transition verification. The experiment uses a fixed calibration/held out split, with 236 held out tasks, paired permutation tests, bootstrap confidence intervals, and Holm correction across three prespecified hypotheses.

The leading result is a statistically significant, intervention-specific improvement in transition continuity. Transition verification exceeds horizon verification by 12.99 percentage points (95% CI: 9.07-17.03 pp; Holm-adjusted p=1.50∗10−4p=1.50*10^{-4}. This shows that targeted prompting can measurably alter the trajectory-level behavior isolated by ARCUS-X which provides experimental evidence of diagnostic sensitivity rather than merely just retrospective failure categorization.

We also quantify the computational trade off that we found: transition verification averages 980 output tokens versus 436 for horizon verification, corresponding to approximately 41.9 additional output tokens per percentage-point continuity gain. The effect is therefore measurable but not cost free. Horizon verification does not significantly improve full-length compliance relative to transition verification, and diagnostic-guided intervention selection does not outperform calibration Exact Match based selection in this experiment. These results provide direct empirical evidence that ARCUS-X diagnostics distinguish intervention responsive dimensions of sequential execution. We think that this advances the benchmark beyond observational analysis by connecting a specific diagnostic to a controlled behavioral intervention. This is all documented in the new appendix section.

We thank the reviewer for their care and help in making ARCUS-X a better paper!

Comments on the paper

4 comments

We thank the reviewer for the careful follow-up and acknowledgment of the earlier corrections. We agree that both questions are important for the interpretation of ARCUS-X's claims. We have now completed two new post-hoc analyses. We have incorporated the results into the revised manuscript, including the main results + the relevant appendices.

1. Taxonomy precedence sensitivity

We agree that the original exclusive taxonomy signatures are sensitive to the precedence policy, particularly for Qwen 3 Coder and GLM-5. We have therefore made this dependence explicit in the abstract, Section 4.4, and the discussion. We have also added precedence independent, non-exclusive diagnostic and label-transition analyses in Appendix A.9.1. The non-exclusive results show that multiple observable diagnostic conditions frequently coexist. The complete label-transition analysis confirms your interpretation. Under transition first, 1994/2160 Qwen 3 Coder observations and 2132/2639 GLM-5 observations move State Tracking to Transition Rule. Thus, their originally dominant State Tracking labels cannot be classified as precedence-invariant. The high activation rates of both indicators show that the trajectories often satisfy overlapping diagnostic conditions, with the exclusive assignment determined by the specified precedence.

Accordingly, we have retained the original deterministic primary taxonomy for reproducibility, but now explicitly distinguish exclusive labels under a chosen policy from overlapping observable diagnostic indicators. We do not interpret either as uniquely identified cognitive failure mechanisms. The revised abstract and main text discussion reflect this.

2.Requested output vs full action convention

We agree that supplying z actions while requesting z coordinates, including the initial state, introduces an ambiguity. To address this, we have made the distinction explicit in Section 3.5 and now report both Exact Match conventions in the main results. Both were recomputed using the same observations. The complete 13 model comparison can be found in Appendix A.1.3.

The differences are particularly substantial for certain models. Kimi K2.5's Exact Match increases from 0.06% under the requested output to 39.06% under the full-action convention. GPT-5 mini from 36.00% to 42.40%. We further decomposed the original Horizon assignments (Appendix A.1.3), finding that 2,570/3,503 (73.4%) of Kimi K2.5's Horizon labels correspond to exact full-action executions, compared with 37/1,628 (2.3%) for Mistral Large. Other overlong responses were separately examined for coordinate and transition errors rather than assumed correct. We also recomputed the taxonomy under a full-action-aware convention (table 9). Kimi K2.5's Horizon share decreases from 56.1% to 14.9%, GPT-5-mini's from 49.0% to 1.2%, and Mistral Large's from 49.4% to 44.6%. These results demonstrate that Horizon signatures are sensitive to the output-length convention to varying degrees across models. We have revised the manuscript accordingly.

We believe this makes the central contribution more precise, as ARCUS-X can separate trajectory execution, requested output compliance, and overlapping error signatures, while exposing how eval conventions affect their interpretations! We appreciate the reviewer deeply for their timely and thorough response that allowed us to address these issues.

Dear reviewer 9bBK,

We wanted to briefly follow up as the author reviewer discussion period approaches a conclusion.

We have addressed both of your follow up questions via additional quantitative analyses and corresponding manuscript revisions, as detailed in our November 13th response.

We would greatly appreciate any further feedback you may have on whether these revisions adequately ( and sufficiently) address your concerns.

Thank you again for your careful consideration and constructive comments!

Thank you for the detailed responses and additional analyses; the paper is now much more transparent. The new tables answer the two questions from my previous comment, but they confirm rather than resolve both concerns.

Taxonomy. Under horizon-first, almost all of Mistral Large’s Transition Rule labels (811 of 816) and Kimi K2.5’s Semantic labels (1,441 of 1,449) also become Horizon Collapse (Table 30), so every non-Horizon signature highlighted in the abstract is precedence-dependent. The multi-label frequencies I asked for show why: 98% of probes that trigger State Tracking also trigger Transition Rule (Table 29), and the two rates are nearly identical on the State-dominated models (Table 28). Only the Horizon labels remain stable, yet for Kimi K2.5, 73% of its Horizon labels are exact z + 1 executions (Table 7). It is hard to see what the exclusive taxonomy adds beyond the underlying indicators themselves.

Output length. The main metrics and taxonomy still score against z, even though the generator, live AFS, the Appendix B ground truth, and the original submission (L_GT = z + 1, Section 3.5) all use z + 1. Under z, all 9,440 exact z + 1 answers are labeled Horizon Collapse (Tables 6 and 7); under z + 1, the same happens to GPT-5-mini’s 2,052 exact z-length answers (Table 10). Table 9 avoids both only by accepting either length. More importantly, the mismatch affect how models answer, not just how they are scored. In Appendix B, Ministral-14B at z = 50 outputs exact 51-state answers in Tiers 1 and 3 (the Tier 3 answer is labeled “50 steps” but lists all 51). Its Tier 2 answer exactly matches the first 50 states, and in Tier 0 it skips the 12th action yet still ends on the 50th. Rescoring cannot undo that.

Because both issues stem from how the benchmark is set up rather than how the results are reported, I am keeping my score for now. I would encourage the authors to fix the prompt and show what the taxonomy adds beyond the raw indicators. With those changes, ARCUS-X could be a really useful diagnostic benchmark.

We thank the reviewer for the careful follow up!

Due to the mutually exclusive taxonomy is precedence-sensitive as alternative orderings reassign 33.6-34.4% of valid primary labels, we have implemented this:

We clarify that these labels are operational classifications rather than uniquely identifiable failure mechanisms, and added non-exclusive indicator frequencies and label transition analyses (appendix A.9.1). We also explicitly distinguish the original requested output (z)(z) convention from the full-action (z+1)(z+1) convention, reporting both Exact match measures and sensitivity analyses. We acknowledge that retrospective rescoring cannot eliminate the original prompting ambiguity. To provide new evidence under an unambiguous (z+1)(z+1) protocol, we conducted AUVS-1 (app. A.11), comprising 1200 responses across 400 paired tasks. Transition verification improved held-out continuity by 12.99 percentage points relative to horizon verification (95% CI: 9.07-17.03 pp; Holm-adjusted (p=1.50∗10−4))(p=1.50*10^{-4})). Although this single-model experiment does not retroactively validate the original cross- model taxonomy but it demonstrates that ARCUS-X's trajectory-level diagnostics capture statistically significant intervention sensitive behavior under a corrected output convention.

We believe these revisions strengthen the benchmark's methodological clarity and empirical utility.

We thank the reviewer for their support and commitment to helping shape ARCUS-X into a stronger tool and paper!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.