- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how long-horizon agents should be managed when they become stalled during execution. The authors consider four common intervention strategies: continuing the current agent, providing advice for in-place revision, restarting with a clean context, or replacing the agent with a new session. The paper argues that existing agent logs cannot reliably compare these interventions because each observed intervention is confounded with multiple factors.
To address this issue, the authors design a controlled 2 × 2 factorial experiment that separates context rebuilding and advice provision. All interventions start from the same stalled checkpoint with identical remaining budget, frozen executor, and shared artifacts when applicable. Experiments on Craftax with two executors show a null result: state-level selection among the four interventions does not provide measurable improvement under the tested setting. The paper also identifies one weak-regime effect where advice without context repair harms performance, while rebuilding context removes this effect
The title and abstract shown on openreview are inconsistent with those in the submitted pdf, which makes it unclear which version should be considered the official submission
Critical strengths
The strongest aspect of this paper is the controlled evaluation methodology. Instead of comparing naturally occurring agent interventions, the authors construct multiple counterfactual branches from the same stalled state. This design addresses an important issue in agent evaluation, where improvements after replacement or revision are often difficult to attribute to a specific factor.
The paper demonstrates strong experimental discipline. The authors carefully analyze statistical issues and explicitly revise previous conclusions after identifying max-of-k bias in earlier estimates. This transparency improves confidence in the reported measurements and shows awareness of potential evaluation pitfalls.
The factorization between advice and context rebuilding provides a more detailed understanding of intervention effects. Rather than treating agent replacement or restart as a single operation, the paper attempts to isolate which components actually contribute to recovery performance.
Critical weaknesses
The main weakness is the limited contribution beyond the controlled measurement itself. The authors state that no component of the framework is new, and the primary contribution is the reported null finding. While carefully designed negative results can be valuable, the paper needs stronger evidence that this finding changes the understanding of long-horizon agent management in a broader sense.
The generality of the conclusion is uncertain. The experiments are conducted only on Craftax with two executors, and the tested intervention mechanisms represent only a limited subset of possible agent recovery strategies. Therefore, the observed null result may reflect limitations of the benchmark or intervention implementation rather than a general property of agent management.
And some intervention components may not represent the strongest available approaches. For example, the quality of advice generation and context reconstruction can significantly influence recovery outcomes. The failure of the current implementations to improve performance does not necessarily rule out the potential value of stronger intervention mechanisms.
Some parts of the paper focus excessively on unnecessary implementation details and experimental procedures, making the presentation feel closer to an engineering report rather than a research paper
Another weakness is that the visualization quality could be improved. Several figures are difficult to interpret and do not effectively communicate the main experimental findings. Since the contribution of this paper relies heavily on careful experimental comparison and analysis, clearer visual presentation would help readers better understand the intervention settings and the implications of the results.
Questions for authors
-
Since the main contribution is a controlled null result, what evidence suggests that the finding reflects a fundamental property of agent intervention rather than limitations of the benchmark or intervention implementations?
-
How sensitive are the results to the quality of the advice generator and handover package construction? Could stronger advice or more effective context reconstruction change the observed conclusions?
-
The experiments focus only on Craftax. Do the authors have preliminary evidence from other long-horizon environments, or arguments explaining why this environment is sufficient for evaluating intervention strategies?
-
The intervention manager is relatively limited compared with a fully trained decision model. Could a stronger learned intervention policy discover useful state-dependent strategies that are not captured by the current evaluation?
5 comments
W4. "Engineering report." We moved the learned manager (now Appendix K), the enforcement of each sharing constraint and the audit of silent defects (Appendices J and O), and the campaign-level table (Appendix M) out of the main text. The main text is now: the question and design (Section 3, with one data table), the estimands and statistics (Section 4), the results (Section 5), and threats and scope (Section 6). It ends on page 8.
W5. Figures hard to interpret. The four hand-drawn figures of the main text are replaced by one script-generated figure (Figure 1), and the two design diagrams moved to Appendix J. In Figure 1, panel (a) shows every contrast for every collection with its 95% and 90% intervals against the ±0.20 equivalence band, so a reader sees at once which contrasts are equivalent to zero, which resolve and which are undetermined; panel (b) shows, for each collection, the bracket on what per-state choice could gain, from the cross-fitted lower bound to the 95% upper bounds.
W6. Title and abstract inconsistent with the PDF. Agreed. The PDF title now begins "Replace or Repair?", as on OpenReview, and the abstract on OpenReview is replaced by the revised one, which no longer mentions environments for which no data were collected.
Q1. Fundamental property or benchmark/implementation limit? We cannot claim a fundamental property, and the revision no longer suggests one. What the data support is narrower: in this environment, with these fixed generators and this window, the four interventions are worth the same to within ±0.20 achievements for a strong executor, and per-state choice among them is bracketed. Two observations argue against a pure floor artefact: the equivalences hold for an executor 3.6× stronger whose handovers are not truncation-bound, and the instrument does resolve a 0.14-achievement effect (E1's pre-specified N−R, robust to removing contaminated runs). One implementation limit cuts the other way and is now stated prominently: the two-observation window, which makes the context factor weak.
Q4. Could a stronger learned policy find state-dependent gains? Any policy, learned or not and however expressive, that picks one of C, R, F, N from what is known at the decision earns at most G*, the gain of the oracle that knows each state's and budget's expected outcomes (Equation 1). The revision bounds G* directly: at most 0.68 achievements with the weak executor and 0.87 with the strong (one-sided 95%; Table 3), so no learned manager can exceed these. As a direct test, lookup policies over observable features (stall flag × remaining budget), fitted on half of the maps and scored on the other half, gain −0.008 [−0.079, +0.061] (E1) and +0.025 [−0.104, +0.116] (E2), although their in-sample values are +0.063 and +0.113: the gap is the winner's curse a learned manager also faces. The heterogeneity a learner would need is present in some collections (permutation p = 0.04 in E1, 0.01 in E2R) but small, with standard deviations of 0.2–0.8 achievements against a single-repeat noise of 1.5–2.7. Our pre-registered manager scores +0.00006 held out (Appendix K). A policy that changes the actions themselves, such as a better advisor or one that inspects the generated advice before deciding, is not bounded by G* and is outside this study.
Dear Authors,
Thank you for the detailed rebuttal and substantial revision. I appreciate your willingness to acknowledge and correct important issues rather than defend the original claims.
The revised interpretation of the cross-fitted statistic, the new upper bounds, the replacement of MDE-based arguments with confidence intervals and equivalence tests, and the reconciliation of data and pre-registration inconsistencies are meaningful improvements to the paper's methodological rigor.
However, I still have concerns about the scientific contribution, generalizability, and presentation quality.
1. Scientific contribution and generalizability remain limited.
The experiments remain restricted to Craftax and a fixed set of interventions. The revised manuscript also discloses that the executor's context contained exactly two observations at every sampled state. This limits the conclusions about context accumulation and reconstruction in realistic long-horizon agents.
The artifact analysis does not establish the quality of the generated advice or handover information, and stronger interventions remain unexplored. The new oracle bounds apply only to selection among the four implemented interventions; they do not rule out gains from stronger strategies or state-dependent policies. The post-hoc choice of the equivalence margin and exploratory E2R results also warrant caution when interpreting the negative findings.
2. The visual explanation of the core mechanism remains insufficient.
I do not think this issue has been adequately addressed. Figure 1 mainly presents statistical results and oracle bounds, rather than explaining the design. The diagrams illustrating the four interventions and branching procedure have been moved to Appendix J (Figures 2 and 3).
For a paper centered on controlled comparisons of agent interventions, I would expect a clear diagram in the main text. Readers should understand what each intervention changes, which artifacts are shared, and how the comparisons work without consulting the appendix.
3. Writing quality and academic presentation still need improvement.
Despite the revision, parts of the manuscript still read more like an engineering report than an academic paper. Examples include implementation details such as verify_paired_branches, env_rng_mode="shared", manifest and analysis-script paths, and lengthy descriptions of implementation defects and regression checks. These support reproducibility, but the authors should distinguish essential methodological information from low-level documentation. Moving details to the appendix does not fully resolve this issue.
The writing also contains formulaic and overly stylized elements, such as repeated headings ("What we do", "What we find", "What we correct") and rhetorical statements like "A dead executor does not look like a crash. It looks like a result." Such phrasing adds limited scientific information and can make the manuscript feel overly packaged.
As a constructive suggestion, I encourage the authors to study Best Paper award-winning papers from previous years, purely from the perspective of academic writing and presentation, not their research topics. This can illustrate how strong papers communicate ideas, structure narratives, use intuitive figures, and present evidence concisely.
I also want to clarify that I am not suggesting authors should avoid AI tools for writing assistance. Regardless of how the initial text is produced, the final manuscript needs careful human review to avoid formulaic expressions, repetition, and rhetoric with limited informational value. The writing should communicate the research naturally and precisely.
Overall, the revision improves methodological rigor and statistical credibility, but the scientific contribution, experimental representativeness, and presentation still have substantial room for improvement. Correcting the original statistical interpretation does not, by itself, increase the significance or breadth of the contribution.
Nevertheless, in recognition of the revision, I am willing to increase my rating from 1 (Clear Reject) to 2 (Weak Reject). I retain my reservations about the paper's significance and generalizability.
Thank you again for the careful response.
Thank you for the careful reading and for the concrete questions. Every change below is in the revised PDF (Appendix A lists them).
W1. Contribution beyond the controlled measurement. We agree that no component of the instrument is new, and the introduction now says so in one sentence. The revision makes three contributions explicit. (i) A paired counterfactual instrument for interventions on agents: four arms from one restored state under one budget and one random stream, with every pairing constraint verified on every one of 5,194 quadruples (Table 1). (ii) A valid way to report what per-state choice is worth from repeated noisy branches: a lower bound from a held-out selector, upper bounds with and without a model, and a permutation calibration (Section 4.3, Appendix C). This matters beyond our setting: in-sample "oracle headroom" figures are common in agent-harness papers, and our own withdrawn +22% shows how far they can be off: shuffling the four arm labels within each quadruple reproduces it in full (1.06 achievements under shuffling against the 1.02 we had reported; p = 0.95). (iii) The measurement, now positive in form: with the strong executor the pre-registered contrasts are equivalent to zero within ±0.20 achievements (1.3% of the Continue mean), and no selector we can build beats the best single arm.
W2/Q3. Generality. We claim nothing outside Craftax, and the abstract now says the data come from one environment. Section 6 explains the choice: the instrument needs, at once, a long horizon with sparse compositional progress (128 turns, 67 achievements behind a tech tree), stochastic dynamics so that repeats are informative, bit-exact snapshot and restore so that four arms start from one state on one random stream, and cheap branches because the bracket needs repeats. It also says what could change elsewhere, in both directions: in software or web tasks the working context holds files and logs that matter, so keeping it is a stronger manipulation; restoring containers is slower, so repeats are scarcer and brackets wider; binary outcomes cost power; and failures may be easier to diagnose from text, which would favour advice. The Factorio and SWE-bench Pro adapters exist and are tested but were never run for data.
W3/Q2. Are these the strongest interventions; sensitivity to advice and handover quality. Two answers. First, a limitation we should have stated in the submission: the executor's working window held exactly two observations at every one of 2,211 sampled states (Table 14). "Keep the context" therefore kept two observations, and "rebuild" replaced them with a handover of at most 640 tokens; the keep/rebuild factor is weak by construction, and the revision narrows every claim about F and N to this manipulation (Sections 3.1, 6). Second, the data allow one clean test of artifact quality: z and H0 are regenerated for every repeat of a fixed state, so within-state differences in their length and truncation are generator randomness, free of state confounding (Table 13). Advice length predicts R−C with opposite signs in the two strong-executor collections (+0.89 and −0.70 achievements per 100 tokens; pooled +0.11 [−0.33, +0.55]); handover length predicts nothing (pooled slope on F−C +0.07 [−0.19, +0.30]). Across executors, the strong model's artifacts are longer and rarely truncated (H0 at its cap in 6% of repeats against 84%), and its contrasts are still equivalent to zero. The texts were not stored, so we cannot measure specificity or content. A stronger advisor, a learned context editor or a larger window is a different intervention; Section 6 says these are outside what the data measure.