acceptodds
Under review as a conference paper at ICLR 2027

Replace or Repair? Isolating Handover Value in Stalled Agents

Abstract

When a long-horizon agent stalls, should its manager replace it with a successor built for the current state, or do advice and a clean context suffice? Existing trajectories cannot answer this: they confound replacement's benefit with that of the advice and context supplied alongside it. We isolate the effect with a 2×2 factorial that resumes every arm from the same frozen snapshot of a real stalled run under a fixed budget and executor, crossing context (keep or rebuild) against advice (withhold or supply). Revise and Replace share one generator's suggestion verbatim; Fresh and Replace share one handover package; all arms reread the same logs at equal charge. The interaction term measures what rebuilding the executor buys beyond advice and context repair alone — a diagnostic, not a claim that identity confers ability; if either subsumes the gain, replacement's contribution reduces to context repair. On these causal estimates we train a budget-aware manager that selects among interventions by expected value rather than the incumbent's failure risk, and evaluate it against failure-risk triggers, periodic restarts, prompted management, and learned context editing in Craftax, the Factorio Learning Environment, SWE-bench Pro, and ARC-AGI-3. The frozen manager is then evaluated on unseen predecessor protocols, a second model family, and shifted budgets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, long-horizon agents, agent handover, context management, counterfactual evaluation, experimental design, value estimation, agent orchestration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2eA9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how long-horizon agents should be managed when they become stalled during execution. The authors consider four common intervention strategies: continuing the current agent, providing advice for in-place revision, restarting with a clean context, or replacing the agent with a new session. The paper argues that existing agent logs cannot reliably compare these interventions because each observed intervention is confounded with multiple factors.

To address this issue, the authors design a controlled 2 × 2 factorial experiment that separates context rebuilding and advice provision. All interventions start from the same stalled checkpoint with identical remaining budget, frozen executor, and shared artifacts when applicable. Experiments on Craftax with two executors show a null result: state-level selection among the four interventions does not provide measurable improvement under the tested setting. The paper also identifies one weak-regime effect where advice without context repair harms performance, while rebuilding context removes this effect

The title and abstract shown on openreview are inconsistent with those in the submitted pdf, which makes it unclear which version should be considered the official submission

Critical strengths

The strongest aspect of this paper is the controlled evaluation methodology. Instead of comparing naturally occurring agent interventions, the authors construct multiple counterfactual branches from the same stalled state. This design addresses an important issue in agent evaluation, where improvements after replacement or revision are often difficult to attribute to a specific factor.

The paper demonstrates strong experimental discipline. The authors carefully analyze statistical issues and explicitly revise previous conclusions after identifying max-of-k bias in earlier estimates. This transparency improves confidence in the reported measurements and shows awareness of potential evaluation pitfalls.

The factorization between advice and context rebuilding provides a more detailed understanding of intervention effects. Rather than treating agent replacement or restart as a single operation, the paper attempts to isolate which components actually contribute to recovery performance.

Critical weaknesses

The main weakness is the limited contribution beyond the controlled measurement itself. The authors state that no component of the framework is new, and the primary contribution is the reported null finding. While carefully designed negative results can be valuable, the paper needs stronger evidence that this finding changes the understanding of long-horizon agent management in a broader sense.

The generality of the conclusion is uncertain. The experiments are conducted only on Craftax with two executors, and the tested intervention mechanisms represent only a limited subset of possible agent recovery strategies. Therefore, the observed null result may reflect limitations of the benchmark or intervention implementation rather than a general property of agent management.

And some intervention components may not represent the strongest available approaches. For example, the quality of advice generation and context reconstruction can significantly influence recovery outcomes. The failure of the current implementations to improve performance does not necessarily rule out the potential value of stronger intervention mechanisms.

Some parts of the paper focus excessively on unnecessary implementation details and experimental procedures, making the presentation feel closer to an engineering report rather than a research paper

Another weakness is that the visualization quality could be improved. Several figures are difficult to interpret and do not effectively communicate the main experimental findings. Since the contribution of this paper relies heavily on careful experimental comparison and analysis, clearer visual presentation would help readers better understand the intervention settings and the implications of the results.

Questions for authors
  1. Since the main contribution is a controlled null result, what evidence suggests that the finding reflects a fundamental property of agent intervention rather than limitations of the benchmark or intervention implementations?

  2. How sensitive are the results to the quality of the advice generator and handover package construction? Could stronger advice or more effective context reconstruction change the observed conclusions?

  3. The experiments focus only on Craftax. Do the authors have preliminary evidence from other long-horizon environments, or arguments explaining why this environment is sufficient for evaluating intervention strategies?

  4. The intervention manager is relatively limited compared with a fully trained decision model. Could a stronger learned intervention policy discover useful state-dependent strategies that are not captured by the current evaluation?

5 comments

Thank you for the careful reading and for the concrete questions. Every change below is in the revised PDF (Appendix A lists them).

W1. Contribution beyond the controlled measurement. We agree that no component of the instrument is new, and the introduction now says so in one sentence. The revision makes three contributions explicit. (i) A paired counterfactual instrument for interventions on agents: four arms from one restored state under one budget and one random stream, with every pairing constraint verified on every one of 5,194 quadruples (Table 1). (ii) A valid way to report what per-state choice is worth from repeated noisy branches: a lower bound from a held-out selector, upper bounds with and without a model, and a permutation calibration (Section 4.3, Appendix C). This matters beyond our setting: in-sample "oracle headroom" figures are common in agent-harness papers, and our own withdrawn +22% shows how far they can be off: shuffling the four arm labels within each quadruple reproduces it in full (1.06 achievements under shuffling against the 1.02 we had reported; p = 0.95). (iii) The measurement, now positive in form: with the strong executor the pre-registered contrasts are equivalent to zero within ±0.20 achievements (1.3% of the Continue mean), and no selector we can build beats the best single arm.

W2/Q3. Generality. We claim nothing outside Craftax, and the abstract now says the data come from one environment. Section 6 explains the choice: the instrument needs, at once, a long horizon with sparse compositional progress (128 turns, 67 achievements behind a tech tree), stochastic dynamics so that repeats are informative, bit-exact snapshot and restore so that four arms start from one state on one random stream, and cheap branches because the bracket needs repeats. It also says what could change elsewhere, in both directions: in software or web tasks the working context holds files and logs that matter, so keeping it is a stronger manipulation; restoring containers is slower, so repeats are scarcer and brackets wider; binary outcomes cost power; and failures may be easier to diagnose from text, which would favour advice. The Factorio and SWE-bench Pro adapters exist and are tested but were never run for data.

W3/Q2. Are these the strongest interventions; sensitivity to advice and handover quality. Two answers. First, a limitation we should have stated in the submission: the executor's working window held exactly two observations at every one of 2,211 sampled states (Table 14). "Keep the context" therefore kept two observations, and "rebuild" replaced them with a handover of at most 640 tokens; the keep/rebuild factor is weak by construction, and the revision narrows every claim about F and N to this manipulation (Sections 3.1, 6). Second, the data allow one clean test of artifact quality: z and H0 are regenerated for every repeat of a fixed state, so within-state differences in their length and truncation are generator randomness, free of state confounding (Table 13). Advice length predicts R−C with opposite signs in the two strong-executor collections (+0.89 and −0.70 achievements per 100 tokens; pooled +0.11 [−0.33, +0.55]); handover length predicts nothing (pooled slope on F−C +0.07 [−0.19, +0.30]). Across executors, the strong model's artifacts are longer and rarely truncated (H0 at its cap in 6% of repeats against 84%), and its contrasts are still equivalent to zero. The texts were not stored, so we cannot measure specificity or content. A stronger advisor, a learned context editor or a larger window is a different intervention; Section 6 says these are outside what the data measure.

W4. "Engineering report." We moved the learned manager (now Appendix K), the enforcement of each sharing constraint and the audit of silent defects (Appendices J and O), and the campaign-level table (Appendix M) out of the main text. The main text is now: the question and design (Section 3, with one data table), the estimands and statistics (Section 4), the results (Section 5), and threats and scope (Section 6). It ends on page 8.

W5. Figures hard to interpret. The four hand-drawn figures of the main text are replaced by one script-generated figure (Figure 1), and the two design diagrams moved to Appendix J. In Figure 1, panel (a) shows every contrast for every collection with its 95% and 90% intervals against the ±0.20 equivalence band, so a reader sees at once which contrasts are equivalent to zero, which resolve and which are undetermined; panel (b) shows, for each collection, the bracket on what per-state choice could gain, from the cross-fitted lower bound to the 95% upper bounds.

W6. Title and abstract inconsistent with the PDF. Agreed. The PDF title now begins "Replace or Repair?", as on OpenReview, and the abstract on OpenReview is replaced by the revised one, which no longer mentions environments for which no data were collected.

Q1. Fundamental property or benchmark/implementation limit? We cannot claim a fundamental property, and the revision no longer suggests one. What the data support is narrower: in this environment, with these fixed generators and this window, the four interventions are worth the same to within ±0.20 achievements for a strong executor, and per-state choice among them is bracketed. Two observations argue against a pure floor artefact: the equivalences hold for an executor 3.6× stronger whose handovers are not truncation-bound, and the instrument does resolve a 0.14-achievement effect (E1's pre-specified N−R, robust to removing contaminated runs). One implementation limit cuts the other way and is now stated prominently: the two-observation window, which makes the context factor weak.

Q4. Could a stronger learned policy find state-dependent gains? Any policy, learned or not and however expressive, that picks one of C, R, F, N from what is known at the decision earns at most G*, the gain of the oracle that knows each state's and budget's expected outcomes (Equation 1). The revision bounds G* directly: at most 0.68 achievements with the weak executor and 0.87 with the strong (one-sided 95%; Table 3), so no learned manager can exceed these. As a direct test, lookup policies over observable features (stall flag × remaining budget), fitted on half of the maps and scored on the other half, gain −0.008 [−0.079, +0.061] (E1) and +0.025 [−0.104, +0.116] (E2), although their in-sample values are +0.063 and +0.113: the gap is the winner's curse a learned manager also faces. The heterogeneity a learner would need is present in some collections (permutation p = 0.04 in E1, 0.01 in E2R) but small, with standard deviations of 0.2–0.8 achievements against a single-repeat noise of 1.5–2.7. Our pre-registered manager scores +0.00006 held out (Appendix K). A policy that changes the actions themselves, such as a better advisor or one that inspects the generated advice before deciding, is not bounded by G* and is outside this study.

Dear Authors,

Thank you for the detailed rebuttal and substantial revision. I appreciate your willingness to acknowledge and correct important issues rather than defend the original claims.

The revised interpretation of the cross-fitted statistic, the new upper bounds, the replacement of MDE-based arguments with confidence intervals and equivalence tests, and the reconciliation of data and pre-registration inconsistencies are meaningful improvements to the paper's methodological rigor.

However, I still have concerns about the scientific contribution, generalizability, and presentation quality.

1. Scientific contribution and generalizability remain limited.

The experiments remain restricted to Craftax and a fixed set of interventions. The revised manuscript also discloses that the executor's context contained exactly two observations at every sampled state. This limits the conclusions about context accumulation and reconstruction in realistic long-horizon agents.

The artifact analysis does not establish the quality of the generated advice or handover information, and stronger interventions remain unexplored. The new oracle bounds apply only to selection among the four implemented interventions; they do not rule out gains from stronger strategies or state-dependent policies. The post-hoc choice of the equivalence margin and exploratory E2R results also warrant caution when interpreting the negative findings.

2. The visual explanation of the core mechanism remains insufficient.

I do not think this issue has been adequately addressed. Figure 1 mainly presents statistical results and oracle bounds, rather than explaining the design. The diagrams illustrating the four interventions and branching procedure have been moved to Appendix J (Figures 2 and 3).

For a paper centered on controlled comparisons of agent interventions, I would expect a clear diagram in the main text. Readers should understand what each intervention changes, which artifacts are shared, and how the comparisons work without consulting the appendix.

3. Writing quality and academic presentation still need improvement.

Despite the revision, parts of the manuscript still read more like an engineering report than an academic paper. Examples include implementation details such as verify_paired_branches, env_rng_mode="shared", manifest and analysis-script paths, and lengthy descriptions of implementation defects and regression checks. These support reproducibility, but the authors should distinguish essential methodological information from low-level documentation. Moving details to the appendix does not fully resolve this issue.

The writing also contains formulaic and overly stylized elements, such as repeated headings ("What we do", "What we find", "What we correct") and rhetorical statements like "A dead executor does not look like a crash. It looks like a result." Such phrasing adds limited scientific information and can make the manuscript feel overly packaged.

As a constructive suggestion, I encourage the authors to study Best Paper award-winning papers from previous years, purely from the perspective of academic writing and presentation, not their research topics. This can illustrate how strong papers communicate ideas, structure narratives, use intuitive figures, and present evidence concisely.

I also want to clarify that I am not suggesting authors should avoid AI tools for writing assistance. Regardless of how the initial text is produced, the final manuscript needs careful human review to avoid formulaic expressions, repetition, and rhetoric with limited informational value. The writing should communicate the research naturally and precisely.

Overall, the revision improves methodological rigor and statistical credibility, but the scientific contribution, experimental representativeness, and presentation still have substantial room for improvement. Correcting the original statistical interpretation does not, by itself, increase the significance or breadth of the contribution.

Nevertheless, in recognition of the revision, I am willing to increase my rating from 1 (Clear Reject) to 2 (Weak Reject). I retain my reservations about the paper's significance and generalizability.

Thank you again for the careful response.

Reviewer_9hV7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies four interventions for long-horizon agents: continue, advise in the existing context, rebuild the context, and rebuild with the same advice. A 2x2 factorial design branches all four treatments from the same Craftax checkpoint, holding the executor, environment state, and within-repeat budgets fixed and sharing advice/handover artifacts across relevant arms. Across weak and strong executors, it reports 14,712 branches and mostly unresolved average contrasts. A weak-executor advice contrast is negative on the full data but does not exclude zero on the infrastructure-clean subset. The paper replaces an upward-biased in-sample maximum with a cross-fitted action-selection statistic and reports near-zero values, interpreting them as a bound on per-state intervention value.

Critical strengths
  1. Clear experimental design: Pairing 2x2 snapshots with shared handover artifacts and a frozen executor cleanly separates advice generation from context rebuilding. Introducing a common environment randomness source and a cost ledger further strengthens the experimental setup.
  2. Solid methodology and commendable transparency: The clean-subset sensitivity analysis and audit of allowance defects provide compelling methodological evidence. Notably, transparently disclosing unresolved effects and the loss of statistical significance after infrastructure filtering significantly improves interpretability.
Critical weaknesses
  1. Cross-fitting a single-repeat argmax estimates a noisy selector's value, which can be below the optimal state-dependent policy value; it is not an oracle upper bound (Appendix E.6). Its near-zero result therefore cannot support the policy-value ceiling claimed in Section 4 and Appendix F.3.
  2. The final 400-campaign/397-map dataset with mixed 2-6 repeats is not reconciled with earlier 100/99 descriptions or conflicting averaging/Holm procedures (Appendices E, G, I). Inconsistent confirmatory/exploratory labels also leave the final weighting, precision, and inferential status unclear.
Questions for authors
  1. What exact quantity is the claimed oracle ceiling intended to estimate? Please justify the upper-bound claim mathematically, or revise it to the held-out value of the specified noisy selector. Can the released data support an uncertainty-aware estimate/bound on max_u E[Y(u)|state,budget], including heterogeneous effects and the fact that remaining allowance is drawn per repeat? A corrected analysis could materially change the recommendation.

  2. Please provide one authoritative table of campaigns, maps, checkpoints, repeat counts and exclusions for each experiment, and reconcile the conflicting descriptions of averaging, weighting, multiplicity correction and pre-registration. Report the final analysis on the genuinely stalled subset separately from uniformly sampled opportunities, and identify which null conclusions survive the clean-data analysis under that exact procedure.

4 comments

Thank you for a review that reads like a referee report from a statistician; each point is correct, and each request is now in the paper.

W1/Q1. What the "oracle ceiling" estimates. It estimates the held-out value of a noisy selector, and the revision says so. Proposition 1 (Section 4.3, proof in Appendix C): for any selector û built from the other repeats of a state, E[Y_û,r − Y_C,r] = E[μ_û(s) − μ_C(s)] ≤ G*s ≤ G*, with G* = E{s,b}[max_u μ_u(s,b) − μ_C(s,b)] the oracle that knows the state and the remaining budget, and G*_s the oracle that knows the state only. It is a lower bound in expectation, so it cannot support the policy-value ceiling of the submission's Section 4 and Appendix F.3; both passages are rewritten (now Section 4.3 and Appendix K.3). We now select on all other repeats of a state rather than on one, which for two repeats is the frozen computation and for E2's six-repeat states uses all six.

Q2. An uncertainty-aware bound on max_u E[Y(u) | state, budget] with heterogeneous effects and per-repeat allowance. We give three, all for the budget-aware oracle G* (and, with the budget treated as noise, for G*_s):

  1. Assumption-free: the in-sample maximum over each (state, budget) cell's repeat means, which is upward-biased for G* cell by cell (Jensen; Proposition 2). Its one-sided 95% map-bootstrap limit is a conservative bound: 0.98 achievements (E1) and 1.52 (E2).
  2. Random effects: D = Δ(s,b) + ε for the repeat-level contrast vector (R−C, F−C, N−C). Repeats that share a state and an allowance identify the noise covariance Σ_ε (½E[(D_r − D_r′)(D_r − D_r′)ᵀ]); T = Cov(D) − Σ_ε is the covariance of the true effects across (state, budget) cells, i.e. the heterogeneity, by the method of moments with inverse-probability weights for which states have such pairs. With Gaussian marginals and any cross-arm dependence, G* ≤ Σ_u E[Δ_u⁺]; bootstrapping the whole pipeline gives 0.68 (E1; 15% of the Continue mean) and 0.87 (E2; 6%). Treating the allowance as noise (all within-state pairs) gives the bound on G*_s: 0.53 and 0.66.
  3. Distribution-free: Aven's inequality with the same means and variances, 0.93 and 1.26.

The per-arm heterogeneity is τ = 0.00/0.35/0.54 achievements in E1 (95% upper limits 0.57/0.69/0.82) against a pure repeat noise of about 1.5, and the allowance is not mere noise (in E1 the advice harm grows from −0.06 at 32 turns to −0.26 at 128; Table 8), which is why we lead with the budget-aware oracle. Assumption checks (Appendix C): repeats are independent by construction; the method of moments needs no Gaussian noise, which is integer-valued and heavy-tailed (Table 7); the Gaussian marginal of the true effects cannot be checked at two repeats, so the distribution-free bound is reported beside it; the permutation null confirms the estimates are calibrated (Table 6). The lower end of the bracket is the selector: +0.007 [−0.068, +0.083] (E1) and +0.003 [−0.111, +0.121] (E2), at the best single arm.

W2/Q3. One authoritative table. Table 1 (main text) gives, for each collection, the executor, map seeds, dates, predecessor campaigns and how they ended, campaigns that yielded no state, sampled and branched states (uniform/stalled draw), repeats per state, quadruples, branches, map clusters, the infrastructure-clean subset, and the pre-specified family with its source. Every quadruple passes six pairing checks (shared allowance, z shared by R/N, H0 by F/N, none for C, one environment stream, four model streams).

The 400/397 versus 100/99 discrepancy, and the mixed repeats (Appendix B). E2 was collected in two stages under one protocol: stage 1 (maps 1000–1099) branched 193 states at six repeats; stage 2 (maps 1100–1399) branched 572 states at two. The submission's "100 maps", "99 clusters" and "six repeats on 100 map seeds" described stage 1 alone and were not updated when stage 2 was added; its 400 campaigns, 765 states, 2,302 quadruples and 397 clusters are both stages. Three of the 400 E2 campaigns died before the first decision point at turn 16 and yielded no state, hence 397. In E1, 43 of 400 campaigns yielded no state (34 rate-limit errors at turn 0, 6 more before turn 16, 3 early deaths or no eligible opportunity), and 11 of 699 sampled states could not be branched because their stored context file was missing (logged), removing 3 more maps: 354. The submission's "0 of its 800 prefix campaigns" counted a third strong-executor collection on 400 fresh maps (E2R), which the revision reports separately for the first time.

Q4. Averaging, weighting, multiplicity and pre-registration. The submission described three averaging rules (none within a state; within a state with equal map weights, in the Holm artefact it cited; within a training example, for the manager) and applied no correction while printing one. The revision uses one rule throughout: repeats within a state, states within a map, then maps with equal weight; percentile intervals from 10,000 map-bootstrap resamples. The frozen quadruple-weighted estimates are printed beside every contrast (Table 10); no pre-registered verdict changes. Multiplicity: Holm within each experiment's pre-specified family; everything else exploratory. Labels from the frozen records (Table 4): E1's family was fixed with the plan on 2026-09-12 as {N−R, N−F, F−C} with Holm, not E2's {N−C, N−F, R−C}, which the pre-registration of 2026-09-23 fixed before any E2 branch. The submission applied E2's family to both experiments, called E1's N−R "exploratory" and F−C and N−R "secondary" (a label no record contains); all corrected. Under the correct labels E1's pre-specified N−R resolves (Holm p = 0.005), and R−C is exploratory in E1.

Q5. Genuinely stalled versus uniformly sampled states. Each predecessor run contributes one state drawn uniformly over its decision points and one drawn from those flagged as observably stalled. Every contrast (Table 9) and every bracket (Table 5) is now reported for the two draws separately. Stalled states carry more heterogeneity and wider upper bounds (E1: 1.03 against 0.57 achievements, budget-aware model bound), and the selector's lower bound is higher on them (+0.091 against −0.069 in E1), but against the best single arm the selector gains at most +0.034 in any stalled subset, and no interval excludes zero. In E1's clean stalled states the selector gains +0.131 [+0.011, +0.257] over Continue, all of it the main effect of N there; we report it as one of many subset analyses, not a finding.

Q6. Which nulls survive the clean-data analysis. On the 910 quadruples over 232 maps that descend from runs without an infrastructure error: E1's pre-specified N−R still resolves (+0.169 [+0.080, +0.259], Holm p = 0.002); the pre-specified N−F (+0.053 [−0.053, +0.157]) and F−C (+0.011 [−0.089, +0.114]) remain equivalent to zero at ±0.20 (Holm-adjusted TOST p = 0.005 and 0.002); N−C (+0.064 [−0.048, +0.178]) remains equivalent to zero; and the exploratory R−C (−0.106 [−0.211, +0.001]) no longer excludes zero. The bracket on the clean subset is [+0.036, 0.79] (budget-aware model bound). E2 and E2R had no infrastructure failures, so their analyses are the clean analyses.

Two further corrections you may want to check. (i) The fresh-map collection E2R gives an exploratory R−C of +0.161 [+0.010, +0.318], so we withdraw the submission's argument that the strong executor's R−C "shrank toward zero like an artefact" (Appendix M.2). E2R's one pre-registered utility prediction (F−C stays unresolved) holds; its pre-registered survival prediction failed (Appendix E). (ii) The executor's working window held two observations at every sampled state, so the keep/rebuild factor is weak by construction (Section 3.1, Table 14). Two exploratory intervals in Table 2 exclude zero under the one averaging rule (E2's F−C, E2R's N−R); Table 10 shows that E2's F−C does not under the frozen weighting, and we draw nothing from either.

Every number above is produced by released scripts that read the same branch records; re-running the frozen pre-registered script unchanged reproduces the submission's numbers exactly (Table 11).

Dear Reviewer 9hV7,

Thank you for your review. In case it is useful before the discussion closes on November 18: our response (Parts 1–3) answers each of your questions and reconciles every count you asked about (Appendix B), and we have uploaded a second revision with a design figure in the main text and plainer appendices; no result changed. We would be glad to clarify anything further.

Best regards, The authors

Reviewer_gdGeOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper studies how to intervene when an agent stalls. Holding the environment state and executor fixed, it compares continuing execution, adding advice, rebuilding the context, and rebuilding the context and then adding advice. The main contributions are separating the effects of advice and context rebuilding, running paired comparisons under two executor configurations, and reporting that most comparisons show no clear difference.

Critical strengths
  1. Starting every intervention from the same state and sharing the same advice and handoff materials helps reduce confounding in the comparisons.
  2. Separating action selection from effect evaluation avoids the optimistic bias of picking the best action from the same noisy results.
Critical weaknesses
  1. The interpretation of the core “gain ceiling” does not hold. Selecting an action from one trial run and evaluating it on another measures the gain of this noisy selection rule, not an upper bound on the gain of the optimal policy. This result is close to zero, which does not show that other state-dependent selection policies can achieve only small gains. The paper’s central inference therefore lacks support.
  2. The interpretation of null results is too strong. The paper treats the minimum detectable effect as a bound that rules out larger gains, but a study’s detection power is not a confidence upper bound on the effect. The current analysis does not support the claim that larger gains are absent.
Questions for authors
  1. Can you provide a valid upper bound on the optimal policy’s gain, or explicitly narrow the conclusion to the performance of the current selection rule?
  2. Can you provide valid confidence bounds or an equivalence analysis for the range of gains you actually aim to rule out?
4 comments

Thank you. Both weaknesses are correct, and the revision is built around them.

W1. The "gain ceiling" is not an upper bound. Agreed. Selecting on one repeat and scoring on another measures the value of a noisy selection rule. Proposition 1 now states it: for any selector built from the other repeats of a state, E[held-out gain] = E[μ_û(s) − μ_C(s)] ≤ G*s ≤ G*, where G* = E{s,b}[max_u μ_u(s,b) − μ_C(s,b)] is the gain of the oracle that knows the state s and the remaining budget b, and G*_s is the same for an oracle that knows only the state. The statistic is a lower bound in expectation; the word "ceiling" is withdrawn. We select on all other repeats of a state (for two repeats, exactly the frozen estimator).

For a valid upper bound we give two, each with a one-sided 95% limit from a bootstrap over maps:

  • Assumption-free (Proposition 2). U averages, over repeats, max_u Ȳ_u − Ȳ_C within the repeat's (state, budget) cell. By Jensen's inequality E[max_u Ȳ_u] ≥ max_u E[Ȳ_u] cell by cell, so E[U] ≥ G*, and an upper confidence limit for E[U] is a conservative one for G*: 0.98 achievements (weak executor) and 1.52 (strong).
  • Random-effects (Proposition 3). Writing the repeat-level contrasts D = Δ(s,b) + ε, replicate repeats that share a (state, budget) cell identify the noise covariance, so T = Cov(D) − Σ_ε identifies the variance τ_u² of the true effects across cells (method of moments). If each Δ_u is Gaussian, then under any dependence between arms, G* ≤ Σ_u E[Δ_u⁺] = Σ_u [δ_u Φ(δ_u/τ_u) + τ_u φ(δ_u/τ_u)]. Bootstrapping the whole pipeline gives 0.68 (weak; 15% of the Continue mean) and 0.87 (strong; 6%). A distribution-free version (Aven's inequality, needing only means and variances) gives 0.93 and 1.26. Appendix C checks each assumption.

The permutation null explains why the assumption-free bound is loose: under within-quadruple label shuffling, the in-sample statistic is as large as observed (0.92 vs 0.95, p = 0.92; 1.43 vs 1.46, p = 0.78). The conclusion is narrowed accordingly. The data exclude per-state gains above these bounds; they do not show that the gain is zero. What they show positively is that no feasible selector beats the best single arm: the cross-fitted selector's gain over the best fixed arm is −0.001 (weak), −0.017 (strong) and −0.020 (fresh-map replication), and simple learned policies do no better (Table 12).

W2. The MDE is not a confidence bound. Agreed: a minimum detectable effect describes a design under an assumed variance and says nothing about the data. Every such sentence is replaced by a 95% interval and two one-sided tests (TOST) at a declared margin (Section 4.2, Table 2). Margins: ±0.20 achievements, the effect size around which E2's pre-registered power table was built, and 5% of the Continue mean (0.23 achievements for the weak executor, 0.75 for the strong). Both were declared after seeing the data, so for every contrast we also report δ*, the smallest margin at which TOST declares equivalence at α = 0.05; a reader can apply any margin. Results:

  • Strong executor, pre-registered family. R−C = +0.020 [−0.112, +0.152], N−C = −0.083 [−0.211, +0.045], N−F = +0.084 [−0.030, +0.199]; δ* = 0.13, 0.19, 0.18; TOST p = 0.005, 0.038, 0.025 at ±0.20. The family is jointly within ±0.20 by the intersection-union test (p = 0.038); as three separate Holm-adjusted claims, p = 0.013, 0.049, 0.049.
  • Weak executor, pre-specified family {N−R, N−F, F−C}. N−R = +0.136 [+0.059, +0.214] differs from zero (Holm p = 0.005); N−F = −0.029 [−0.115, +0.059] and F−C = +0.008 [−0.073, +0.091] are equivalent to zero at ±0.20 (Holm-adjusted TOST p = 0.0003). The same holds on the infrastructure-clean subset (N−R = +0.169 [+0.080, +0.259]).
  • At 5% of the Continue mean, every contrast for the strong executor (E2, the fresh-map replication E2R, and both pooled) is equivalent to zero.

The power section is removed from the main text; the variance decomposition stays in Appendix L as a description of why the bracket is wide (single-repeat noise of 1.5–2.7 achievements against heterogeneity of 0.2–0.8).

Other changes relevant to your assessment. One averaging rule everywhere (repeats within a state, states within a map, then maps), with the frozen quadruple-weighted estimates printed beside it; no pre-registered verdict depends on the choice (Table 10). One multiplicity rule (Holm within each experiment's pre-specified family), with labels taken from the frozen records, which corrected E1's family. A previously unreported strong-executor collection on 400 fresh maps is analysed separately. And a design limit we now state plainly: the executor's working window held two observations, so the keep/rebuild factor is weak (Section 6).

Dear Reviewer gdGe,

Thank you again for your review. Your two points, that the cross-fitted "ceiling" is not an upper bound and that a minimum detectable effect is not a confidence bound, shaped the main changes of the revision, and Reviewer 2eA9 has since described these changes as meaningful improvements in methodological rigor. In brief:

W1. The cross-fitted statistic is now reported as what it is: the held-out value of a noisy selector, and hence a lower bound on the oracle's gain G* (Proposition 1). We add two valid upper bounds, an assumption-free one (Proposition 2) and a random-effects one that allows any dependence between arms (Proposition 3), each with a permutation calibration (Section 4.3, Table 3). The one-sided 95% upper bounds are 0.68 achievements for the weak executor and 0.87 for the strong.

W2. Every argument from a minimum detectable effect is replaced by 95% intervals and two one-sided tests at a declared margin, with δ*, the smallest margin the data certify, reported for every contrast, so that no verdict depends on our choice of margin (Section 4.2, Table 2).

We have also uploaded a second revision with a design figure in the main text and plainer appendices; no result changed.

If any part of W1 or W2 remains unresolved, we would be glad to address it before the discussion closes on November 18. If the revision resolves them, we would be grateful if you would consider updating your assessment.

Best regards, The authors

Thank you for the thorough and candid response. You have directly and rigorously addressed both of my main concerns. These revisions substantially improve the methodological rigor and clarity of the paper. I will update my soundness score and overall rating accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.