-
The deployment motivation exceeds the formal contract. Eq.(2) guarantees one action block from a prespecified initial-state law, while Section 8.1 states that review need not restore that law. What guarantee applies to the second and later blocks in the repeated review workflow motivating the paper?
-
Upper-coding does not by itself resolve admission-induced intervention. Unknown outcomes are conservatively coded as failures, but denied requests can change which actions occur and can depend on shared spending history. Is admission independent of latent regression outcomes, and do the conclusions persist with prospectively uncontaminated data or a dependence-robust analysis?
-
The adaptive mechanism is not empirically exercised. Section 7.2 reports that adaptive depth retains its initial depth on every observed path and is therefore identical to fixed depth. Please provide a setting in which adaptation changes a decision, or narrow the claimed contribution to fixed-depth selection.
-
The policy savings are retrospective rather than realized. Section 6.3 reads policy-requested prefixes from a common depth-six archive, whose physical collection costs 18.163–19.547B qualification tokens and 9.941–11.121B validation tokens (Table 10). How would the policies compare when executed prospectively with their actual stopping and shared-budget dynamics?
-
The shared-depth ranking is insufficiently robust. The objective uses a setting-specific H−λC with λ=.02/C2, one outer seed, and accounting intervals rather than sampling uncertainty. Please report sensitivity to λ, repeated seeds, and uncertainty in both horizons and costs.
-
The theoretical novelty remains unclear. Theorem 5.1 is an identification characterization, Theorem 5.2 uses the first unsafe coordinate, Theorem 5.3 applies a standard data-processing lower bound, and Proposition 5.4 follows from prefix consistency. What new statistical or algorithmic principle is contributed beyond ordered risk testing and time-uniform sequential inference [1,2]?
-
The sequential guarantee appears inapplicable to the collected archive. Section 4.2 and Appendix B.3 require prespecified conditional laws and conditional independence within a balanced round, yet Appendix D.7 reports admission failures in 155/540 qualification roots and 275/432 validation roots, and Section 8.1 states that the shared mechanism can couple trajectories. Please identify the result that covers this process or restrict the empirical guarantee accordingly.
-
The operational meaning of "safe before review" is unclear. The experiments use α=.40, and the event is regression under specified external tests; Section 8.1 notes that this excludes upgrade failure and untested harms. Why is this risk level and event definition appropriate for the claimed review decision rather than only for benchmark non-regression?
[1] Angelopoulos, A. N., Bates, S., Candès, E. J., Jordan, M. I., and Lei, L. (2021). Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control. arXiv:2110.01052.
[2] Howard, S. R., Ramdas, A., McAuliffe, J., and Sekhon, J. (2021). Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics, 49(2), 1055–1080.
We thank the reviewer for the constructive feedback for our presentation. We agree that the introduction and Figure 1 should better explain the motivation and evaluation workflow. We will expand the introduction with a concrete example of a coding agent performing consecutive repository upgrades, illustrating why independently resetting the environment after each action cannot reveal regression risk accumulated across inherited repository states. We will also explicitly introduce and explain Figure 1 in the main text. The figure contrasts three evaluation strategies: independent clean‑reset trials, full persistent trajectories, and shorter persistent prefixes. The key distinction is that persistent prefixes preserve state within each trajectory while allowing more independent roots to be evaluated under a fixed resource budget. Regarding experimental scope, we agree that broader evaluation would strengthen the generality of the conclusions.