- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes VALVE, a validation-gated method for long-horizon test-time evolution of repository-level code agents without weight updates.
Critical strengths
The unusually long evaluation horizon is a meaningful empirical contribution. The study follows roughly one thousand task arrivals per repository, reports both final and peak performance, and shows that ungated memory accumulation can collapse after an early peak. VALVE improves all twelve reported model–repository cells and substantially reduces both drawdown and bank size.
Critical weaknesses
The title and framing overstate what is certified in the realized experiments. The guarantees concern same-distribution expected task success under i.i.d. sampling, disjointness, a prespecified checkpoint family, and valid variance upper bounds. They do not certify software correctness, security, robustness to repository drift, or monotonic population improvement. More importantly, the 50-task validation panel is smaller than every per-update sample size shown in Table 4 for even a 10-point population-loss tolerance. The paper does not report the actual run-specific lower bounds, so the operational certificate may be weak or vacuous.
Questions for authors
-
What are the actual Theorem 3 lower bounds for every run at n = 50, using the realized gate-query count, accepted-update count, measured margin, and chosen confidence level? Are any nontrivial?
-
How can the prospective certificate use a gate horizon of 60 when observed runs reach 85 gate queries?
-
Can the empirical variance estimates be replaced with independently justified high-probability upper bounds?
4 comments
Part 2/2: Theorem 3 bounds (W1/Q1), query horizon T (Q2), and variance bound (Q3)
W1/Q1. What are the actual Theorem 3 lower bounds for every run at n = 50? Are any non-trivial?
We computed them for every accepted update in all 12 runs, using the realized acceptance index , the measured margin, and :
Table R7: Certified per-update population loss at (empirical variance, as in the paper)
| Backbone | Accepts per run | , 1st accept | Runs with all accepts at pp | Share of accepted gain with pp | Non-vacuous |
|---|---|---|---|---|---|
| GPT-5.5 | 1 / 1 / 15 / 1 | 7.0–15.4 pp | 3 / 4 | 55% | 18 / 18 |
| Claude-4.6 | 2 / 7 / 6 / 5 | 6.3–11.8 pp | 1 / 4 | 56% | 20 / 20 |
| MiniMax-M2.7 | 5 / 15 / 14 / 14 | 14.2–15.2 pp | 0 / 4 | 11% | 48 / 48 |
Every accepted update has a non-trivial bound, and the bounds are tight where it matters. The first update in each run is controlled to within 6–15 pp. For GPT-5.5 and Claude-4.6, every update is within 15 pp in 4 of the 8 runs, and updates with pp carry over half of the accumulated panel gain. Bounds widen for many small late updates, because the union over accept paths grows like , which is the price of allowing proposals to react to earlier verdicts. With the variance-assumption-free bound of Q3, the first-update bound widens to 37–46 pp.
The pattern across backbones also follows the theory. MiniMax-M2.7, whose panel outcomes are about twice as variable, makes the most accepts (12 per run) with the smallest margins (median 3.3 points), so few of its updates are tightly bounded, which is exactly the case where the sizing rule calls for a larger panel.
We view this as the analysis working as intended: Thm 3 guards each update against large losses, while the improvement guarantee behind the reported results is placed on the accumulated gain (Table R5), where it is non-trivial even without a variance assumption. We will add Table R7 to the appendix and describe the two theorems in these roles.
Q2. How can the prospective certificate use T = 60 when observed runs reach 85 gate queries?
Thank you for this careful check. In Thm 3, is an a-priori upper bound on the number of gate queries, not a typical value. The in Table 4 was an illustrative choice near the average number of queries per run, and pairing it with the observed runs was an oversight on our part, since some runs exceed it. The correct horizon follows from the protocol itself: the gate is queried at most once per mixed-outcome task, and each run stops after 100 such tasks (Sec. 4.1), so is fixed before any run starts. It covers all 12 runs (15–97 queries; see our response to fyRY). Because enters the bound only through , the change is mild: the 10-point / panel size in Table 4 goes from 136 to 155. All numbers in Table R7 already use , and we will update Table 4 and App. A.9 accordingly.
Q3. Can the empirical variance estimates be replaced with independently justified high-probability upper bounds?
Yes, and we have now done so. With , , and has a closed-form upper confidence bound from the lower-tail Bernstein inequality for (formula in our response to fyRY). It uses only the sample and Assumption 1, at the cost of a split. The resulting certified final gains are ≥ +11.5 (GPT-5.5) and ≥ +12.3 (Claude-4.6) at 95% (Table R5). For reference, the trivially valid would give a 15.5-point radius at , about twice the data-driven one. We will also use this bound in the sizing rule, so the recommended holdout sizes are prospectively valid.
Dear Reviewer,
Thank you so much for your thoughtful initial reviews!
With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.
Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!
Warm regards
Thanks for the detailed rebuttal. I maintain my score to support this paper.
Part 1/2: Scope of certification and validation-panel size (W1)
We thank the reviewer for the positive assessment of the long-horizon study and for the precise questions about what the guarantees mean in practice. Below we clarify the role of the theory in the paper and give the run-specific numbers.
Thank you for this point. The guarantees address a question we consider central for self-evolving agents: whether improvements carry over to the future in-distribution tasks the agent will face, beyond the panel used to accept them, and how much held-out data is needed to establish this. That is what "certified" refers to.