acceptodds
Under review as a conference paper at ICLR 2027

SEED: Separating Admission from Deployment Evidence in Adaptive Coding-Agent Updates

Abstract

Adaptive coding agents select prompt and workflow updates using evaluator feedback. Once a hidden split helps choose the surviving update, however, it no longer supplies independent deployment evidence. We introduce SEED, a four-stage protocol that separates visible ranking, hidden admission, disjoint canary confirmation, and deployment or rollback. Partitions are fixed within each run, not resampled across generations. The analysis spans 24 model lines, six benchmark families, two serving stacks, and all 432 model–benchmark–profile cells, while distinguishing retry-inclusive gate exposures from distinct incumbent–candidate decisions. Canary checks reject 25.1%, 25.3%, and 33.6% of first admissions in the Anchor, Horizon, and Confirmatory profiles. A seemingly large retry-inclusive Anchor–Horizon increase of percentage points disappears after keeping only the first exact pair per run ( points, 95% interval ), so the raw contrast measures operational gate burden rather than increased distinct-decision non-transfer. Among 652 first-exact rollback decisions, 271 prevent a local untouched-final pass-or-gap regression and 159 prevent direct pass loss, but recall is about 33% and fixed-budget endpoint intervals all cross zero. We provide a bounded, retry-aware deployment-trace protocol and an empirical account of its behavior. The results do not show that a canary is an oracle, identify an optimal budget split, or establish contamination-free or non-coding transfer.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
large language models, coding agents, adaptive evaluation, prompt optimization, deployment validation, canary testing, rollback, retry-aware analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_p9yBOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SEED: a canary-based agent evaluation pipeline for selecting and deploying coding-agent configurations. SEED uses a disjoint canary check to separate admission from deployment decisions, while reserving a final split for reporting. The experiments study a fixed library of six coding-agent configurations across 24 model lines and six benchmark families using a fixed default gating rule. The paper reports that the canary rejects 25–34% of initially admitted candidates. Some rejections prevent a lower score on the final test set (159 of 652 distinct rejections, or 24.4%), and the experiments do not establish an overall performance improvement: the fixed-budget comparisons estimate gains of 0.29–0.57 percentage points over hidden-only evaluation, with all confidence intervals including zero (Tables 7 and 9).

Critical strengths

N/A

Critical weaknesses

The gating mechanism and search policy are insufficiently justified and evaluated. The scoring weights and gate thresholds lack adequate justification, and a sensitivity analysis is missing (§3.1; Appendix A.1). Equation 1’s gap penalty is also redundant under the hard gap gate, contradicting its stated role in selection. The search analysis further lacks a control that prevents redundant returns to earlier incumbents, which neither exact-pair filtering nor failed-pair blacklisting fully removes (§5.3). The experiments do not establish a methodological or empirical contribution. The experiments do not demonstrate what SEED adds beyond established practices for separating selection and evaluation, canary checks, and rollback. The matched-budget comparisons do not establish improved final performance (Table 9). Furthermore, the authors make no clear claim that these results provide insight into agent evaluation beyond the specific pipeline studied. The presentation obscures the claims. The introduction reports findings before explaining “Anchor”, “Horizon”, and “Confirmatory”, while extensive terminology and accounting obscure how the experiments support the claims. The reporting recommendations (§6, lines 473–476) read as an underdeveloped checklist, and do not connect back to the evidence in the paper.

Questions for authors

Gate and search validation: clarification and further analysis. Please clarify the intended role of Equation 1’s gap penalty under the hard gap constraint. Furthermore, could the author’s provide sensitivity analyses for the active scoring weights and gate thresholds, and repeat the main analyses with a search policy that excludes known failures and redundant returns to earlier incumbents? Further experiments. Can the authors provide evidence that SEED improves final selection over standard held-out selection and no-rollback controls under matched evaluation budgets, using realistic candidate searches and substantially larger evaluation splits?

1 comment
AuthorsOpenReview ↗

Gate and search validation, contribution and final selection, presentation

Thank you. Pages and lines refer to the revised 43-page PDF. 95% intervals resample model lines, or tasks in H.6; pp = percentage points.

W1 / Q1. Gate and search validation

Eq. (1) gap penalty. It has no role, as you say (the hard gap gate zeroes it); Eq. (1) now omits it (p. 3).

Weights and thresholds. They are unvalidated implementation choices, and the records argue against one of them, the gap rule at the canary (Appendix A.1, p. 22). On 300 cache-verified runs replayed end to end (post hoc; Appendix N, pp. 41–42), 20/80, 50/50 and 60/40 weights change final passes by −3, 0 and −4 of 2,200. Of ten one-task threshold shifts, only the two stricter gap margins change final passes detectably, both downward (−3.09 pp [−6.85, −0.18] at admission, −4.13 [−8.64, −0.91] at the canary); the other eight intervals include zero. A pass-only canary chosen post hoc gave +0.82 pp [+0.18, +1.64]. On new DeepSeek and Qwen MBPP+ matrices with pre-set rules (Appendix H.6, pp. 33–35), it gives −0.00 pp [−0.57, +0.13] and −0.11 [−0.50, +0.15] at 8/8, no weight setting moves final passes over 0.62 pp, and 1 and 5 of 40 one-task shifts have intervals excluding zero, all upward (Qwen's at Anchor 8/8, incl. both stricter gap margins).

Final outcomes (Section 5.4, p. 8). Of 652 first-exact-pair rollbacks with final evidence, 159 prevented a final-pass loss and 202 blocked a gain. By trigger (post hoc), the canary pass rule prevented 125 and blocked 107; the gap rule alone prevented 34 and blocked 95, rejecting candidates 6.73 pp above the incumbent on average [4.30, 9.11].

Search without failures or returns. Partly. A replay excluding failed pairs and former incumbents (2 of 6 families, 10 of 24 lines; Appendix N, p. 41) completes 276 of 300 runs. Final pass rate changes by +0.35 pp (bound over 300 [−4.91, +3.09]). On 77 matched pairs Horizon−Anchor disagreement is +6.18 pp [+0.50, +11.71]; the 23 missing Horizon runs were rollback-heavy: a sensitivity check (all-pair bound [−11.2, +30.5]). Its 104 rollbacks with final evidence prevented 31 losses and blocked 34 gains (gap rule alone: 2 and 12). The failed-pair blacklist replay gives +1.75 pp [−1.27, +6.14] (Table 5).

W2 / Q2. Contribution and final selection

Contribution. A retry-aware trace and counting rules that separate first checks from repeats (p. 1; Section 2, p. 3).

Final selection. No (table). Hidden-only uses half the held-out tasks. With 16 held-out tasks and equal search limits (row 2), SEED-FULL trails hidden-only in 6 of 8 conditions.

Final-task passesSEED-FULLNo Rollbackhidden-only
Core panel, 2,160 runs (p. 8)9,387 of 15,8409,4659,409
12 workflows, 2 models (Table 24, p. 43)712 of 1,280674728
DeepSeek LiveCodeBench, 5 seeds (p. 9)7 of 301010

Core panel: SEED-FULL minus hidden-only −0.14 pp [−0.97, +0.67]; minus No Rollback, −0.14 to −0.97 by profile (Horizon [−1.53, −0.42]). Resampling the 30 shared task partitions gives [−2.36, +0.05] for Horizon: no gain and possibly a loss. Static public-test-repair (REFLECTION-ONLY) beat SEED-FULL by 2.18 pp [0.72, 3.50] in Anchor and 3.40 [1.67, 5.08] in Confirmatory, and both controls too, at about 9% of the calls.

We ran no realistic search. For larger splits, MBPP+ matrices on two hosted models (378 tasks; DeepSeek and Qwen3.7-plus, families in the core panel) replay 200 split seeds; their shared pre-set power gate (simulated on archived lines) failed, half-widths 1.35/1.47 pp, and was amended before data; Anchor misses the new one. At 32+32 against 64-task hidden-only, Anchor reads "not a small-split effect" (pre-set rule) on both lines (SEED-FULL −0.20 pp [−0.85, +0.28]; +0.17 [−0.47, +0.60]); Horizon is "inconclusive" for DeepSeek (−0.39 [−2.64, +1.07]) and "not a small-split effect" for Qwen (−1.14 [−2.98, +0.50]). SEED-FULL keeps the base workflow in 198/182 (DeepSeek) and 197/160 (Qwen) seeds. The 12-workflow panel has 32 untouched final tasks but 16 held-out ones, a fixed library and one generation at 192 calls; never updating passes 720 of 1,280. All 16 paired intervals include zero; none met the prespecified cost-and-quality criterion (Appendix O, p. 43). On LiveCodeBench DeepSeek is one comparison (profiles identical; 211 of 504 responses capped, not SEED-FULL's on the 3 tasks it lost); re-sent a day later, only 29 of 100 requests matched (13 flipped pass/fail): no evidence of a gain (Appendix H.5, pp. 32–33).

Beyond this pipeline. We tested no other pipeline, so we claim only Section 6's reporting rule (p. 9, lines 473–479), each part now tied to a result here.

W3. Presentation

The abstract and introduction (p. 1, lines 050–053) now define the three profiles before any finding. Terminology is not reduced, but Section 3.1 (p. 3) now says SINGLE EVALUATOR, hidden-only, HO and H are one policy.

Reviewer_ZncwOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

They propose a new protocol to evaluate adaptive coding agents. The study covers 24 models and six coding benchmarks.

Critical strengths

Their study is quite comprehensive - spanning across 24 models and six coding benchmarks.

Critical weaknesses

In general, I found this paper extremely hard to read / follow. I am not able to decipher what is the motivation of this work - is this just an analysis work? How does this framework help over other existing protocols? The Figure 1 has 4 plots which does not make any sense to me. The 4th subplot is also not formatted properly.

Questions for authors

None

3 comments
AuthorsOpenReview ↗

Readability, motivation, value over existing protocols, Figure 1

Thank you. Pages refer to the revised PDF (43 pages); changed text is in blue. Intervals are 95% and resample the 24 model lines unless stated.

Readability. We rewrote the abstract and introduction (pp. 1–2). Both now present it as an analysis paper and define the three search profiles before any result; the introduction follows one example (workflow A, candidate B).

Motivation: is this just an analysis work? Yes, it is an analysis paper. Once a held-out set has helped choose a coding agent's update, it is no longer independent evidence about that update (abstract, p. 1). We measure what a second, separate check adds before the update replaces the current workflow. SEED is a four-stage update procedure assembled from existing controls (abstract, p. 1; Related Work, p. 3).

How does it help over existing protocols? It does not give a better final workflow than the usual practice of choosing the update on one held-out set and reporting one final score (hidden-only selection, p. 1). We ran no other published protocol; VeRO and HarnessOpt-Bench are only discussed (p. 3). What it adds is a record (Table 3, p. 5) that keeps apart which tasks chose an update, which confirmed it and how often the same pair was checked again.

Setting (Table 1, p. 3; Section 4, pp. 5–6). The current workflow (incumbent) is A; the agent proposes B, adding a repair step. Visible tasks rank candidates; a holdout set, which helped choose B, admits it; a separate canary set decides whether B is deployed or A is kept (a rollback); final tasks only score the result. Anchor: 4 holdout and 4 canary tasks, 2 generations of 3 proposals; Horizon: same tasks, 4 generations of 5; Confirmatory: 8 and 8 tasks, 2 of 3. SEED-FULL keeps A when the canary fails; No Rollback scores the canary but deploys anyway; hidden-only (SINGLE EVALUATOR, p. 3) has no canary: the holdout ranks and admits, so it uses half the held-out tasks.

Final-task passesSEED-FULLNo Rollbackhidden-only
Core panel, 2,160 runs (24 model lines × 6 benchmarks × 3 profiles × 5 seeds), of 15,840 (p. 8)9,3879,4659,409
12 workflows, 2 models, 2 search limits, 5 seeds, of 1,280 (Table 24, p. 43); never updating passes 720712674728
DeepSeek Flash, 5 LiveCodeBench seeds, of 30 (Anchor; Horizon identical; one draw: re-sent a day later, only 29 of 100 requests returned the same text, so it gives no evidence of a gain; pp. 9, 33)71010

Core panel (p. 8): SEED-FULL minus hidden-only −0.14 percentage points (pp) [−0.97, +0.67]; minus No Rollback −0.49 pp [−0.98, −0.04] pooled, largest in Horizon (−0.97 [−1.53, −0.42]). Resampling the 30 task partitions that all lines share gives [−2.36, +0.05] for Horizon. We claim no gain and a possible loss. In the 12-workflow and five-task-budget checks all intervals include zero (p. 43; Table 9, p. 9). A fixed public-test-repair workflow (REFLECTION-ONLY, no search) scored 2.18 pp [0.72, 3.50] above SEED-FULL in Anchor and 3.40 pp [1.67, 5.08] in Confirmatory (no Horizon arm), also above the other two policies, at about 9% of the calls (p. 8).

What the record shows.

  1. The second check often disagrees with the first: the canary rejected the first admitted candidate in 145 of the 577 Anchor runs with an admission (25.1%), 169 of 667 in Horizon (25.3%) and 177 of 527 in Confirmatory (33.6%) (p. 6).

  2. Most rejections are repeats. Proposals depend only on the incumbent, so after a rollback the agent proposes B again (p. 4). In all 691 SEED-FULL runs with a rollback, every later check was that pair, rejected again, and the run ended on A; 981 of the 1,672 rejections are such repeats (p. 6). A repeat reuses stored canary scores and adds no model calls (150 of 150 checked; Appendix C.2, p. 26). Counting every check, Horizon's rejection rate is 12.26 pp above Anchor's [9.64, 15.04]; counting the first check of each exact pair, −0.73 pp [−2.37, +0.91] (Table 5, p. 7).

  3. More rollbacks blocked a gain than prevented a loss. Of the 652 first-exact-pair rollbacks with final evidence, 159 rejected a candidate with fewer final passes than the incumbent, 202 one with more and 291 a tie (Section 5.4, p. 8). The canary rejected 33.3% of the 478 candidates that would have lost passes and 23.8% of the 850 that would have gained (difference 9.5 pp [3.9, 15.2], p. 29), but the latter were more common.

In short, the 12.26 pp rise comes from re-checking the same pair, and rollback did not improve the final workflow on the core panel.

Figure 1 (p. 2) now has two panels, not four: (a) the change in item 2; (b) the former fourth panel, redrawn (symmetric axis, legend below): SEED-FULL against hidden-only and No Rollback at the five-task budget.

Reviewer_ZncwOpenReview ↗

Replying to the Rebuttal

I have acknowledged the rebuttal. I still find it to be extremely difficult to follow. I will maintain my rating.

AuthorsOpenReview ↗

What happened when the same update was rejected four times

Thank you for telling us the first reply was hard to follow. In one HumanEval+ run, a coding agent proposed adding a step to plan for edge cases. One set of problems had already helped select this change. Four separate problems determined whether to keep it. The old workflow solved three and the proposed one solved two, so the old workflow stayed. The same change came up three more times. Each time, the check reused the original results.

On six problems held back for the final test, the proposed workflow solved four and the old one solved two. At this decision, the check blocked a useful change. Across the main study, 652 rejected pairs had final scores for both workflows, counting each pair once per run. Keeping the old workflow prevented a loss in 159 cases, blocked an improvement in 202, and made no difference in 291 (Table 7 and accompanying text, p. 8). All four rejections in the example rested on the same evidence. We appreciate your patience with the earlier explanation.

Reviewer_ngMcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

SEED studies how to distinguish evidence used to select an adaptive coding-agent update from evidence supporting its deployment. The protocol separates visible candidate ranking, hidden admission, canary confirmation and rollback, and untouched final reporting. Its main empirical analysis compares retry-inclusive gate events with the first appearance of each exact incumbent-candidate pair within a realized run. The paper also measures local rollback protection, fixed-budget final outcomes, and sensitivity to repartitioning, failed-pair blacklisting, and a fresh task distribution. The contribution is a retry-aware trace schema and empirical study of adaptive evaluation, rather than a new canary mechanism or an adaptive-valid statistical guarantee.

Critical strengths
  1. The counting-unit analysis identifies a substantial and practically relevant change in interpretation. The Anchor–Horizon disagreement contrast is +12.26 percentage points for retry-inclusive events, with an interval of [9.64, 15.04], but -0.73 points for first exact pairs, with an interval of [-2.37, .91]. Repeated encounters with the same pair can therefore make a gate-workload statistic look like evidence about a different population of update decisions. The paper makes the distinction explicit and preserves the realized trajectory when defining its first-pair analysis. This is useful for agent self-improvement studies in which rejected candidates can recur after rollback.

  2. The evaluation distinguishes local protection from overall improvement and reports the uncertainty honestly. Among 652 first-exact rollbacks, 271 prevent a local untouched-final pass-or-gap regression and 159 prevent a direct pass loss. The paper does not turn those local cases into a claim of guaranteed endpoint improvement: the fixed-budget comparisons retain intervals across zero. Conditional repartitioning, trajectory-changing blacklist replay, and the Fresh-LiveCodeBench extension also provide more informative checks than a four-way split protocol alone. The breadth of the main panel, covering 24 model lines and six benchmark families, adds value to this measurement effort.

Critical weaknesses
  1. The numerical findings have limited demonstrated transfer beyond the evaluated search design. The main study uses six bounded prompt/workflow configurations, and Anchor and Horizon change proposal breadth as well as search depth. The authors explicitly acknowledge that the resulting contrasts are descriptive rather than one-factor causal estimates. The reporting distinction between repeated exposures and first exact pairs is useful beyond these runs, but the numerical rates remain tied to this search regime. The Fresh-LiveCodeBench extension covers four hosted Qwen model lines and gives a first-exact contrast of +8.47 percentage points with an interval of [-8.97, 21.67]. It is too imprecise to establish either a positive distinct-pair contrast or a meaningful difference from the core estimate. Separately, the trajectory-changing blacklist replay has 275 completed Anchor-Horizon run pairs out of a 300-pair target. Its full-target bounds cross zero for rollback probability per admission and the probability of any rollback per run. The positive bound for rollbacks per run concerns a different estimand and can partly reflect the greater number of opportunities in Horizon. These are carefully disclosed limitations, but they leave the quantitative reach of the measurement contribution narrow.

  2. The measured local protection does not yet identify a preferred final-quality/resource trade-off. For the 4/1 allocation, Table 9 reports an endpoint difference of +.57 percentage points [-.81, 1.89] relative to hidden-only evaluation and +.34 [-.38, 1.12] relative to allocation-matched no rollback. The other evaluated allocations also have endpoint intervals across zero. This supports uncertainty about the preferred policy, rather than either equivalence or an absence of benefit. Table 8 does account for realized computation: moving from 4/1 to 2/3 reduces mean calls from 87.7 to 78.4 and tokens from 105.8k to 93.9k while increasing protective rollbacks. The changing execution path makes this a comparison of complete policies, rather than an isolated effect of canary task count. The useful operational result is therefore a measured set of trade-offs, not a universal allocation rule. My reservation is whether these bounded rates and trade-offs provide enough additional guidance to justify the empirical contribution at this venue; it is not an allegation that the authors claim a causal depth effect or a proven deployment gain.

The trace schema, counting-unit distinction, and explicit null results are substantive strengths. My remaining concern is the scope and practical significance of the evidence, rather than a technical flaw established by these limitations.

Questions for authors

None. Section 6 and Appendix K already state the reporting implications; Section 5.3, Table 6, and Appendix E distinguish the observed blacklist replay from its full-target bounds; Section 5.5 states the narrower allocation conclusion. My remaining reservations concern the scope and significance of those findings rather than a factual clarification.

1 comment
AuthorsOpenReview ↗

Scope and transfer, depth versus width, final quality and cost

Thank you. Pages refer to the revised PDF (changes in blue). Intervals are 95%, resampling model lines unless stated; pp = percentage points.

W1. Scope and transfer

These limits stand, except that depth and width can be separated.

Depth vs width (Section 5.1, p. 6; Table 5, p. 7). Horizon's first two generations are a five-proposal search on Anchor's task splits: against Anchor they differ only in width, against full Horizon only in depth. No 4-generation, 3-proposal runs exist.

  • All checks: width +1.34 pp [−0.57, +3.22], depth +10.92 [+9.37, +12.55].
  • First exact pairs: width +1.23 [−0.08, +2.53], depth −1.96 [−2.97, −0.84].

Mechanism (Section 5.1, p. 6; Appendix C.2, p. 26). Proposals depend only on the incumbent, so a rollback freezes the search: in all 691 SEED-FULL runs with a rollback, every later check repeated the rolled-back pair and was rolled back again. So 981 of the 1,672 retry-inclusive rollbacks are repeats, which reuse stored canary scores and add no model calls (150 of 150 checked). The 194 first exact pairs that depth adds come from runs without a prior rollback; 27 (13.9%) were rolled back, against 25.2% in generations 1–2.

Other limits. No other search space was tested for disagreement. The failed-pair blacklist replay you discuss is unchanged (+1.75 pp [−1.27, +6.14], 275 pairs; Section 5.3, p. 7). A post hoc replay that also excludes former incumbents (2 of 6 families, 10 of 24 lines; Appendix N, p. 41) completes 276 of 300 runs; on 77 matched pairs Horizon−Anchor is +6.18 pp [+0.50, +11.71] (originally +15.99 raw, +1.66 first exact), but missing Horizon runs were rollback-heavy, so over all pairs it is only bounded, [−11.2, +30.5].

W2. Final quality and cost

No search policy is preferred: on the core panel, rollback did not improve the final workflow in any profile and a static workflow beat all three in its core-panel runs; the smaller checks are mixed.

Final-task passesSEED-FULLNo Rollbackhidden-only
Core panel, 2,160 runs (Section 5.4, p. 8), of 15,8409,3879,4659,409
Table 20 (p. 32), 1 model, 4 seeds, of 24: HumanEval+; MBPP+13; 1712; 1414; 15
12 workflows, 2 models, 5 seeds, 2 limits (Table 24, p. 43), of 1,280712674728
DeepSeek Flash, 5 LiveCodeBench seeds, of 30 (Anchor; Horizon identical)71010

Core panel, SEED-FULL minus hidden-only: −0.14 pp [−0.97, +0.67]; minus No Rollback: Horizon −0.97 [−1.53, −0.42], pooled −0.49 [−0.98, −0.04]. Resampling the shared task partitions gives [−2.36, +0.05] (Horizon) and [−1.17, +0.09] (pooled; Appendix C.1, p. 26), so we report no gain and a possible loss. With 12 workflows, none of 16 comparisons met Appendix O's cost-and-quality criterion and never updating passes 720 of 1,280.

Local protection (Section 5.4, p. 8). Of the 652 first-exact-pair rollbacks with final evidence, 159 rejected a candidate with fewer final passes than the incumbent, 202 one with more and 291 a tie. By trigger (post hoc), the 429 after a canary pass drop prevented 125 losses and blocked 107 gains; the 223 from the gap rule alone prevented 34 and blocked 95, rejecting candidates +6.73 pp [4.30, 9.11] above the incumbent on average. All four DeepSeek first-exact rollbacks were of this kind (candidates with one or two more final passes). 211 of 504 responses hit the 2,600-token cap; SEED-FULL's answers on the three tasks it lost to No Rollback did not, though the cap may have shaped its search. Re-sent unchanged a day later (temperature 0), only 29 of 100 DeepSeek requests gave the same text and 13 flipped pass/fail, so by a pre-set rule the 8,192-token re-send was not run; we read its totals only as no evidence of a gain (Appendix H.5, pp. 32–33). Correction: our 271 pass-or-gap protective rollbacks you cite are 261 of 652; ten were floating-point artefacts, all rejecting a candidate with more final passes (Appendix F, p. 29).

Cost (Appendix I and Table 22, pp. 35–36). On the core panel the logs charge hidden-only, which uses half the held-out tasks, for unused canary scoring (at least 36,683 calls). Without it, hidden-only used at most 255,118 calls and SEED-FULL at least 2.4% more. The static public-test-repair workflow (REFLECTION-ONLY) beat SEED-FULL by 2.18 pp [0.72, 3.50] in Anchor and 3.40 pp [1.67, 5.08] in Confirmatory, and the other two policies too, at about 9% of the calls (p. 8).

What follows. A reporting rule; each part changes a conclusion here. Count each exact pair once: where proposals depend only on the incumbent (greedy search over a fixed library), repeats dominate and Horizon−Anchor falls from +12.26 pp to −0.73 (over task partitions [+9.07, +15.25] and [−2.89, +1.35]; Appendix L, p. 38). Report blocked gains beside prevented losses and the rule behind each rollback, and each policy's final pass rate beside a static or no-update baseline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.