- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes SEED: a canary-based agent evaluation pipeline for selecting and deploying coding-agent configurations. SEED uses a disjoint canary check to separate admission from deployment decisions, while reserving a final split for reporting. The experiments study a fixed library of six coding-agent configurations across 24 model lines and six benchmark families using a fixed default gating rule. The paper reports that the canary rejects 25–34% of initially admitted candidates. Some rejections prevent a lower score on the final test set (159 of 652 distinct rejections, or 24.4%), and the experiments do not establish an overall performance improvement: the fixed-budget comparisons estimate gains of 0.29–0.57 percentage points over hidden-only evaluation, with all confidence intervals including zero (Tables 7 and 9).
Critical strengths
N/A
Critical weaknesses
The gating mechanism and search policy are insufficiently justified and evaluated. The scoring weights and gate thresholds lack adequate justification, and a sensitivity analysis is missing (§3.1; Appendix A.1). Equation 1’s gap penalty is also redundant under the hard gap gate, contradicting its stated role in selection. The search analysis further lacks a control that prevents redundant returns to earlier incumbents, which neither exact-pair filtering nor failed-pair blacklisting fully removes (§5.3). The experiments do not establish a methodological or empirical contribution. The experiments do not demonstrate what SEED adds beyond established practices for separating selection and evaluation, canary checks, and rollback. The matched-budget comparisons do not establish improved final performance (Table 9). Furthermore, the authors make no clear claim that these results provide insight into agent evaluation beyond the specific pipeline studied. The presentation obscures the claims. The introduction reports findings before explaining “Anchor”, “Horizon”, and “Confirmatory”, while extensive terminology and accounting obscure how the experiments support the claims. The reporting recommendations (§6, lines 473–476) read as an underdeveloped checklist, and do not connect back to the evidence in the paper.
Questions for authors
Gate and search validation: clarification and further analysis. Please clarify the intended role of Equation 1’s gap penalty under the hard gap constraint. Furthermore, could the author’s provide sensitivity analyses for the active scoring weights and gate thresholds, and repeat the main analyses with a search policy that excludes known failures and redundant returns to earlier incumbents? Further experiments. Can the authors provide evidence that SEED improves final selection over standard held-out selection and no-rollback controls under matched evaluation budgets, using realistic candidate searches and substantially larger evaluation splits?
Gate and search validation, contribution and final selection, presentation
Thank you. Pages and lines refer to the revised 43-page PDF. 95% intervals resample model lines, or tasks in H.6; pp = percentage points.
W1 / Q1. Gate and search validation
Eq. (1) gap penalty. It has no role, as you say (the hard gap gate zeroes it); Eq. (1) now omits it (p. 3).
Weights and thresholds. They are unvalidated implementation choices, and the records argue against one of them, the gap rule at the canary (Appendix A.1, p. 22). On 300 cache-verified runs replayed end to end (post hoc; Appendix N, pp. 41–42), 20/80, 50/50 and 60/40 weights change final passes by −3, 0 and −4 of 2,200. Of ten one-task threshold shifts, only the two stricter gap margins change final passes detectably, both downward (−3.09 pp [−6.85, −0.18] at admission, −4.13 [−8.64, −0.91] at the canary); the other eight intervals include zero. A pass-only canary chosen post hoc gave +0.82 pp [+0.18, +1.64]. On new DeepSeek and Qwen MBPP+ matrices with pre-set rules (Appendix H.6, pp. 33–35), it gives −0.00 pp [−0.57, +0.13] and −0.11 [−0.50, +0.15] at 8/8, no weight setting moves final passes over 0.62 pp, and 1 and 5 of 40 one-task shifts have intervals excluding zero, all upward (Qwen's at Anchor 8/8, incl. both stricter gap margins).
Final outcomes (Section 5.4, p. 8). Of 652 first-exact-pair rollbacks with final evidence, 159 prevented a final-pass loss and 202 blocked a gain. By trigger (post hoc), the canary pass rule prevented 125 and blocked 107; the gap rule alone prevented 34 and blocked 95, rejecting candidates 6.73 pp above the incumbent on average [4.30, 9.11].
Search without failures or returns. Partly. A replay excluding failed pairs and former incumbents (2 of 6 families, 10 of 24 lines; Appendix N, p. 41) completes 276 of 300 runs. Final pass rate changes by +0.35 pp (bound over 300 [−4.91, +3.09]). On 77 matched pairs Horizon−Anchor disagreement is +6.18 pp [+0.50, +11.71]; the 23 missing Horizon runs were rollback-heavy: a sensitivity check (all-pair bound [−11.2, +30.5]). Its 104 rollbacks with final evidence prevented 31 losses and blocked 34 gains (gap rule alone: 2 and 12). The failed-pair blacklist replay gives +1.75 pp [−1.27, +6.14] (Table 5).
W2 / Q2. Contribution and final selection
Contribution. A retry-aware trace and counting rules that separate first checks from repeats (p. 1; Section 2, p. 3).
Final selection. No (table). Hidden-only uses half the held-out tasks. With 16 held-out tasks and equal search limits (row 2), SEED-FULL trails hidden-only in 6 of 8 conditions.
Core panel: SEED-FULL minus hidden-only −0.14 pp [−0.97, +0.67]; minus No Rollback, −0.14 to −0.97 by profile (Horizon [−1.53, −0.42]). Resampling the 30 shared task partitions gives [−2.36, +0.05] for Horizon: no gain and possibly a loss. Static public-test-repair (REFLECTION-ONLY) beat SEED-FULL by 2.18 pp [0.72, 3.50] in Anchor and 3.40 [1.67, 5.08] in Confirmatory, and both controls too, at about 9% of the calls.
We ran no realistic search. For larger splits, MBPP+ matrices on two hosted models (378 tasks; DeepSeek and Qwen3.7-plus, families in the core panel) replay 200 split seeds; their shared pre-set power gate (simulated on archived lines) failed, half-widths 1.35/1.47 pp, and was amended before data; Anchor misses the new one. At 32+32 against 64-task hidden-only, Anchor reads "not a small-split effect" (pre-set rule) on both lines (SEED-FULL −0.20 pp [−0.85, +0.28]; +0.17 [−0.47, +0.60]); Horizon is "inconclusive" for DeepSeek (−0.39 [−2.64, +1.07]) and "not a small-split effect" for Qwen (−1.14 [−2.98, +0.50]). SEED-FULL keeps the base workflow in 198/182 (DeepSeek) and 197/160 (Qwen) seeds. The 12-workflow panel has 32 untouched final tasks but 16 held-out ones, a fixed library and one generation at 192 calls; never updating passes 720 of 1,280. All 16 paired intervals include zero; none met the prespecified cost-and-quality criterion (Appendix O, p. 43). On LiveCodeBench DeepSeek is one comparison (profiles identical; 211 of 504 responses capped, not SEED-FULL's on the 3 tasks it lost); re-sent a day later, only 29 of 100 requests matched (13 flipped pass/fail): no evidence of a gain (Appendix H.5, pp. 32–33).
Beyond this pipeline. We tested no other pipeline, so we claim only Section 6's reporting rule (p. 9, lines 473–479), each part now tied to a result here.
W3. Presentation
The abstract and introduction (p. 1, lines 050–053) now define the three profiles before any finding. Terminology is not reduced, but Section 3.1 (p. 3) now says SINGLE EVALUATOR, hidden-only, HO and H are one policy.