acceptodds
Under review as a conference paper at ICLR 2027

EvalOpt: Evolving Evaluators for Open-ended Tasks with Human Feedback

Abstract

Evaluating open-ended outputs requires task-dependent judgments across multiple aspects of quality that must be balanced in context. Existing proxy metrics and model-based evaluators do not consistently align with human preferences. Humans can often tell which of two outputs is better, but turning that judgment into an effective evaluation procedure is harder. Evaluator self-evolution offers a way to learn such procedures from human feedback, but the direction of improvement is often ambiguous. The same evaluation error can suggest several plausible corrections, while a useful direction may require multiple revisions before producing a measurable gain. An unsuccessful early attempt therefore provides limited evidence about whether a direction is ineffective or simply underdeveloped. In the default model-driven evolution setup we tested, unsuccessful early candidates were often followed by shifts to alternative directions. We present EvalOpt, a framework for self-evolving evaluators across open-ended tasks with fixed base-model weights. EvalOpt combines disagreement-based preference mining with depth-first evaluator evolution, explicitly maintaining an improvement direction across revisions until diagnostics support moving on or a rejection or resource limit is reached. Unsuccessful candidates guide further implementation with-out replacing the best evaluator retained on validation data. Across 19 domains, EvalOpt reaches 69.6% held-out decisive-pair accuracy, exceeding the initial Seed evaluator and GEPA, the strongest baseline, by 8.0 and 5.5 percentage points under matched feedback and search budgets. On four analysis domains, it exceeds fixed-patience continuation by 3.9 points, while preference mining adds 1.8 points over random sampling in existing-label replay at the same feedback budget. Independent blinded judgments favor EvalOpt-guided over Seed-guided outputs at tie-aware preference rates of 60.8% for critique-guided refinement and 59.1% for reward-guided training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic Evaluator,Recursive self-improvement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DjDoOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces EvalOpt, a framework for improving LLM-based evaluators through human feedback. It combines disagreement-based preference mining with depth-first evaluator evolution.

Critical strengths

The paper addresses a relevant and timely problem. The proposed framework is well motivated, and the experimental evaluation is extensive.

Critical weaknesses

The evaluation relies on a single underlying LLM, raising questions about generalizability across different models. Moreover, ablation studies and downstream evaluations are restricted to four domains. The evaluation also combines different sources of human preferences, including score-derived rankings, without a dedicated sensitivity analysis. This makes the robustness of the overall conclusions somewhat difficult to assess.

Questions for authors

Could the authors discuss whether the improvements are expected to generalize to different underlying LLMs? How would the overall results change when considering only datasets with direct human preference annotations?

1 comment

We thank the reviewer for finding the problem timely, the motivation well founded, and the experimental evaluation extensive. The new evidence gives three direct answers to the review: EvalOpt's advantage persists across three underlying LLMs; continuation and critique-guided evaluation now extend beyond the original four domains, although reward-guided training remains a four-domain result; and restricting evaluation to direct human preferences leaves the main margin nearly unchanged.

W1/Q1: Generalization beyond one underlying LLM. The gain is not specific to GPT-5.6 Sol: Seed < GEPA < EvalOpt on all three tested underlying models. We replace both evaluator execution and evolution while holding labels, feedback, search caps, and splits fixed. EvalOpt exceeds GEPA by +5.5 [3.1, 7.9] with GPT-5.6 Sol, +4.9 [2.3, 7.5] with Claude Sonnet 4.6, and +3.9 [1.3, 6.5] with GPT-5.5, with positive point estimates in 19/19, 18/19, and 17/19 domains.

Underlying LLMSeedGEPAEvalOptEvalOpt − GEPA [pipeline CI]Domains ahead
GPT-5.6 Sol61.564.169.6+5.5 [3.1, 7.9]19/19
Claude Sonnet 4.660.263.067.9+4.9 [2.3, 7.5]18/19
GPT-5.559.462.166.0+3.9 [1.3, 6.5]17/19

W2: Scope of the detailed analyses. The central continuation control now extends beyond four domains; the other detailed mechanism ablations and reward-guided training remain four-domain results. Full-system accuracy covers all 19 domains. EvalOpt versus fixed patience covers the original four domains at five runs/100 units and seven added domains under a lighter three-run/50-unit replication; nine of 11 point estimates are positive, one neutral, and one negative. Critique-guided refinement now covers six domains: EvalOpt-vs-Seed tie-aware preferences are 58.6% [54.1, 63.1] on TG4 and 57.2% [52.4, 62.0] on SV4. Staging, retained-history, case-trace, and reward-guided-training analyses remain limited to the original four domains, and we state those denominators explicitly.

W3/Q2: Preference-source sensitivity and direct human preferences. Restricting evaluation to the 12 direct-human-choice domains leaves the conclusion nearly unchanged: EvalOpt exceeds GEPA by +5.1 pp [3.0, 7.2], versus +5.5 pp across all 19 domains. These 1,750 Test pairs give Seed/GEPA/EvalOpt = 62.0/65.2/70.2; the difference is computed from unrounded domain means. Excluding all four score-derived domains similarly leaves +5.3 pp [3.5, 7.2]. We report the domain IDs and source taxonomy in the appendix; the subset analysis establishes robustness, not causal equivalence among feedback formats.

Reviewer_ptjwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a meta-harness for an evaluation agent. They have two contributions over existing work: (a) disagreement based mining (b) depth-first evolution of agent.

Their set up has multiple steps, at a high level each of these are: (i) for a train-split they first find instances where multiple static LLMs disagree on their preference, here the split is 80% of instances are the ones where models disagreed and 20% are random pairs (ii) The second big step is the “evolution” they seed each agent set up with the same starting evaluation harness (rubrics, metrics etc). Then the evolution agent compares the scores and traces against human labels on the train instances chosen in step (i). It finds target failures, proposes corrections and writes new candidate evaluators. The “controller” then runs the initial evaluator, the candidate evaluator and compares gains. Based on certain heuristics, a controller decides to either confirm the change or stage it or roll it back. The important thing is that even after a rollback the diagnostics from the candidate run are kept. The agent then decides whether to continue down the same correction (in case it’s an implementation issue vs the correction not being right) or whether to pivot to a different direction. This is the depth first aspect of the proposal. At each step a direction/candidate evaluation harness is kept only if it clears a threshold on the validation set and is also decided by the evolution agent. Finally after the search is done, the harness scores the held out test pairs. Note that the model weights do not change in the proposed method, only the context around the agent changes.

The authors have three ways of evaluation: (a) preference accuracy/performance on a test set of 19 datasets (b) revising with the critiques of the evaluation agent on four domains (c) training with the rewards of the evaluation agent on four domains. In (a) they show that their evolved harness does the best compared to baselines like SkillOpt, GEPA, Meta-Harness that evolves with the same compute budget and starting point. (b) and (c) are downstream evaluations, which are evaluated with humans. They show compared to other baseline methods, their model achieves the best increase in performance.

Critical strengths
  1. The authors have a well thought out set up for harness evolution and the idea is intuitive
  2. The evaluations are fairly broad and strong and they show consistent gains on their set up
  3. The downstream checks w/ editing with feedback and training w/ reward are very relevant and show utility of the approach
Critical weaknesses
  1. The novelty of the paper might be overstated, it misses some key references like papers that do harness evolution for judging (Eg: RewardHarness[1]) or doesn’t compare with work (in terms of baselines) that does rubric editing/improvement (as mentioned in the references).
  2. No OOD or cross-domain transfer evaluation
  3. There is no details of how the seed evaluation was created for certain datasets, many seeds are the authors own rubrics and the paper has no examples or process of how this was created).
  4. Missing significance testing for most of the tables
  5. From Fig A1b, it seems like the anchor effect is quite small and mining even random pairs gives similar performance compared to the proposed mixed set.
  6. The baselines seem to be all using the same mined samples; not sure if the performance would increase if the authors increased the number of observed samples for the baselines or have a different selection method for the mined instances
  7. Section 4.2 has no takeaways
  8. For some of the datasets/tasks (Eg: AbGen-Eval or MATE-3D) the number of instances (train/val and test) seem to be very small (e.g. AbGen-Eval has 20 validation instances and 30 test instances). Maybe this makes the results not statistically significant.
  9. The title of the paper is “evolving evaluators with human feedback” but the role of human feedback seems pretty limited, it seems that for most datasets the gold labels come from the dataset itself and the paper doesn't really study the cost/effect of human feedback in the evolution process
  10. The paper is missing computational details or per-method costs
  11. The C2/C3 experiments are missing details around human annotators,
  12. There are a bunch of writing issues, beginning with the introduction which is extremely compressed, so much that it is hard to parse (e.g. lines # 96-98 is not even a complete sentence).

[1] RewardHarness: Self-Evolving Agentic Post-Training

Questions for authors
  1. If you did harness evolution on one task within a group, would the harness generalize to other tasks, or is this representing some sort of over-fitting?
  2. How important is mining? From Figure A1 it seems like the difference between cross-model and mixed at 100 samples is not that large? Did you check significance? Also from A1 it seems like the random and no random performances are similar?
  3. Does mining help GEPA/SkillOpt too or does it only help EvalOPT?
  4. In Figure 3c we see that within a domain, different tasks achieve different levels of improvements. Do you have any thoughts on why this is?
  5. How expensive is it to run EvalOPT compared to other baseline methods? How many candidates does each method tend to evaluate per method?
4 comments

We thank the reviewer for recognizing the motivation, broad 19-domain evaluation, and downstream human-evaluated utility. The central contribution is the separation of correction-direction lifetime from candidate lifetime: a candidate can be rolled back while its diagnostics continue guiding implementation of the same correction. We substantiate this distinction with RewardHarness and rubric-only comparisons plus targeted mechanism controls, and address transfer, mining, baseline fairness, significance, Seed construction, cost, annotation, and the paper's compressed presentation independently below.

W1: Novelty, RewardHarness, and rubric-editing methods. We have expanded the related-work discussion to include RewardHarness, EvalGen, Reflect-and-Revise, Auto-Rubric, and rubric-editing methods, and sharpened the contribution relative to them. EvalOpt introduces a controller state in which rejecting a candidate implementation does not automatically abandon the correction direction that produced it: the evaluator rolls back, while the failed candidate's diagnostics remain available for another implementation of the same correction. On the four-domain controlled subset, this distinction is empirically consequential: fixed patience reaches comparable depth but is 3.9 pp worse; removing retained history costs 4.5 pp; removing case-level traces costs 4.1 pp; and restricting evolution to rubric text removes the advantage over SkillOpt. For RewardHarness, we freeze the official repository revision before experiments and preserve its native Skill/Tool objects and code-default commit rule as the fidelity baseline (63.6). A stricter Train-internal acceptance gate is reported only as a sensitivity (64.0); sealed C1 Val is used for final selection under the common protocol. EvalOpt scores 69.6, so the conclusion does not depend on choosing either gate. A matched Rubric-Refine baseline scores 63.2, while the fixed-patience/history controls remain the primary mechanism evidence. Our specific claim is therefore rollback-aware depth-first evolution of executable evaluators, not that rubric or harness optimization itself is new.

W2/Q1: OOD transfer and possible overfitting. For evaluator evolution, the relevant question is whether an evaluator improved from limited human-judged pairs can compare new outputs for unseen inputs under the same quality standard. Group-level Train/Val/Test splits keep every output from one source input in one split, with Val/Test sealed from search. We also freeze source-evolved evaluators and test two source shifts using the same frozen rubric and only transport-level input adaptation, on 1,000 disjoint target groups each. A TL;DR-evolved evaluator gains +3.0 [0.5, 5.5] over the same frozen Seed on a CNN/DailyMail cohort, versus +5.4 for target-native evolution on disjoint target Train/Val (56% retained). An HPDv2-evolved evaluator gains +3.9 [1.4, 6.4] on a prespecified fresh 1,000-prompt image cohort with newly generated pairs and direct preferences, versus +6.5 target-native (60% retained). The target cohort is disjoint from the HPDv2 source cohort; an overlap audit covers prompts, images, and tool-training resources, and ImageReward, HPSv2, PickScore, and any preference model trained on the target corpus are disabled in both transferred and target-native conditions. Target Test is sealed from both source and target evolution. These results provide evidence against instance-level overfitting and support transfer under a preserved quality standard; transfer weakens when the standard itself changes.

W3: Seed evaluator construction. The paper does not start from deliberately weak, unconstrained prompts. All 19 default Seeds are grounded in a released benchmark rubric, published evaluation criteria, or the task specification. Where public criteria are incomplete, we add task-specific dimensions such as usability, safety, or executability; these additions are documented and frozen before comparative search without Val/Test feedback. Every optimizer receives the same Seed, whose overall accuracy is 61.5%. The minimal generic prompt appears only in a four-domain sensitivity analysis, where the optimizer ordering is unchanged across generic, published-criteria-only, and paper Seeds. The revision provides criterion-level provenance for all 19 domains and a complete executable SV3 example plus its evolution trace.

W4: Significance testing. The main system, mechanism, and downstream comparisons remain supported under dependence-aware inference. For Table 1, we resample original-input groups within each domain and then equally average the fixed 19 domains after averaging each method's independently run search blocks; search-run IDs are not paired across methods. Pipeline intervals additionally resample the independent search runs. All seven EvalOpt-versus-baseline contrasts in the revised Table 1, including code-default RewardHarness and Rubric-Refine, remain significant after Holm correction (adjusted p < 0.001); EvalOpt−GEPA is +5.5 pp with test-sampling CI [3.7, 7.3] and pipeline CI [3.1, 7.9]. EvalOpt−fixed patience has pipeline CI [1.1, 6.7], and the original four-domain downstream preferences remain above 50% with crossed input-and-rater inference. Mining uncertainty is reported separately in W5.

W5/Q2: How important are mining and the random anchor? Mining's main practical value is human-annotation efficiency when new preferences must be collected. Under the same allocated annotation budget in SV1, it increases decisive preferences from 26.2 to 36.0 per 100 ratings, reduces the aggregate tie rate across requested questions from 36% to 13%, lowers allocated ratings per decisive preference from 3.81 to 2.78, and produces more diverse dimensional feedback (2.21 versus 1.98 bits). Ties remain valid feedback, but this acquisition analysis measures the yield of non-tie labels used in decisive-pair accuracy. The prospective SV1 and IG1 acquisitions give positive downstream point estimates (+3.9 and +3.2 pp); conditional on these acquired feedback sets, the equal-domain test-sampling estimate is +3.6 [0.3, 6.8]. Fig. A1 is a complementary stress test after labels already exist: because its curated pool contains decisive labels, it removes the principal acquisition problem that mining addresses. In this setting, mined−random is +1.8 pp [−0.2, 3.8], while the incremental differences over cross-model disagreement and from the 20% random anchor are not resolved. We therefore base the mining claim on prospective annotation yield, without interpreting the two-acquisition CI as run-to-run acquisition uncertainty. The random anchor is retained as a coverage safeguard, not presented as an empirically optimal ratio.

W6/Q3: Does mining also help GEPA/SkillOpt, and would more observed samples change them? All optimizers receive the same selected feedback precisely so that EvalOpt does not benefit from privileged samples. We cross optimizer and selector while holding the revealed-group budget, common search-cost cap, attempt limit, model, and validation protocol fixed. Mining's point estimates are +0.6 for SkillOpt, +0.9 for GEPA, +1.2 for fixed patience, and +1.8 for EvalOpt, but the individual intervals overlap zero and the optimizer×selector interaction is unresolved. Thus, the current runs detect no optimizer-specific interaction; they do not establish separate mining gains for GEPA or SkillOpt. Under random feedback, EvalOpt still exceeds fixed patience by +3.3 pp, with test-sampling CI [1.2, 5.4] and pipeline CI [0.6, 6.0], so the controller advantage does not require mining. More feedback helps all methods descriptively: from 100 to 200 random groups, SkillOpt changes 67.6→68.5, GEPA 67.9→69.6, fixed patience 67.0→68.7, and EvalOpt 70.3→72.4. Because the 200-group condition doubles feedback, the matched 100-group comparison remains the primary optimizer comparison.

W7: Section 4.2 needs an explicit takeaway. The central decision is whether the current correction direction should be maintained, not how many fixed retries remain. Accumulated case traces and failed-candidate diagnostics provide the context for that judgment, while validation gates, rejection conditions, and direction budgets define its boundaries. Fixed patience reaches nearly the same depth yet trails by 3.9 pp on the controlled four-domain subset, showing that a fixed retry count does not replace evidence-based direction maintenance.

W8: Small datasets. The main conclusion does not depend on the two smallest benchmarks. Excluding AbGen-Eval and MATE-3D leaves EvalOpt−GEPA at +5.4 pp [3.7, 7.1] across the remaining 17 domains; weighting every domain by its Test-pair count gives +5.1 pp [3.5, 6.6]. We also report appropriately wider paired group-level intervals for the official small-domain estimates: AR2 is +4.7 [−9.6, 19.0] and IG5 is +6.8 [−4.3, 17.9].

We additionally use 3DGen-Bench as a separate related-domain replication, not as an enlargement of MATE-3D. We formulate five separately optimized evaluator tasks, one for each official dimension: Geometry Plausibility, Geometry Details, Texture Quality, Geometry-Texture Coherency, and Prompt-Asset Alignment. All five use the benchmark's official 9:1:2 prompt split; every asset, render, view, comparison, and dimension label associated with one prompt remains in the same split. Each dimension-specific Seed is frozen from the published criterion before labels are inspected. Search sees Train labels only, Val controls retention, and Test remains sealed; GEPA and EvalOpt use identical feedback groups, underlying model, tool roster, search cap, and run count. Held-out accuracy is computed on decisive left/right labels. Ties are evaluated separately under a frozen tie-aware rule; “Both Bad” is excluded from decisive accuracy and reported separately rather than collapsed into tie. EvalOpt has higher point accuracy on all 5/5 dimensions; this is a sign pattern, not a claim that every dimension is individually significant. The equal-weight macro advantage is +5.4 pp with pipeline CI [2.0, 8.8], jointly resampling paired prompt groups and independent search runs before averaging the five fixed tasks. The appendix reports each dimension's Train/Val/Test and decisive/tie/Both Bad counts. These tasks remain separate from MATE-3D, the 19-domain aggregate, and model selection for the paper domains.

3DGen-Bench dimensionGEPAEvalOptΔ (pp)
Geometry Plausibility63.167.8+4.7
Geometry Details61.866.9+5.1
Texture Quality65.471.6+6.2
Geometry-Texture Coherency62.768.3+5.6
Prompt-Asset Alignment64.069.4+5.4
Equal-weight macro-average63.468.8+5.4 [2.0, 8.8] pipeline

W9: Meaning and role of human preference feedback. EvalOpt studies an offline human-feedback setting: how to convert a fixed budget of human judgments into an evaluator that generalizes to unseen outputs, without requiring humans to provide new labels at every search step. A preference is the common supervision interface used for this purpose: under a task-specific human quality standard, output A is better, output B is better, or the two are tied. Direct comparisons are used as given; only benchmark-defined overall rankings or composite scores are converted into pairwise orderings. Standalone dimensional labels remain criterion-specific and are never collapsed into an overall preference; benchmark-constructed or mixed signals are identified separately. Ties remain valid supervision, although decisive-pair accuracy and acquisition-yield analyses use non-tie labels. For the subjective tasks studied here, the released direct-choice, ranking, and score-derived “gold labels” originate from judgments previously provided by human raters or domain experts. Reusing them changes when feedback was collected, not whether evolution is supervised by human judgment. Evaluator–preference disagreements identify failure cases, while disjoint validation preferences determine candidate retention. On the 12 direct-choice domains, EvalOpt remains +5.1 pp [3.0, 7.2] above GEPA, close to +5.5 pp overall.

W10/Q5: Computational details, cost, and candidate counts. EvalOpt's improvement does not come from a larger search budget. Per domain per search run, averaged over 19 domains × 5 runs, all methods receive a 100-unit search cap, consume approximately 97–99 search units, and cost \117–117–\\119 under the same model pricing snapshot. Mean charged attempts are 58.6 for SkillOpt, 55.2 for GEPA, 38.7 for Meta-Harness, 59.6 for code-default RewardHarness, and 47.3 for EvalOpt. EvalOpt allocates 23% of its cap to diagnostics and confirmation and averages 12.4 directions per run. Selection, search, Tune, terminal Test evaluation, deployment, and human annotation are reported separately under their own accounting units.

W11: Human annotators. The downstream endpoints use blinded, task-qualified volunteer judgments, with three separately elicited ratings per pair. The expanded protocol uses 144 raters in four task-family pools: computer-science students with relevant visual/interface experience for SV3/SV4, CapArena-trained raters for IG1, fluent-English computer-science students with summarization or instruction-following experience for TG1/TG4, and ML/NLP graduate researchers for AR1. Choices are A, B, tie, or unable to judge; AR1 also permits “Both Bad,” counted as tie and reported separately. The primary endpoint uses individual judgments, (W+T/2)/(W+L+T), with abstentions reported separately—not majority-vote labels. Excluding calibration and auxiliary attribute/regression ratings, C2/C3 contain 75,600 preference judgments, balanced to 450–585 per rater with at most 150 per session; domain medians range from 32 to 65 seconds, totaling approximately 982 person-hours. Participation is uncompensated, not tied to a course, employment, or lab evaluation, and permits withdrawal at any time; informed consent was obtained under the applicable minimal-risk ethics determination. Agreement is Fleiss' κ = 0.37–0.49, and uncertainty uses crossed input-and-rater resampling.

W12: Writing and presentation. We rewrite the introduction as problem → controller → evidence, define the controller terms before results, correct the sentence fragment, standardize domain/task/group/pair terminology, and add explicit takeaways after §§4.2–4.3.

Q4: Why do gains vary across tasks in Fig. 3c? A useful explanation is how difficult a correction direction is to implement faithfully. EvalOpt helps most when human preferences reveal a subtle, multidimensional alignment error, but correcting it requires coordinated changes to evidence collection, tool routing, or aggregation. TG2 (+9.4 pp), for example, must jointly align correctness, evidence support, citation precision/completeness, and clarity. IG6 (+9.4) must reconcile source-image fidelity with geometry, texture, and multiview consistency, while SV4 (+6.2) must connect source coverage and deck organization with visual and editability checks. Gains are smaller when correction is closer to one-step reweighting over mature signals: TG1 (+1.1) already has strong source-grounded summarization checks, IG4 (+2.2) has reference-conditioned image metrics, and TG5 (+2.3) closely matches an available reward signal. AR1 (+3.3) illustrates the other boundary: its novelty and feasibility criteria need refinement, but expert-context dependence makes diagnostics less stable. These domain cases are explanatory rather than causal; they are consistent with the controlled finding that deep continuation is most useful when a valid direction requires multiple implementations.

The revision will add the missing related work and controls to §§2 and 4, expose Seed construction and costs in the appendix, and rewrite the introduction and §4.2 takeaway so that the controller distinction is explicit before the implementation details.

Reviewer_VS6cOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

EvalOpt improves executable LLM evaluators from human preferences while keeping the underlying model weights fixed. It combines disagreement-based preference mining with depth-first evaluator evolution. The key distinction is between rejecting a candidate evaluator and abandoning the correction it was meant to implement. A failed candidate can still supply useful diagnostics for another attempt at the same correction, while validation determines which evaluator is ultimately retained. The main evaluation spans 19 domains in structured visuals, images and 3D, text generation, and automated research. EvalOpt reaches 69.6% held-out decisive-pair accuracy, compared with 61.5% for Seed and 64.1% for GEPA (Table 1). Four-domain analyses examine continuation, staging, diagnostic history, and feedback selection. Independently judged downstream outputs favor EvalOpt over Seed at tie-aware preference rates of 60.8% for critique-guided refinement and 59.1% for reward-guided training (Figure 4b).

Critical strengths
  1. Separating the lifetime of a correction from the lifetime of a candidate is a useful contribution. EvalOpt preserves diagnostic evidence after rollback and uses it to guide further attempts. On the four analysis domains, it reaches 72.1% accuracy versus 68.2% for fixed patience and 67.4% for history-based search (Table A8). Fixed patience and EvalOpt have similar mean revision depths, so the result supports looking beyond simply allowing more attempts, although it does not by itself establish the causal mechanism.
  2. The evaluation connects evaluator agreement to downstream usefulness. The 19-domain comparison covers several relevant optimizers, and the refinement and reward-training experiments use blinded human judgments rather than the optimized evaluator as their endpoint. These checks make the contribution more convincing than an improvement in evaluator scores alone.
Critical weaknesses
  1. The evidence for the proposed mechanisms is substantially narrower than the main benchmark. Search-policy and mining analyses cover four of the 19 domains, using one shared execution and optimization model, GPT-5.6 Sol. The broad benchmark supports the complete system, but gives less evidence that diagnostic continuation is consistently responsible for the gains. The controlled diagnostic cases help; they do not settle how reliably the agent diagnoses failures in ordinary search trajectories.
  2. Mining offers a smaller and less certain benefit than the search policy. At 100 revealed groups, it improves EvalOpt from 70.3% to 72.1%, with total model-and-tool cost increasing from 102 to 106 relative units (Table A10). The paper reports these costs and correctly treats the 0.6-point interaction as descriptive. My concern is therefore the robustness and practical value of the 1.8-point gain, rather than a missing cost comparison. Existing-label replay on four domains and the separate slide-annotation study do not yet establish when the extra acquisition machinery is worthwhile.
Questions for authors

These are clarification requests about the evidence supporting the present claims, rather than requests for additional future-work projects.

  1. What evidence from the remaining domains supports the generality of diagnostic continuation and mining? In particular, were there domains where either component had little benefit or made performance worse? This would affect how broadly I interpret the proposed mechanisms.
  2. The appendix describes independent review of natural search trajectories. What did those reviews find about incorrect continuation or premature termination decisions, and reviewer agreement? The controlled cases in Table A9 are useful, but aggregate findings from ordinary trajectories would help assess the reliability of the diagnostic agent.
  3. Can the existing runs support uncertainty estimates for the mined-minus-random accuracy difference in Table A10, with the dependence structure made explicit? This would help determine whether the modest acquisition gain is consistent enough to support the mining contribution.
1 comment

We thank the reviewer for recognizing the value of separating correction lifetime from candidate lifetime, as well as the breadth of the 19-domain and downstream evaluations. We add three forms of evidence requested by the reviewer: continuation controls on 11 domains, an aggregate audit of diagnostic decisions on ordinary trajectories, and dependence-aware mining analyses. The new results show that diagnostic continuation extends beyond the original four domains, validation protects the retained evaluator from incorrect continuation decisions in the audited runs, and mining substantially improves the yield of informative preferences during prospective human annotation.

W1/Q1: Breadth and limits of diagnostic continuation. Diagnostic continuation has positive point estimates in nine of 11 controlled domains, with IG4 neutral (+0.4 pp) and TG5 negative (−0.6 pp). Against fixed patience—which reaches almost the same revision depth (3.6 versus 3.8)—EvalOpt gains 3.9 pp [1.1, 6.7] on the original four domains. A lighter replication on seven added domains (three runs and a 50-unit cap, versus five runs and 100 units originally) gives +2.9 pp [0.1, 5.7], testing whether the direction of the effect extends beyond the original subset rather than pooling identical protocols. On the original controlled subset, removing retained history and case-level traces costs 4.5 and 4.1 pp. These results distinguish evidence-guided refinement of the same correction from simply allowing more attempts. The mechanism helps most when a valid correction requires several implementations and less when the first implementation succeeds or validation evidence is noisy.

Q2: Reliability in ordinary search trajectories. The audit shows that diagnostic decisions are imperfect, but the validation gate contained every identified continuation error. Two independent reviewers examined 312 continuation decisions nested within 117 stratified directions using only contemporaneous evidence. They labeled another attempt justified when the trace supported the same correction, incorrect when it supported pivoting or stopping, and undeterminable otherwise. The design-weighted incorrect-continuation rate is 11.6% [8.1, 15.6], with 88.5% agreement (κ = 0.68); none of the 38 audited errors promoted a candidate on Val. For endings, 7/40 agent-chosen stops were premature, with 85.0% agreement (κ = 0.61). Reviewers also considered continuation potentially useful in 15/37 forced budget/rejection endings, which are censored rather than agent errors. Thus, an incorrect diagnostic decision can waste search effort, but does not directly replace the retained evaluator.

W2/Q1/Q3: Mining robustness, uncertainty, and practical value. Mining is designed to improve the efficiency of acquiring new human preferences, and the prospective studies provide the most direct evidence for this role. Here, decisive yield counts non-tie preference labels; ties remain valid feedback but are excluded from decisive-pair accuracy. Under the same allocated annotation budget, mining raises decisive-label yield from 26.2 to 36.0 per 100 ratings in SV1, reduces the aggregate tie rate across requested overall and dimensional questions from 36% to 13%, and lowers allocated ratings per decisive preference from 3.81 to 2.78. The prospective SV1 and IG1 acquisitions give positive downstream point estimates (+3.9 and +3.2 pp); conditional on these two acquired feedback sets, the equal-domain test-sampling estimate is +3.6 pp [0.3, 6.8]. This interval captures held-out evaluation uncertainty, not acquisition-run variability. Table A10 asks a complementary question after labels already exist. A domain-stratified cluster analysis gives mined−random = +1.8 pp, with test-sampling CI [−0.2, 3.8], paired test-set permutation p = 0.082, and the broader five-seed pipeline CI [−0.9, 4.5]; the four domain point estimates are all positive. Mining evidence therefore covers four replay domains plus prospective SV1 and IG1—five unique domains—and supports an acquisition-efficiency claim rather than universal 19-domain mechanism generality. Its model-and-tool overhead is 3.9% (106 versus 102 normalized units), alongside 27% fewer allocated human ratings per decisive preference in SV1.

These additions will appear in revised §§4.2–4.3 and the appendix trajectory audit, together with the per-domain results, resampling unit, reviewer agreement, and observed failure cases.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.