This paper proposes a meta-harness for an evaluation agent. They have two contributions over existing work: (a) disagreement based mining (b) depth-first evolution of agent.
Their set up has multiple steps, at a high level each of these are: (i) for a train-split they first find instances where multiple static LLMs disagree on their preference, here the split is 80% of instances are the ones where models disagreed and 20% are random pairs (ii) The second big step is the “evolution” they seed each agent set up with the same starting evaluation harness (rubrics, metrics etc). Then the evolution agent compares the scores and traces against human labels on the train instances chosen in step (i). It finds target failures, proposes corrections and writes new candidate evaluators. The “controller” then runs the initial evaluator, the candidate evaluator and compares gains. Based on certain heuristics, a controller decides to either confirm the change or stage it or roll it back. The important thing is that even after a rollback the diagnostics from the candidate run are kept. The agent then decides whether to continue down the same correction (in case it’s an implementation issue vs the correction not being right) or whether to pivot to a different direction. This is the depth first aspect of the proposal. At each step a direction/candidate evaluation harness is kept only if it clears a threshold on the validation set and is also decided by the evolution agent. Finally after the search is done, the harness scores the held out test pairs. Note that the model weights do not change in the proposed method, only the context around the agent changes.
The authors have three ways of evaluation: (a) preference accuracy/performance on a test set of 19 datasets (b) revising with the critiques of the evaluation agent on four domains (c) training with the rewards of the evaluation agent on four domains. In (a) they show that their evolved harness does the best compared to baselines like SkillOpt, GEPA, Meta-Harness that evolves with the same compute budget and starting point. (b) and (c) are downstream evaluations, which are evaluated with humans. They show compared to other baseline methods, their model achieves the best increase in performance.
We thank the reviewer for finding the problem timely, the motivation well founded, and the experimental evaluation extensive. The new evidence gives three direct answers to the review: EvalOpt's advantage persists across three underlying LLMs; continuation and critique-guided evaluation now extend beyond the original four domains, although reward-guided training remains a four-domain result; and restricting evaluation to direct human preferences leaves the main margin nearly unchanged.
W1/Q1: Generalization beyond one underlying LLM. The gain is not specific to GPT-5.6 Sol: Seed < GEPA < EvalOpt on all three tested underlying models. We replace both evaluator execution and evolution while holding labels, feedback, search caps, and splits fixed. EvalOpt exceeds GEPA by +5.5 [3.1, 7.9] with GPT-5.6 Sol, +4.9 [2.3, 7.5] with Claude Sonnet 4.6, and +3.9 [1.3, 6.5] with GPT-5.5, with positive point estimates in 19/19, 18/19, and 17/19 domains.
W2: Scope of the detailed analyses. The central continuation control now extends beyond four domains; the other detailed mechanism ablations and reward-guided training remain four-domain results. Full-system accuracy covers all 19 domains. EvalOpt versus fixed patience covers the original four domains at five runs/100 units and seven added domains under a lighter three-run/50-unit replication; nine of 11 point estimates are positive, one neutral, and one negative. Critique-guided refinement now covers six domains: EvalOpt-vs-Seed tie-aware preferences are 58.6% [54.1, 63.1] on TG4 and 57.2% [52.4, 62.0] on SV4. Staging, retained-history, case-trace, and reward-guided-training analyses remain limited to the original four domains, and we state those denominators explicitly.
W3/Q2: Preference-source sensitivity and direct human preferences. Restricting evaluation to the 12 direct-human-choice domains leaves the conclusion nearly unchanged: EvalOpt exceeds GEPA by +5.1 pp [3.0, 7.2], versus +5.5 pp across all 19 domains. These 1,750 Test pairs give Seed/GEPA/EvalOpt = 62.0/65.2/70.2; the difference is computed from unrounded domain means. Excluding all four score-derived domains similarly leaves +5.3 pp [3.5, 7.2]. We report the domain IDs and source taxonomy in the appendix; the subset analysis establishes robustness, not causal equivalence among feedback formats.