acceptodds
Under review as a conference paper at ICLR 2027

DUET: Co-Evolving Solver and Grader Agents

Abstract

Agentic workflows are increasingly used across domains such as technology, finance, and enterprise operations. As these agents become more widely deployed, continually improving them becomes increasingly important. This raises an immediate challenge: How should the agent evolve? This evolution requires effective evaluation that can assess outcomes and provide useful feedback for optimization. As the agent evolves, its behaviors and failure modes may also change, making a fixed evaluator increasingly inadequate. Another fundamental question: How should we evaluate an evolving agent? These two challenges are inherently coupled; changes in agent behavior can expose limitations of the current evaluator, while a stronger evaluator provides more informative feedback for improving the agent. Motivated by this interaction, we introduce DUET, a framework that jointly optimizes a solver agent and a grader agent to improve both. DUET iteratively selects training tasks, executes them with the solver, evaluates the resulting outcomes with the grader, and uses a tool-using update module to revise the solver and the grader, alternating between the two across rounds. By updating the grader within the optimization loop, DUET turns evaluation from a fixed source of feedback into a first-class optimization objective that adapts alongside the solver. Experiments across four agent benchmarks show that DUET improves both solver and grader performance and consistently outperforms baselines that optimize the solver with a fixed grader.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agent Optimization, Agent Evaluation, Prompt Optimization, Co-Evolution

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MkzJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DUET, a framework that jointly optimises the prompts (optionally, skills) of a solver agent and a grader agent without needing reference answers during training updates. The paper shows that DUET achieves higher solver scores than SkillOpt and GEPA, as well as higher grader pairwise agreement than an initial grader on constructed pair sets.

Critical strengths
  • The results are carefully reported with proper seeds, standard deviations, ablations of each design element, and hyperparameter sensitivity analyses. The paper also includes the full updater prompts and learned prompts, which is appreciated.

  • The problem is well motivated, and the reference-free setting where task instructions and environments exist but references and rubrics don't is realistic for open-ended agent tasks.

Critical weaknesses
  • Much of the gap to the baselines seems to come from the optimiser rather than from co-evolving the grader. For example DUET(G⋆) which fixes the oracle grader and only optimises the solver already beats SkillOpt(G⋆) and GEPA(G⋆) on τ²-Bench and GDPval.
  • In Appendix C.14, three of the four learned grader prompts contain explicit instructions about emitting a verdict that can be parsed reliably by the evaluation harness. Similarly the GDPval prompt notes that grades were silently discarded by the parser and recorded as a default middle score. This makes me wonder how much of the evidence for grader improvement is attributed to better compliance with the evaluation harness's way of parsing outputs?
  • The update agent is instructed to search the web (Section 4.3, Appendix B.2), with no domain restrictions described, while all four benchmarks publicly release reference answers, gold deliverables, or grading rubrics. This is cause for concern to me, since the updater (GitHub Copilot CLI with Claude Opus 4.8) could plausibly have encountered this material without anyone intending it (see my question below).
Questions for authors
  • Could you report optimisation training costs (tokens or dollars, including updater, grader and validation calls) for DUET versus the baselines?

  • Could the authors confirm whether the benchmarks' domains were blocked, and whether search logs were audited?

8 comments
AuthorsOpenReview ↗

Responses to Reviewer MkzJ (Part 1)

We thank the reviewer for the careful reading and thoughtful comments. We address the reviewer's questions in the following and have revised the paper accordingly. The changes are marked in blue in our revision. We hope the responses below address the reviewer's concerns.

W1: Much of the gap to the baselines seems to come from the optimiser rather than from co-evolving the grader.

We thank the reviewer for this insightful comment. To isolate the contribution of co-evolution, we conducted ablation studies and additional controlled experiments.

Co-evolution ablation. In Figure 3(a), we compare DUET with three ablations on Finance Agent Benchmark: (1) optimizing only the solver with a fixed grader, (2) optimizing only the grader with a fixed solver, and (3) sequential optimization, where the grader is optimized first and then frozen during solver optimization. DUET achieves the strongest overall performance, supporting the benefit of jointly optimizing the solver and grader within the same framework. We also extend this ablation to τ²-Bench in Appendix C.13 and observe similar results.

Cross-checkpoint grader evaluation. To assess whether grader quality improves as the solver evolves, we selected initial, intermediate, and final solver and grader checkpoints from the same DUET run, denoted as A0A_0, AmidA_{\mathrm{mid}}, and ATA_T for the solvers, and G0G_0, GmidG_{\mathrm{mid}}, and GTG_T for the graders. Then we used each solver to perform the same held-out tasks, and each grader to evaluate the resulting outputs. We used the benchmark's independent success checks to measure solver performance and grader AUROC, which reflects the grader's ability to distinguish successful from failed executions.

Grader AUROCA0A_0 outputsAmidA_{\mathrm{mid}} outputsATA_T outputsOverall
G0G_00.8140.6320.5890.707
GmidG_{\mathrm{mid}}0.8330.7480.6180.755
GTG_T0.8420.7400.6660.771

As the solver evolved, its success rate independently measured by the benchmark improved from 80.0% to 88.3% to 89.2%. The final grader achieved higher AUROC than the initial grader across all solver stages, with overall AUROC improving from 0.707 to 0.771. These results provide additional evidence that grader evolution improves evaluation quality as solver behavior changes.

Controlled comparison of grader feedback. To assess whether the learned grader provides more useful feedback for solver optimization, we conducted controlled experiments on SpreadsheetBench and τ²-Bench. Starting from the same initial solver checkpoint A0A_0, we used G0G_0 and GTG_T to independently evaluate the same training trajectories and generate scores and critiques. We then applied the same updater to update the solver, repeating each condition three times. The resulting solvers were evaluated on the same held-out tasks using the benchmark's reference evaluator.

SpreadsheetBenchτ²-Bench
Initial solver A0A_073.3%80.0%
Updated with G0G_0 feedback82.5%88.1%
Updated with GTG_T feedback85.7%88.9%

Our results show that updates guided by the learned grader achieve higher solver performance on both benchmarks. Since the updater is fixed, these results provide additional evidence that improved grader feedback contributes to solver optimization.

Motivation for SkillOpt (G⋆G^\star) and GEPA (G⋆G^\star). In our experiments, we include SkillOpt (G⋆G^\star) and GEPA (G⋆G^\star) as stronger baselines by allowing both methods to use the benchmark's reference evaluator during solver optimization. Here, G⋆G^\star denotes the reference evaluator provided by the benchmark authors, which may rely on reference answers and manually designed task-specific grading criteria. This provides them with a strong, task-specific evaluation signal that DUET does not require. Despite optimizing from a minimal grader without ground-truth supervision, DUET outperforms both baselines across all four benchmarks, even when they use G⋆G^\star.

We have included the cross-checkpoint grader evaluation in Section 5.1 and the controlled comparison of grader feedback in Section 5.3 of the updated version.

AuthorsOpenReview ↗

Responses to Reviewer MkzJ (Part 2)

W2: How much of the evidence for grader improvement is attributed to better compliance with the evaluation harness's way of parsing outputs?

We thank the reviewer for raising this concern. The parsing-related instructions were learned to address output-format issues encountered during optimization. To investigate this, we audited grader parsing and conducted additional evaluations. On GDPval, even when restricting the evaluation to pairs for which both graders produced valid scores, the learned grader achieves higher pairwise agreement than the initial grader (77.8% vs. 60.7%). Furthermore, in our additional cross-checkpoint evaluation on τ²-Bench in our response to W1, all 1,080 grader evaluations were successfully parsed, and overall AUROC still improves from 0.707 to 0.771. These results provide evidence that grader improvements are not solely due to better output-format compliance.

Q1: Optimization cost compared with baselines.

DUET and the baselines (SkillOpt and GEPA) all require solver rollouts, grading or reward evaluation, and solver optimization. DUET additionally updates the grader during optimization. In a 30-round τ²-Bench run, these two steps accounted for 8.2% of the total optimization cost. This suggests that DUET achieves better solver performance than the baselines with relatively modest additional optimization cost from grader evolution.

Furthermore, the baseline methods require a predefined evaluator, which may involve manually designing task-specific grading criteria or collecting reference answers. DUET can learn its grader starting from a minimal prompt, reducing the need for manual evaluator design while achieving competitive solver performance.

We have included this analysis in Appendix C.10 of the updated version.

W3 and Q2: Benchmark contamination through web search.

We thank the reviewer for raising this concern. We would like to clarify how web access is controlled in DUET.

Who can search: In DUET, only the updaters can use web search to research general improvement strategies. The solver and grader agents do not have web-search tools, except on the Finance Agent Benchmark, where web search is part of the benchmark environment, as this benchmark contains web research tasks.

What the updater sees: The updaters see only training-split tasks and their trajectories; held-out test tasks, outputs and scores never enter their workspace, so the updater has no test-task content to search for.

What the updater searches for: The scope of the updater's web searches focuses on general strategies for improving agent performance instead of benchmark-specific answers. For example, the solver updater is instructed to "web search current practice (agent design, tool-use, planning, self-verification)" when proposing improvements.

What the updater may write: Its output must be a general prompt, and rules that "hard-code a sheet name, cell address, or value" are considered overfitting and are not allowed. So even a retrieved answer would have nowhere legitimate to go.

We also audited the updater's web-search logs and found no benchmark reference answers were retrieved during optimization. For example, on the Finance Agent Benchmark, we found no benchmark repository pages or grading-rubric assertions in the retrieved content.


We thank the reviewer again for the helpful comments. If our responses have addressed your concerns, we would appreciate it if you could consider raising your rating. We would also be happy to address any further questions you may have.

Dear authors of Submission12660,

Thank you for your rebuttal and for including revisions to the paper following the review(s). You have addressed my question regarding the web search tool.

To clarify my original question regarding costs: I was hoping for the total optimisation cost of DUET compared with SkillOpt and GEPA (e.g, total tokens or dollars per run, including updater, grader, and validation calls). Since DUET runs validation every round and uses a tool-using updater with web search, the absolute totals would help readers judge the comparison.

Regarding the controlled feedback comparison (Section 5.3, Table 6):

  • Please report mean & SD over the three repeats and state the number of runs in the caption. On τ2\tau^2-Bench, the gap is 0.8 where Table 1 reports SDs of 2.1-6.4 on this benchmark. It's difficult to interpret this as a difference without variance estimates.

  • Could you describe the setup more fully? For example how many update steps were applied to A0A_0, how many training trajectories were re-graded, which DUET run GTG_T was taken from, and what is different across the repeats?

  • The G0G_0-guided result on τ2\tau^2-Bench (88.1%) is higher than the fixed-grader ablation in Table 14 (83.0%) and SkillOpt/GEPA in Table 1 (84.7% / 83.3%). Could you explain this difference? If this is already reached by updates guided by the initial grader, it would suggest to me that much of the gain comes from the updater rather than from grader co-evolution.

Also: the initial-solver results in Table 6 differ from A0A_0 in Table 1. Are these separate evaluation runs?

Minor presentation nit: could you please report the standard deviations, as well as the number of runs consistently throughout the revisions? For example Table 10 is lacking the standard deviations that were given in Table 1.

Reviewer_9kc7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DUET studies whether a solver agent and the grader providing its feedback can improve together. With model weights and tools fixed, a separate update agent alternately revises the solver’s and grader’s prompts or skills using execution trajectories, artifacts, and feedback. Training tasks are selected adaptively based on recent score changes, and an edit budget limits each revision. Training updates do not use reference answers, although the default procedure uses labeled validation for checkpoint selection and stopping.

The paper evaluates DUET on spreadsheet manipulation, customer-service tool use, and financial question answering. Solver performance improves over SkillOpt and GEPA across all four benchmarks in the reference-free setting, including comparisons where these baselines receive reference evaluators. Grader improvements are larger on Finance and GDPval and small on T2-Bench and SpreadsheetBench. Ablations examine sequential versus alternating optimization, task selection, and update constraints, with most conducted on Finance Agent Benchmark.

Critical strengths

Across all four benchmarks, DUET achieves the highest mean solver performance among the reference-free methods. It also exceeds the reported means of these baselines when they use reference evaluators. This consistency across different tasks is the strongest reason for my positive assessment: the overall procedure appears useful even where the mechanism behind its gains remains uncertain.

I appreciated the comparison with sequential grader-then-solver optimization, which tests whether continued interaction contributes beyond obtaining a better grader once. Separate grader evaluation and the fixed-round GDPval experiment provide additional checks on the approach. Together, these experiments offer meaningful support for the method beyond a comparison against an unoptimized starting point, although their coverage could be broader.

Critical weaknesses

It is unclear how much the grader updates contribute to the solver gains. On T2-Bench and SpreadsheetBench, solver performance improves substantially, but grader agreement increases by only 0.6-0.9 percentage points. The grader may be producing more useful critiques without becoming better at ranking outputs, but the paper does not test this directly. The co-evolution ablations provide some evidence on Finance, but a similar test on another benchmark would make the central claim more convincing. For instance, one useful experiment would start from the same solver checkpoint and compare updates guided by the initial versus learned grader.

The paper does not experimentally compare with an existing co-evolution method. Its main baselines, GEPA and SkillOpt, keep the grader fixed, so the results establish an advantage over fixed-grader optimization but leave the advance over prior co-evolution approaches unresolved. Given that jointly improving task solving and evaluation is the central contribution, a comparison with a relevant method such as CoEvoSkills or Double Ratchet is warranted. Absence of this comparison would incline me to lower my score.

Questions for authors
  • Could you compare the initial and learned graders as sources of feedback for updates from the same solver checkpoint, using the same tasks and evaluating the resulting solvers with the reference evaluator?

  • Could the sequential-training or partially frozen-grader ablation be extended to one additional benchmark, preferably one with substantial solver gains but little change in grader agreement?

  • Do the grader gains persist on a modest, unfiltered sample of held-out pairs?

  • Approximately how much additional optimization cost comes from grader evolution?

4 comments
AuthorsOpenReview ↗

Responses to Reviewer 9kc7 (Part 1)

We thank the reviewer for the careful reading and thoughtful comments. We address the reviewer's questions in the following and have revised the paper accordingly. The changes are marked in blue in our revision. We hope the responses below address the reviewer's concerns.

W1 and Q1: It is unclear how much the grader updates contribute to the solver gains.

We thank the reviewer for this insightful comment. For τ²-Bench and SpreadsheetBench, the reference evaluators provide only binary success or failure labels. Therefore, the preference pairs constructed for grader evaluation in Table 1 mainly capture coarse pass–fail differences, instead of finer distinctions in output quality. This may explain the relatively small improvements in grader agreement reported in Table 1.

Cross-checkpoint grader evaluation. To further assess grader improvement as the solver evolves, we conducted an additional evaluation on τ²-Bench. We selected initial, intermediate, and final solver and grader checkpoints from the same DUET run, denoted as A0A_0, AmidA_{\mathrm{mid}}, and ATA_T for the solvers, and G0G_0, GmidG_{\mathrm{mid}}, and GTG_T for the graders. Then we used each solver to perform the same held-out tasks, and each grader to evaluate the resulting outputs. We used the benchmark's independent success checks to measure solver performance and grader AUROC, which reflects the grader's ability to distinguish successful from failed executions.

Grader AUROCA0A_0 outputsAmidA_{\mathrm{mid}} outputsATA_T outputsOverall
G0G_00.8140.6320.5890.707
GmidG_{\mathrm{mid}}0.8330.7480.6180.755
GTG_T0.8420.7400.6660.771

Overall AUROC increased from 0.707 to 0.755 to 0.771 across grader checkpoints. Pairwise agreement also improved from 73.2% to 79.6% to 85.2%, showing a larger gain than in the original constructed evaluation set. These results provide additional evidence that grader adaptation improves evaluation quality across evolving solver behaviors.

Controlled comparison of grader feedback. To assess whether the learned grader provides more useful feedback for solver optimization, we conducted controlled experiments on SpreadsheetBench and τ²-Bench, following the reviewer's suggestion. Starting from the same initial solver checkpoint A0A_0, we used G0G_0 and GTG_T to independently evaluate the same training trajectories and generate scores and critiques. We then applied the same updater to update the solver, repeating each condition three times. The resulting solvers were evaluated on the same held-out tasks using the benchmark's reference evaluator.

SpreadsheetBenchτ²-Bench
Initial solver A0A_073.3%80.0%
Updated with G0G_0 feedback82.5%88.1%
Updated with GTG_T feedback85.7%88.9%

Our results show that updates guided by the learned grader achieve higher solver performance on both benchmarks. For example, they outperform G0G_0-guided updates by 3.2 percentage points on SpreadsheetBench.

We have included the cross-checkpoint grader evaluation in Section 5.1 and the controlled comparison of grader feedback in Section 5.3 of the updated version.

AuthorsOpenReview ↗

Responses to Reviewer 9kc7 (Part 2)

W2: The paper does not experimentally compare with an existing co-evolution method.

We thank the reviewer for this suggestion.

CoEvoSkills focuses on generating agent skills by co-evolving a skill generator and a surrogate verifier that produces executable tests. Its optimization loop also uses feedback from an external ground-truth oracle to guide refinement. In contrast, DUET jointly optimizes a task-solving agent and a grader agent without ground-truth supervision during training updates. Our grader can evaluate aspects of task quality that are difficult to capture through executable tests, such as evidence grounding and the appropriateness of intermediate actions. These differences in optimization targets, evaluation mechanisms, and external supervision requirements make an empirical comparison difficult to conduct fairly without substantially modifying the original methods.

Double Ratchet is concurrent work released on arXiv on July 30, 2026. We examined its official implementation and found several challenges in applying it to our benchmarks. Double Ratchet evolves evaluation metrics composed of task-specific drawback detectors, using a labeled reference set to guide metric selection. Its released implementations target code generation, text-to-SQL, and report generation. Applying it to our agent benchmarks would require manually developing new task-specific detectors and substantially adapting its solver and evaluation infrastructure. We are working on adapting Double Ratchet to our benchmarks and will include the comparison in the final version of the paper if a faithful comparison can be established.

We also emphasize that GEPA and SkillOpt are strong, representative baselines for agent optimization. To provide a stronger comparison, we evaluate these baselines not only in the reference-free setting but also with access to the benchmark's reference evaluator (G⋆G^\star) during solver optimization. This provides them with a strong, task-specific evaluation signal that DUET does not require. Despite optimizing from a minimal grader without ground-truth supervision, DUET outperforms both baselines across all four benchmarks, even when they use G⋆G^\star.

We have added the discussion of CoEvoSkills and Double Ratchet in Section 2 of the updated version.

Q2: Could the sequential-training or partially frozen-grader ablation be extended to one additional benchmark?

We thank the reviewer for this suggestion. We have extended the sequential training ablation in Figure 3(a) to τ²-Bench.

MethodSolver Success Rate
Fixed grader83.0%
Sequential optimization86.4%
DUET (joint co-evolution)89.4%

The results show that DUET outperforms both fixed-grader and sequential optimization by 6.4 and 3.0 percentage points, respectively. These results further demonstrate the benefit of joint co-evolution over optimizing the solver and grader separately.

We have included these additional ablation results in Appendix C.13 of the updated version.

Q3: Do the grader gains persist on a modest, unfiltered sample of held-out pairs?

Yes. As detailed in cross-checkpoint grader evaluation in our response to W1, we evaluated three grader checkpoints (G0G_0, GmidG_{\mathrm{mid}}, GTG_T) on 360 unfiltered solver outputs from 120 held-out τ²-Bench tasks. Our results show that the final grader achieves a higher overall AUROC of 0.771 compared with the initial grader (0.707), providing further evidence that grader improvements persist on unfiltered held-out evaluations.

Q4: Approximately how much additional optimization cost comes from grader evolution?

We analyzed the optimization cost of a 30-round DUET run on τ²-Bench. Grader evolution accounts for approximately 8.2% of the total optimization cost. Therefore, the additional cost of grader evolution is relatively modest.

We have included this analysis in Appendix C.10 of the updated version.


We thank the reviewer again for the helpful comments. If our responses have addressed your concerns, we would appreciate it if you could consider raising your rating. We would also be happy to address any further questions you may have.

Thanks to the authors for their thorough rebuttal. I believe incorporating these changes will help improve the paper. My concerns are addressed and I maintain my positive rating.

Thank you very much for your positive feedback! Your thoughtful comments have helped us improve the quality of our work. We truly appreciate your time and effort.

Reviewer_qUe6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DUET, a framework for jointly optimizing a solver agent and a grader agent. It addresses the limitation of existing agent optimization methods that rely on fixed graders, which may become less effective as solver capabilities and failure modes evolve.

DUET alternates between updating the solver and grader using execution trajectories, evaluation scores, and natural-language critiques. The framework incorporates adaptive task selection to prioritize informative training examples and bounded updates to control changes to agent prompts and skills. Training updates do not require ground-truth labels, although labeled validation data is used for checkpoint selection and early stopping.

The authors evaluate DUET on four benchmarks spanning spreadsheet manipulation, multi-turn tool use, and financial question answering: SpreadsheetBench, GDPval, τ²-Bench, and Finance Agent Benchmark. Results show consistent improvements in solver performance over fixed-grader optimization baselines, including SkillOpt and GEPA. Grader performance also improves, although gains are substantial primarily on Finance Agent Benchmark and GDPval and marginal on the other two benchmarks. Ablation studies examine the contributions of co-evolution, adaptive task selection, and alternating updates.

Overall, the paper suggests that jointly evolving evaluation and task-solving capabilities can improve agent optimization and reduce reliance on carefully designed initial graders.

Critical strengths
  1. Important and well-motivated problem. The paper addresses a real challenge in agent optimization: as solver capabilities evolve, a fixed grader may no longer provide sufficiently accurate or informative feedback. Jointly improving the solver and grader is an interesting and practically relevant direction.

  2. Practical approach with encouraging results. DUET proposes a general framework combining alternating solver–grader updates, adaptive task selection, and bounded revisions. The experiments cover four benchmarks and show consistent solver improvements over fixed-grader baselines. The ablations also provide useful evidence for the proposed design choices.

Critical weaknesses
  1. Stability and soundness of co-evolution. My primary concern is how the framework ensures the quality and consistency of the grader throughout co-evolution. Without ground-truth supervision during updates, the solver and grader may reinforce each other's biases, leading to grader drift, reward hacking, or loss of previously learned evaluation capabilities. Similar to instability in adversarial optimization, the coupled process could converge toward undesirable behaviors. Alternating updates and bounded revisions do not necessarily prevent these failures. I did not see a systematic analysis of such risks.

  2. Dependence on validation data and potential overfitting. Although the method is described as reference-free, labeled validation data is used for checkpoint selection and early stopping after each round. The validation sets are very small (3 tasks for GDPval, 6 for Finance, 8 for τ²-Bench, and 20 for SpreadsheetBench). Repeated selection against these small sets raises concerns about overfitting. More importantly, the reliability of the evolving grader may depend heavily on how well the validation distribution covers the diversity of tasks, failure modes, and evaluation criteria. This dependence is not sufficiently investigated.

  3. Experimental evidence does not fully support the co-evolution claims. While solver performance improves, grader improvements are uneven: only 0.6 and 0.9 percentage points on τ²-Bench and SpreadsheetBench, respectively. Grader quality is primarily measured through pairwise agreement on relatively small, specially constructed test sets. It remains unclear whether the grader actually becomes more reliable across evolving solver behaviors or simply improves on the measured evaluation dimensions. The experiments also do not directly demonstrate that a fixed grader becomes inadequate as the solver evolves.

Questions for authors
  1. Co-evolution stability: How does DUET prevent the solver and grader from reinforcing each other's errors? Have the authors observed reward hacking, grader drift, oscillation, or degradation of previously learned evaluation capabilities?

  2. Longitudinal evaluation: Can the authors evaluate grader checkpoints against solver outputs from different optimization rounds, using an independent reference evaluator? This would help establish whether grader adaptation is necessary and whether its quality consistently improves.

  3. Validation overfitting and distribution sensitivity: How sensitive are the results to validation-set size, composition, and distribution shift? Given the very small validation sets and repeated checkpoint selection, can the authors extend the fixed-budget, no-validation-selection ablation to all four benchmarks?

  4. Adaptive task selection: Since both the solver and grader change, how can score differences across rounds reliably indicate learning progress rather than changes in grader calibration or scoring preferences?

  5. Reward hacking and grader property retention: Can the authors analyze whether improvements in learned-grader scores correlate with independently measured task quality, and whether the grader preserves its ability to detect previously recognized failure modes?

3 comments
AuthorsOpenReview ↗

Responses to Reviewer qUe6 (Part 1)

We thank the reviewer for the careful reading and thoughtful comments. We address the reviewer's questions in the following and have revised the paper accordingly. The changes are marked in blue in our revision. We hope the responses below address the reviewer's concerns.

W1, Q1, Q2, and Q5: Stability and soundness of co-evolution.

We thank the reviewer for the insightful comments. DUET does not directly optimize the solver to maximize grader scores as in adversarial optimization. The grader's feedback is used to identify potential strengths and weaknesses, but the updater must verify these findings against the actual execution trajectories and artifacts before making changes. As explicitly stated in the solver updater prompt (Appendix B.2), "judge the agent's actual work against the real workbook—the grader is evidence, not ground truth." This means that the updater does not blindly follow the grader's feedback. Similarly, when updating the grader, the updater examines the grader's report and investigation process against the actual artifacts to identify missed errors or incorrect evaluation practices. This design helps mitigate the risk of self-reinforcing errors during co-evolution.

We appreciate the reviewer's suggestion to conduct a cross-checkpoint evaluation on τ²-Bench. We selected initial, intermediate, and final solver and grader checkpoints from the same DUET run, denoted as A0A_0, AmidA_{\mathrm{mid}}, and ATA_T for the solvers, and G0G_0, GmidG_{\mathrm{mid}}, and GTG_T for the graders. Then we used each solver to perform the same held-out tasks, and each grader to evaluate the resulting outputs. We used the benchmark's independent success checks to measure solver performance and grader AUROC, which reflects the grader's ability to distinguish successful from failed executions.

Grader AUROCA0A_0 outputsAmidA_{\mathrm{mid}} outputsATA_T outputsOverall
G0G_00.8140.6320.5890.707
GmidG_{\mathrm{mid}}0.8330.7480.6180.755
GTG_T0.8420.7400.6660.771

As the solver evolved, its success rate independently measured by the benchmark improved from 80.0% to 88.3% to 89.2%. Meanwhile, the final grader's mean score increased from 7.33 to 7.78, suggesting that higher grader scores reflected genuine improvements in task performance. The final grader achieved higher AUROC than the initial grader across all solver stages, with overall AUROC improving from 0.707 to 0.771. Importantly, on the initial solver's outputs, AUROC did not deteriorate (0.814 →\rightarrow 0.842), suggesting no forgetting of earlier evaluation capabilities.

Overall, we observed genuine solver improvement and no evidence of grader drift or forgetting during co-evolution. These results provide empirical support for the stability of DUET across different stages.

We have included the additional experimental results in Section 5.1 of the updated version.

W2 and Q3: Dependence on validation data and potential overfitting.

To investigate whether DUET's performance depends on validation data, we extended the fixed-budget ablation in Appendix C.12 to all four benchmarks, as suggested by the reviewer. In this setting, DUET runs for a fixed number of rounds without validation-based checkpoint selection or early stopping, and we evaluate the final solver and grader checkpoints on the held-out test sets.

BenchmarkSolver (Val. selected)Solver (Fixed budget)Grader (Val. selected)Grader (Fixed budget)
Finance Agent96.596.775.073.1
τ²-Bench89.486.789.388.0
GDPval52.253.178.574.0
SpreadsheetBench84.787.072.873.2

The results show that DUET achieves comparable performance without validation-based checkpoint selection or early stopping. While our default setting uses validation data, the fixed-budget variant provides a practical alternative when a representative validation set is unavailable.

We thank the reviewer for the insightful suggestion and have included the additional experimental results in Appendix C.12 of the updated version.

AuthorsOpenReview ↗

Responses to Reviewer qUe6 (Part 2)

W3: Experimental evidence does not fully support the co-evolution claims.

We thank the reviewer for this comment. For τ²-Bench and SpreadsheetBench, the reference evaluators provide only binary success or failure labels. Therefore, the preference pairs constructed for grader evaluation in Table 1 mainly capture coarse pass–fail differences, instead of finer distinctions in output quality. This may explain the relatively small improvements in grader agreement reported in Table 1.

To further assess grader improvement as the solver evolves, we conducted the cross-checkpoint evaluation described in our response to W1. On τ²-Bench, we evaluated initial, intermediate, and final grader checkpoints on all 360 outputs from three solver checkpoints, using the benchmark's independent success checks. We measured grader AUROC to assess how well each grader distinguishes successful from failed executions, without filtering outputs or constructing selected pairs.

Overall AUROC increased from 0.707 to 0.755 to 0.771 across grader checkpoints. Pairwise agreement also improved from 73.2% to 79.6% to 85.2%, showing a larger gain than in the original constructed evaluation set. Furthermore, the initial grader's AUROC decreased from 0.814 on initial-solver outputs to 0.589 on final-solver outputs, while the final grader achieved 0.666 on final-solver outputs. These results provide additional evidence that grader adaptation improves evaluation quality across evolving solver behaviors and highlight the limitations of keeping the grader fixed.

Q4: Adaptive task selection.

We would like to clarify that the score difference in DUET is not intended to estimate true solver learning progress. As discussed in Section 4.1, it measures changes in task evaluations arising from either solver or grader updates. Both types of changes can provide useful training signals in our co-evolution setting. Changes in solver behavior may expose remaining execution weaknesses, while changes in grader judgments may reveal evaluation practices that need further refinement. Revisiting such tasks allows the updater to examine the underlying execution evidence and identify potential improvements to either agent. We therefore use absolute score changes as a heuristic for identifying potentially informative tasks.

In addition to prioritizing tasks with larger score changes, DUET also explores tasks that have been visited less frequently. This balances exploitation and exploration, ensuring that tasks with small score changes can still be revisited.


We thank the reviewer again for the helpful comments. If our responses have addressed your concerns, we would appreciate it if you could consider raising your rating. We would also be happy to address any further questions you may have.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer qUe6,

Thank you again for your thoughtful and constructive feedback. We have carefully responded to your concerns in our rebuttal and revised the paper accordingly. In particular, we would like to highlight two additional experiments:

  1. Cross-checkpoint evaluation: We evaluated grader checkpoints against outputs from different solver stages using independent benchmark success checks. The results show that grader AUROC improves from 0.707 to 0.771, while the initial grader becomes less reliable on outputs from the evolved solver. These findings provide additional evidence for the importance and stability of grader co-evolution.

  2. Validation-free ablation: We extended the fixed-budget ablation to all four benchmarks. The results show that DUET achieves comparable performance without validation-based checkpoint selection or early stopping, addressing concerns about dependence on small validation sets.

As the discussion period is ending soon, we would appreciate your feedback on our responses. We hope the additional results and clarifications have addressed your concerns, and we would be grateful if you could consider updating your initial rating accordingly. We would also be happy to address any further questions.

Thank you again for your time and consideration!

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.