acceptodds
Under review as a conference paper at ICLR 2027

Verifier–Harness Co-Evolution for Training-Free Self-Improvement on Open-Ended Tasks

Abstract

Training-free self-improvement lets an agent use feedback to revise the response it is about to return and to update the harness state that shapes how it handles later tasks. On open-ended tasks that feedback rarely comes from reference answers or executable tests, so the agent has to rely on a model-based verifier. When the verifier is wrong, the current response can get worse, and misleading experience can persist across the task stream. A verifier that never changes also grows stale as the agent's responses improve and task demands shift. We introduce Unified Co-Evolution (UCoE), a training-free method that evolves an executable verifier program together with the agent's harness state while all model parameters stay fixed. UCoE synthesizes candidate verifier programs and tests them on response contrasts, pairs of responses that differ in one declared property and have a known preferred side. It selects the candidate whose discrimination on its weakest contrast family is highest, a quantity we call driving capacity. The selected verifier diagnoses one weakness in each draft, guides a single repair, decides which response is served, and determines whether the accepted repair may enter the harness state. As tasks complete, their diagnostics seed new candidate verifiers, which must beat the current verifier on the same contrasts before adoption. Across six open-ended benchmarks, UCoE averages preference against matched direct generation with Qwen3-8B and with Qwen3.8-27B, margins of and percentage points. On three verifiable benchmarks, the same method raises the mean from for direct generation to .

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Training-free agent self-improvement, verifier synthesis, verifier evolution, test-time adaptation, continual learning, open-ended generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LFTnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new training-free self-improvement method for a model-based verifier that co-evolves with the agent's harness state by focusing on improving the weakest criteria family in each iteration. At scheduled checkpoints, new candidate verifiers must outperform the previous ones on response contrasts before being adopted. The experimental results show that the proposed UCoE method outperforms previous feedback methods and the direct generation baseline across six open-ended benchmarks on two sizes of Qwen models (8B and 27B), as well as on three verifiable benchmarks with the 8B model.

Critical strengths
  • The proposed method outperforms previous methods and the direct generation baseline by a large margin on diverse open-ended tasks across two model sizes (8B and 27B).
  • The proposed method also transfers to other domains, such as the codex agent harness on spreadsheet workflows.
  • Good ablation studies verify the usefulness of each component.
Critical weaknesses
  • The performance gain seems to stem mainly from standard single-turn revision guided by verifier feedback rather than the co-evolutionary mechanism itself. Freezing the verifier drops preference by only 1.7%, indicating that the complex verifier synthesis and update pipeline contributes relatively little over a fixed verifier baseline.
  • Larger models are not evaluated on the verifiable benchmarks, and the performance gains are relatively marginal on these tasks (e.g., Hard-MATH and bigcodebench-hard).
  • The method lacks a compute comparison against the baselines in the main paper; leaving it unclear how much of the advantage stems from higher inference compute versus the proposed framework itself.
Questions for authors

Does UCoE generalize to distinct open-weight model families?

3 comments
AuthorsOpenReview ↗

Response to Reviewer LFTn (1/2)

We thank you for the positive assessment of the results across tasks and model sizes, and for recognizing the transfer to an agent harness and the value of the ablation studies. We address each of your points below.

W1: On the incremental contribution of co-evolution.

Thank you for asking us to isolate the contribution of continued verifier evolution. We extended the Fixed-Verifier UCoE comparison to all six open-ended and three verifiable benchmarks with Qwen3-8B.

Fixed-Verifier UCoE retains the initial verifier, the same revision procedure, and ongoing memory updates. Both variants use the same initial verifier synthesis and contrast-based selection. Full UCoE can replace its verifier at later checkpoints.

For open-ended tasks, Qwen3.5-397B-A17B evaluates responses against Direct Generation in both presentation orders, with method identities hidden and half credit for ties. Scores below are preference percentages.

BenchmarkFull UCoE vs DirectFixed-Verifier UCoE vs Direct
Arena-Hard58.3056.60
WritingPrompts62.7962.79
WildBench57.4254.44
WritingBench56.9055.82
MultiChallenge57.4256.80
RefineBench61.1556.95

Full UCoE scores higher on five of the six benchmarks, with gains of 2.98 percentage points on WildBench and 4.20 on RefineBench. Fixed-Verifier UCoE already improves on Direct Generation on all six benchmarks, so continued evolution adds gains beyond those provided by the initial verifier and ongoing memory updates. WritingPrompts is unchanged because neither candidate met the update criteria, so Full UCoE retained its initial verifier.

The Arena-Hard Full UCoE score includes relations. With matched requirements-only settings, Full UCoE scores 58.10% versus 56.60% for Fixed-Verifier UCoE against Direct Generation. Matched direct comparisons give Full UCoE 51.30% preference on Arena-Hard and 53.12% on WildBench. The WildBench difference is significant with p≈10⁻⁶, while the Arena-Hard difference is not, with p=.23.

For verifiable tasks, we retain the paper's benchmark-specific configurations, including select-only on IFBench. Scores are accuracy percentages for IFBench and Hard-MATH, and Pass@1 percentages for BigCodeBench-Hard.

BenchmarkFull UCoEFixed-Verifier UCoE
IFBench34.6733.33
Hard-MATH59.1659.16
BigCodeBench-Hard21.6221.62

Evolution adds two correct answers on IFBench. Hard-MATH retains the initial verifier because no candidate met the update criteria. On BigCodeBench-Hard, a verifier update changed the responses while leaving aggregate Pass@1 unchanged. Together, these results show that continued evolution can improve on the initial verifier and memory loop, with larger observed benefits on the open-ended benchmarks.

W2: On larger actors and the verifiable-task results.

Thank you for suggesting evaluation with a larger actor on verifiable tasks. We evaluated Qwen3.8-27B on all three benchmarks using the paper's task sets, verifier backbone, output limits, and benchmark-specific configurations, including select-only on IFBench. The LLM as a Judge baseline uses binary judgments to update memory without a separate revision. Scores below are official checker results as percentages.

Method / Qwen3.8-27BIFBench loose prompt accuracyHard-MATH accuracyBigCodeBench-Hard Pass@1
Direct Generation42.0072.5129.05
LLM as a Judge38.0076.9631.08
UCoE43.3381.1529.73

UCoE scores above Direct Generation on all three benchmarks, with gains of 1.33, 8.64, and 0.68 percentage points. It has the highest accuracy on IFBench and Hard-MATH, while LLM as a Judge scores higher on BigCodeBench-Hard. The largest gain is on Hard-MATH under the original 4,096-token per-response limit, where UCoE solves 33 additional tasks while retaining all of Direct Generation's correct answers.

These results show that a larger actor can still benefit from verifier-guided feedback, particularly on mathematical reasoning. The smaller gains on IFBench and BigCodeBench-Hard indicate that the benefit varies by task. Reference answers and checker results are used only for final evaluation.

AuthorsOpenReview ↗

Response to Reviewer LFTn (2/2)

W3: On inference cost and a compute-matched comparison.

Thank you for asking for a comparison under a common compute budget. UCoE's all-in cost includes relation discovery, initialization, checkpoint updates, and online calls. It averages USD 0.0142 per task versus USD 0.0050 to 0.0063 for the simpler feedback baselines. We will bring these costs into the main paper.

We gave LLM as a Judge and RaR additional revision budgets for best-of-K revision, with budget ceilings matched to UCoE's all-in method cost. With Qwen3-8B, LLM as a Judge generates up to six revision candidates and RaR up to four, allowing each baseline to choose from more candidates. Each baseline's candidates start from the same draft and fixed initial feedback, and each baseline retains its scoring, acceptance, and memory rules from the paper.

The table reports preference percentages against Direct Generation.

MethodArena-Hard preference vs DirectRefineBench preference vs Direct
LLM as a Judge50.6051.98
LLM as a Judge, best-of-654.0052.88
RaR-style feedback51.2051.26
RaR-style feedback, best-of-453.3059.53
UCoE58.3061.15

Both baselines improve with more revision candidates, especially RaR on RefineBench, which rises from 51.26% to 59.53%. UCoE retains higher observed preference on both benchmarks at 58.30% and 61.15%. This suggests that the feedback used to guide and select revisions matters alongside the revision budget.

Q1: On distinct open-weight model families.

Thank you for asking whether UCoE generalizes to other open-weight model families. We evaluated NVIDIA Nemotron-3.5-Lightning-30B-A3B as a non-Qwen actor on Arena-Hard and RefineBench. Direct Generation, LLM as a Judge, and UCoE use the same actor and decoding settings. UCoE initializes its verifier on Nemotron calibration responses and starts with empty memory.

The verifier remains Qwen3.6-35B-A3B, and Qwen3.5-397B-A17B evaluates responses using the original blinded, both-order protocol with half credit for ties. Arena-Hard uses mined relations. RefineBench uses the relation-free configuration because its mined relations did not pass calibration. Scores below are preference percentages.

BenchmarkTasksLLM as a Judge vs DirectUCoE vs DirectUCoE vs LLM as a Judge
Arena-Hard50054.2058.7053.10
RefineBench, without relations27855.0460.2553.78

UCoE reaches 58.70% preference against Direct Generation on Arena-Hard and 60.25% on RefineBench. Its direct comparisons with LLM as a Judge also favor UCoE.

These results show that UCoE's verifier-guided feedback can improve responses from a non-Qwen actor while retaining the same verifier backbone. The RefineBench result further shows that this benefit extends to a setting without mined relations.

Dear Reviewer LFTn,

As the discussion period draws to a close, we would greatly appreciate your feedback on our responses and revised manuscript. We have expanded the fixed-verifier comparisons, added evaluations on another model family, and provided budget-matched comparisons to address your concerns.

Please let us know whether these additions address your questions. If any concerns remain, we would be happy to clarify them before the discussion ends. We would also appreciate it if you could reconsider your assessment after reviewing these updates.

Thank you again for your time and thoughtful feedback!

Sincerely, The Authors

Reviewer_e8DnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UCoE, a training-free method that iteratively updates the actor harness and the verifier together for open-ended tasks. It proposes an interesting idea of driving capacity to select evaluators by their weakest category during the harness and verifier updates. The experiments show that, when we run UCoE on a specific benchmark in a stream of open-ended tasks, UCoE often achieves higher performance than other training-free baselines.

Critical strengths

This paper extends the idea of co-evolving actor and verifier to open-ended tasks and demonstrate accuracy improvements on specific benchmarks with evidence showing that the verifier updates improve feedback quality.

Critical weaknesses

The experiments do not sufficiently isolate and validate the proposed mechanism. Please see my questions for authors.

Questions for authors
  1. In the ablation study (Table 3), fixing the verifier only decreases the performance of UCoE from 58.3% to 56.6%, which still outperforms all other baselines. Therefore, the benefits of co-evolving the verifier and actor harness is unclear. It would be more convincing if UCoE with a fixed verifier is evaluated as a separate baseline in Table 1 and 2.

  2. The verifier iteration is supplied with a set of relation RR discovered with a stronger model than the evaluated actor model. Do the baseline methods also receive RR as an input? What's the impact of RR on the performance of UCoE? It seems removing RR is not included as an alternative configuration in Table 3.

  3. It seems to me the benefits of UCoE is unclear or small on verifiable tasks, as shown in Table 2. However, the analysis states that "UCoE also improves verifiable tasks". Can you establish statistical significance to support this statement?

  4. To support the claim that "UCoE transfers to a codex agent harness", the experiment use a small SpreadsheetBench-2 (n=75, 23, 30, and 22 for each task category). Can you establish confidence intervals or perform inference tests to show the benefits of UCoE over baselines with statistical significance?

  5. By construction, UCoE improves harness and verifier, and solves tasks simultaneously. This requires practitioners to run UCoE online when resolving incoming queries or tasks. How quickly do verifier and harness adaptation yield measurable benefits? How do UCoE perform on benchmarks that are unseen during evolution? This would clarify adaptation efficiency and whether the acquired improvements generalize beyond the adaptation stream.

  6. In terms of token costs, how do UCoE compare to the evaluated baseline methods? If all methods are given the same costs (e.g., increasing or decreasing the revision budgets for baseline methods), how do UCoE compare to baselines?

  7. From my understanding, SpyRL is a training algorithm that changes the model parameter. However, "Baselines" paragraph of Section 4 states that "we adapt all four feedback baselines to the same fixed-model, one-revision protocol." How do you evaluate a training algorithm with a fixed-model protocol? If only a part of SpyRL is evaluated, naming the method SpyRL is unfair, I think.

8 comments
AuthorsOpenReview ↗

Response to Reviewer e8Dn (1/4)

We thank you for the careful review and for recognizing the extension of co-evolution to open-ended tasks and the improvements in feedback quality. We address each of your questions below.

W1 & Q1: On isolating verifier evolution with a fixed-verifier baseline.

Thank you for suggesting a fixed-verifier baseline to measure the contribution of verifier evolution. We added this comparison on all six open-ended and three verifiable benchmarks with Qwen3-8B.

Fixed-Verifier UCoE retains the initial verifier, the same revision procedure, and ongoing memory updates. Both variants use the same initial verifier synthesis and contrast-based selection. Full UCoE can replace its verifier at later checkpoints.

For open-ended tasks, Qwen3.5-397B-A17B evaluates responses against Direct Generation in both presentation orders, with method identities hidden and half credit for ties. Scores below are preference percentages.

BenchmarkFull UCoE vs DirectFixed-Verifier UCoE vs Direct
Arena-Hard58.3056.60
WritingPrompts62.7962.79
WildBench57.4254.44
WritingBench56.9055.82
MultiChallenge57.4256.80
RefineBench61.1556.95

Full UCoE scores higher on five of the six benchmarks, with gains of 2.98 percentage points on WildBench and 4.20 on RefineBench. Fixed-Verifier UCoE already improves on Direct Generation on all six benchmarks, so continued evolution adds gains beyond those provided by the initial verifier and ongoing memory updates. WritingPrompts is unchanged because neither candidate met the update criteria, so Full UCoE retained its initial verifier. The Arena-Hard Full UCoE score includes relations. With matched requirements-only settings, Full UCoE scores 58.10% versus 56.60% for Fixed-Verifier UCoE.

For verifiable tasks, we use the same benchmark-specific configurations, including select-only on IFBench. Scores are accuracy percentages for IFBench and Hard-MATH, and Pass@1 percentages for BigCodeBench-Hard.

BenchmarkFull UCoEFixed-Verifier UCoE
IFBench34.6733.33
Hard-MATH59.1659.16
BigCodeBench-Hard21.6221.62

On IFBench, evolution adds two correct answers, increasing accuracy from 33.33% to 34.67%. On Hard-MATH, Full UCoE kept the initial verifier because the proposed candidates did not meet the update criteria. On BigCodeBench-Hard, a verifier update changed the responses, with both variants solving 32 of 148 tasks. Overall, verifier evolution improves performance on most open-ended benchmarks, with smaller effects on the verifiable tasks.

Q2: On the stronger model and the relation inventory R\mathcal{R}.

Thank you for asking about the stronger model's contribution and the role of the relation inventory R\mathcal{R}. The stronger model supplies development responses. The verifier model extracts recurring differences from response pairs presented in random order with system identities hidden. We fix R\mathcal{R} before deployment. The feedback baselines do not receive these mined relations.

To measure their contribution, we compared Full UCoE with and without R\mathcal{R} on Arena-Hard and RefineBench using Qwen3-8B. Removing R\mathcal{R} removes both the mined criteria and their contrast families. Public requirements, general contrast families, revision rules, and memory rules remain unchanged. Scores below are preference percentages against Direct Generation.

BenchmarkFull UCoE with R\mathcal{R}Full UCoE without R\mathcal{R}
Arena-Hard58.3058.10
RefineBench61.1557.73

Full UCoE remains above Direct Generation without R\mathcal{R} on both benchmarks, showing that public requirements and general contrasts already provide useful feedback for improvement. Adding R\mathcal{R} increases preference by 0.20 percentage points on Arena-Hard and 3.42 on RefineBench. Direct comparisons between the two configurations also favor Full UCoE with R\mathcal{R}, at 51.20% and 54.14%, respectively. These results suggest that mined relations and their associated contrasts can provide additional task-specific guidance, especially on RefineBench.

AuthorsOpenReview ↗

Response to Reviewer e8Dn (2/4)

Q3: On the size and statistical support of the verifiable-task gains.

Thank you for asking about the size and statistical support of the verifiable-task gains. We analyzed paired Qwen3-8B outcomes using two-sided exact McNemar tests, with Holm correction across the three benchmarks. The table reports accuracy on IFBench and Hard-MATH and Pass@1 on BigCodeBench-Hard. Scores are percentages and differences are in percentage points. UCoE-only and Direct-only count tasks solved by one method but not the other.

BenchmarkTasksDirectUCoEDifferenceUCoE-only / Direct-onlyHolm-adjusted p-value
IFBench15030.0034.67+4.678 / 10.0781
Hard-MATH38256.5459.16+2.6210 / 00.0059
BigCodeBench-Hard14821.6221.620.001 / 11.0000

UCoE improves IFBench by 4.67 percentage points and Hard-MATH by 2.62, while matching Direct Generation on BigCodeBench-Hard. The Hard-MATH gain remains significant after correction, with adjusted p=.0059. IFBench improves in observed accuracy, though its adjusted p=.0781 does not reach the 0.05 threshold. IFBench uses the official loose prompt-level accuracy, and we will correct the appendix's metric label.

The paired outcomes show that UCoE solves additional tasks on IFBench and Hard-MATH while retaining most of Direct Generation's correct answers. These gains extend the benefit of verifier-guided feedback beyond open-ended tasks to these verifiable benchmarks, with reference answers and checker results used only for final evaluation.

Q4: On the size and uncertainty of the agent-harness study.

Thank you for asking about the statistical support for the agent-harness results. We computed paired two-sided exact McNemar tests on the results reported in the paper and obtained p=.2891 against both Direct Generation and the fixed holistic judge.

UCoE shows an observed improvement over both baselines, although the differences are not statistically significant. We will evaluate more tasks and run additional experiments across model and harness settings to test whether these gains persist.

AuthorsOpenReview ↗

Response to Reviewer e8Dn (3/4)

Q5: On adaptation speed and transfer beyond the evolution stream.

Thank you for asking about adaptation speed and transfer to new tasks. UCoE can use its initialized verifier to revise responses from the first task. We examined when the benefits of verifier updates appear and whether the resulting verifier and memory remain useful on new benchmarks without further adaptation.

To examine adaptation speed, we analyzed the Qwen3-8B runs after task 128 on Arena-Hard with relations and task 256 on WildBench without relations. From the same checkpoint state, Full UCoE continues verifier evolution, while the control retains the checkpoint verifier. Both continue memory updates.

We grouped the existing post-checkpoint comparisons into the first 64 tasks, the next 64, and the remaining tasks. Scores are preference percentages for Full UCoE over the control, using blinded judgments in both presentation orders, with half credit for ties and 50% indicating parity. Wins and losses are counted from Full UCoE's side.

BenchmarkPost-checkpoint segmentTask numbersFull UCoE preference, %Wins / losses / ties
Arena-HardFirst 64 tasks129 to 19247.669 / 12 / 43
Arena-HardNext 64 tasks193 to 25646.884 / 8 / 52
Arena-HardRemaining tasks257 to 50053.6936 / 18 / 190
WildBenchFirst 64 tasks257 to 32059.3812 / 0 / 52
WildBenchNext 64 tasks321 to 38451.567 / 5 / 52
WildBenchRemaining tasks385 to 102453.9196 / 46 / 498

On WildBench, Full UCoE reaches 59.38% preference in the first 64 tasks after the update, with 12 wins and no losses, and remains above parity in both later segments. On Arena-Hard, the benefit appears later, with preference reaching 53.69% in the remaining tasks after trailing the control in the first two segments. Verifier updates can therefore help soon after adoption, while the observed timing varies across streams. Arena-Hard's final segment follows the transition from coding to mathematics, and WildBench's final segment includes a second verifier update.

To test transfer beyond the evolution stream, we froze both the final verifier and harness memory after 500 Arena-Hard tasks or 1,024 WildBench tasks, then applied each state to 100 tasks from a benchmark unseen during evolution. Both transfers to RefineBench use the same sample. The transferred state guides one revision of each matched Direct Generation draft. Qwen3.5-397B-A17B judges each pair in both presentation orders. Scores are preference percentages against Direct Generation, with half credit for ties and 50% indicating parity.

Source → targetTarget tasksUCoE vs Direct preference
WildBench → RefineBench10057.50
Arena-Hard → RefineBench10057.50
Arena-Hard → WildBench10058.00
WildBench → WritingPrompts10058.00

UCoE reaches 57.5 to 58.0% preference across all four transfer settings. These results show that the verifier and retained experience can be reused to improve responses on new benchmarks without further adaptation.

Q6: On token costs and equal-budget baselines.

Thank you for asking about token costs and comparisons under a common budget. UCoE's reported cost includes relation discovery, initialization, checkpoint updates, and online calls, and is 2.2 to 2.8 times that of the three simpler feedback baselines.

We gave LLM as a Judge and RaR additional revision budgets for best-of-K revision, with budget ceilings matched to UCoE's all-in method cost. With Qwen3-8B, LLM as a Judge generates up to six revision candidates and RaR up to four, allowing each baseline to choose from more candidates. Each baseline's candidates start from the same draft and fixed initial feedback, and each baseline retains its scoring, acceptance, and memory rules from the paper.

The table reports preference percentages against Direct Generation.

MethodArena-Hard preference vs DirectRefineBench preference vs Direct
LLM as a Judge50.6051.98
LLM as a Judge, best-of-654.0052.88
RaR-style feedback51.2051.26
RaR-style feedback, best-of-453.3059.53
UCoE58.3061.15

Both baselines improve with more revision candidates, especially RaR on RefineBench, which rises from 51.26% to 59.53%. UCoE retains higher observed preference on both benchmarks at 58.30% and 61.15%. This suggests that the feedback used to guide and select revisions matters alongside the revision budget.

Reviewer_tba7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UCoE, a training-free framework that jointly updates an executable verifier and an agent’s procedural memory. It selects verifiers using their weakest-family discrimination on constructed response contrasts, then uses them to guide revisions and memory updates. Across six open-ended benchmarks, UCoE achieves average preference scores of 59.00% and 57.19% against direct generation for two actors, with additional validation on verifiable tasks and other model and harness settings.

Critical strengths
  1. The paper addresses an important problem with a coherent and inspectable framework. Incorrect feedback can damage both the current response and future behavior through retained experience. UCoE addresses these coupled effects by adapting evaluation criteria, reward aggregation, and procedural memory within a single workflow. The executable verifier representation and detailed examples make its updates and failure modes concrete and inspectable.

  2. The empirical evidence supports the practical value of the overall system. Improvements span multiple tasks and actors and are supported by more than the primary preference judge: native benchmark metrics, verifiable tasks, and an independent WritingBench Critic audit offer complementary evidence.

Critical weaknesses
  1. The validity of the constructed preferences remains insufficiently established. The method assumes that targeted edits degrade a particular property and therefore warrant a lower aggregate score. However, deletions or insertions can affect several dimensions simultaneously, and a local degradation need not imply lower overall quality. Improvements in driving capacity may therefore partly reflect sensitivity to the construction procedure rather than better judgment of natural responses. Source-disjoint selection and held-out splits reduce sample-level overfitting but do not eliminate biases shared by the editing mechanism.
  2. The scope of the comparative claims should be more precise. The baselines are adapted to a common fixed-model, single-revision interface rather than evaluated as complete implementations of their original methods. This is useful for controlling the workflow, but it limits the broader conclusions that can be drawn about superiority over those methods.
Questions for authors
  1. Have you independently audited the validity of the constructed preferences, for example using a separate LLM judge? If so, what proportion of the presumed preference directions were confirmed?
  2. Across verifier candidates or update checkpoints, do improvements in driving capacity consistently predict better selection outcomes on natural revisions?
  3. Do the reported costs include initialization and verifier updates? Are results available for a strong pairwise-judging baseline under a comparable total budget?
4 comments
AuthorsOpenReview ↗

Response to Reviewer tba7 (1/2)

We thank you for the thoughtful assessment and for recognizing the importance of the problem, the inspectability of the framework, and its practical value across tasks and actors. We address each of your points below.

W1 & Q1: On the validity of constructed preferences.

Thank you for suggesting an independent audit of contrast validity. We audited 600 contrasts from the six Qwen3.8-27B benchmark runs to test both the intended quality change and the overall preference direction.

We sampled 100 pairs per benchmark, stratified by edit family. GPT-5.6-luna with medium reasoning effort served as an independent judge, separate from verifier construction. Responses were presented in random A/B order with the construction direction hidden. One pass assessed overall preference. A separate pass received the target-property description and checked whether that property deteriorated and whether other important properties changed.

Following the constructions described in the paper's contrast-construction appendix, we report predefined general-task edits and predefined narrative edits separately. The latter are used for story generation. Relation-derived edits target properties identified through relation discovery. All rates below are percentages of the full audited group.

Contrast groupAudited pairsTarget property worsenedBase preferredEdited preferredTie
Predefined general-task edits34390.197.40.91.7
Predefined narrative edits8088.896.30.03.8
Relation-derived edits17752.070.111.318.6
All contrasts60078.789.23.87.0

The independent judge prefers the base response in 89.2% of pairs and confirms deterioration in the targeted property in 78.7%. This agreement supports using the constructed preference direction to test whether verifiers distinguish better from worse responses. Agreement is strongest for the predefined general-task and narrative edits, whose combined base preference reaches 97.2%. For relation-derived edits, target-property deterioration is confirmed in 52.0% of pairs, while the base response is preferred in 70.1%. Their overall preference direction is therefore supported more often than the specific targeted change.

The judge also identified changes to other important properties in 36.0% of pairs, indicating that some edits affect multiple quality dimensions.

Q2: On whether driving capacity predicts selection quality on natural revisions.

Thank you for asking whether improvements in driving capacity translate into better selection on natural revisions. We evaluated 24 comparisons between candidate and incumbent verifiers at archived checkpoints across six Qwen3-8B benchmark runs.

For each benchmark, the verifiers selected between the same 100 natural draft and revision pairs using their original scoring programs and acceptance margin. The pairs include both accepted and rejected revisions. Independent Qwen3.5-397B-A17B judgments in both presentation orders identify the preferred response. Selection accuracy measures how often a verifier chooses that response, with half credit for tied pairs.

The table shows all five candidates with increased signed driving capacity on the original contrast selection split. Base is the incumbent verifier at each checkpoint, and New is the candidate evaluated against it. Checkpoints are listed by task number. The final column reports the change in selection accuracy on natural revisions, in percentage points.

Benchmark / checkpointBase capacityNew capacityCapacity gainSelection gain, pp
Arena-Hard / 128−0.04170.0000+0.0417−1.0
WildBench / 2560.06250.0833+0.0208+5.0
WildBench / 5120.14580.2083+0.0625+2.0
WritingPrompts / 5120.11960.1250+0.0054+3.0
WritingPrompts / 10240.06250.0833+0.0208+2.0

Four of the five candidates with higher driving capacity also improve selection on natural revisions, with a mean gain of 2.2 percentage points across all five. These gains on matched response pairs support using driving capacity measured on constructed contrasts to identify verifiers that make better choices on natural revisions.

W2: On the scope of the baseline comparison.

Thank you for asking us to clarify the baseline comparison. We adapt feedback mechanisms from related methods to serve as verifiers for open-ended tasks in our training-free setting. These include RaR's rubric scoring, DREAM's reward programs, and SpyRL's task-transformation feedback. The adapters share the same actor, single-revision opportunity, and memory-update rules.

We will explicitly label these baselines as training-free adaptations of their feedback mechanisms in the experimental setup and baseline implementation appendix.

AuthorsOpenReview ↗

Response to Reviewer tba7 (2/2)

Q3: On initialization costs and budget-matched pairwise judging.

Thank you for asking about initialization costs and comparisons under a comparable total budget. Yes, the reported average of USD 0.0142 per served task includes initialization and verifier updates. It covers relation discovery, verifier synthesis, contrast construction and scoring, checkpoint updates, and all online generation, revision, selection, and memory calls. Setup and update costs are amortized over the served task stream. Final evaluation is excluded for all methods. We will make this coverage explicit in the revision.

We increased the budgets of LLM as a Judge and RaR for best-of-K revision, setting their budget ceilings to UCoE's full method-side cost, including verifier synthesis and updates. With Qwen3-8B, LLM as a Judge generates up to six revision candidates and RaR up to four. Each candidate starts from the same draft and initial feedback. Each baseline retains its scoring, acceptance, and memory rules from the paper.

The table reports preference percentages against Direct Generation.

MethodArena-Hard preferenceRefineBench preference
LLM as a Judge50.6051.98
LLM as a Judge, best-of-654.0052.88
RaR-style feedback51.2051.26
RaR-style feedback, best-of-453.3059.53
UCoE58.3061.15

Both baselines improve with larger revision budgets, especially RaR on RefineBench, which rises from 51.26% to 59.53%. UCoE remains higher on both benchmarks at 58.30% and 61.15%. More revision candidates narrow the gap, but do not close it at the tested budget ceilings. This suggests that the feedback used to guide and select revisions matters alongside the revision budget.

Dear Reviewer tba7,

As the discussion period draws to a close, we would greatly appreciate your feedback on our responses and revised manuscript. We have added an independent audit of the constructed contrasts, budget-matched comparisons, and clearer descriptions of the baseline settings.

Please let us know whether these updates address your concerns. If any points remain unclear or unresolved, we would be happy to provide further clarification.

Thank you again for your time and constructive feedback!

Sincerely, The Authors

Thank you for the additional experiments and clarifications, which largely address my concerns.I maintain my recommendation of weak acceptance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all three reviewers for their careful reading and constructive feedback. We appreciate their recognition of the importance of reliable feedback for open-ended self-improvement, the inspectability of the executable verifier, and the breadth of the empirical evaluation. Their suggestions guided additional experiments on the contribution of verifier evolution, the quality of its feedback, and its use across tasks and models.

During the response period, we extended the evaluation and summarize the main findings here before responding to each review in detail.

  • Contribution of verifier evolution. We extended the fixed-verifier comparison to all six open-ended and three verifiable benchmarks, retaining initial contrast-based verifier selection and ongoing memory updates in the control. Full UCoE has higher preference against Direct Generation on five open-ended benchmarks, with gains over the fixed-verifier scores of 2.98 percentage points on WildBench and 4.20 on RefineBench. Matched direct comparisons also favor Full UCoE on WildBench and Arena-Hard. UCoE also improves over Direct Generation both with and without mined relations on Arena-Hard and RefineBench.
  • Validity of contrasts and driving capacity. An independent audit of 600 contrasts confirms the overall preference direction in 89.2% of pairs and deterioration in the targeted property in 78.7%. Agreement on the overall preference direction reaches 97.2% for predefined general-task and narrative edits. On fixed natural draft-revision pairs, four of the five candidates with increased driving capacity also improve response selection, with a mean gain of 2.2 percentage points across the five.
  • Performance under expanded baseline budgets. The reported cost includes initialization, verifier updates, and online inference. We expanded LLM as a Judge to best-of-6 revision and RaR to best-of-4, with budget ceilings matched to UCoE's total method cost. UCoE achieves higher observed preference against Direct Generation than both expanded baselines on Arena-Hard and RefineBench. We will also label RaR, DREAM, and SpyRL explicitly as training-free adaptations of their feedback mechanisms.
  • Transfer beyond the adaptation stream. We froze both the verifier and harness memory after evolution and evaluated four transfers to benchmarks unseen during evolution. Frozen UCoE achieves 57.5 to 58.0% preference against Direct Generation without updating either component on the target tasks.
  • Larger actors and another open-weight model family. We completed all three verifiable benchmarks with Qwen3.8-27B. Under the original output limits, UCoE improves over Direct Generation on all three, with the largest gain on Hard-MATH, from 72.51% to 81.15%. With the non-Qwen actor NVIDIA Nemotron-3.5-Lightning-30B-A3B, UCoE reaches 58.70% preference against Direct Generation on Arena-Hard and 60.25% on RefineBench. The RefineBench result further demonstrates improvement with the relation-free configuration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.