acceptodds
Under review as a conference paper at ICLR 2027

How Much Reward Does Self-Evolution Need? Pairwise Validation Without Task-Reward Feedback

Abstract

Self-evolving agents rely on task rewards to guide revisions, yet outside benchmarks labeled examples may be scarce, expert feedback costly, and automated evaluators incomplete. Can agents improve their prompts without reward feedback at every evolution step? We introduce PairEvolve, which removes task scores and gold answers from mutation and uses a frozen LLM to compare the original and revised outputs for acceptance. Two variants address different levels of reward access. PairEvolve-Val retains validation-score parent selection when a labeled set is available to choose which prompt to revise next. PairEvolve-Elo replaces that feedback with Elo ratings—numerical ratings updated from pairwise outcomes—so both acceptance and parent selection can proceed without task rewards. Both retain labeled final selection in our evaluation. Across four prompt tasks and two agent models, Val and Elo improve mean test scores over no evolution in five and six of eight settings, respectively, retaining full-reward performance in some settings and trading performance for reduced reward access in others. Validator studies identify how comparison format, thinking, and the treatment of undecided comparisons affect this feedback. Integrations with two program-evolution engines extend the comparison interface to code while retaining execution checks and proxy scores. Together, these results show how pairwise feedback can support evolution with fewer reward-dependent decisions.

Primary area:
optimization
Keywords:
Self-evolving agents, Pairwise validation, Prompt optimization, LLM judges, Reward-free evolution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RHrpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PairEvolve, a framework for reducing task-reward dependence in self-evolving agents. For prompt evolution, PairEvolve removes task scores and gold answers from mutation feedback and uses pairwise LLM judgments to decide whether to accept a revised prompt. Two variants are considered: PairEvolve-Val retains validation-based parent selection, while PairEvolve-Elo uses Elo ratings derived from pairwise comparisons. Experiments on four prompt optimization tasks show improvements over the initial prompts in several settings, although performance relative to full-reward evolution varies across tasks. The paper further investigates validator biases, acceptance rules, and extensions to program evolution.

Critical strengths
  1. The paper provides a useful decomposition of reward dependence into mutation, acceptance, and parent selection. The distinction between PairEvolve-Val and PairEvolve-Elo is clearly motivated, and the experiments demonstrate that useful prompts can be obtained without task-reward feedback at every evolution step.

  2. The validator analysis is a valuable part of the paper. In particular, the investigation of position bias and tie handling reveals an important failure mode: pairwise comparisons can provide little effective selection pressure when ambiguous judgments are frequently accepted.

Critical weaknesses
  1. The benefit of pairwise acceptance itself is not convincingly isolated. In Table 8, the no-validator baseline performs comparably to or numerically better than symmetric pairwise acceptance on both tasks. Although this experiment uses a different configuration from the main study, it raises the question of whether the gains in Table 1 primarily come from reward-blind mutation and downstream candidate selection rather than the proposed validator. A matched control for the main configurations is missing.

  2. The empirical evidence is somewhat limited by the evaluation protocol. The test sets contain only 30--50 examples, and several tasks use modified or permissive metrics, including partial constraint satisfaction on IFBench and substring-based label extraction on HoVer. Together with the substantial variability in some results, this makes the robustness of the reported gains uncertain. The program-evolution results are also based on a single run per configuration.

Questions for authors
  1. Could the authors include a matched no-validator or unconditional-acceptance baseline for the primary experiments? This would help establish whether pairwise judgments provide benefits beyond reward-blind mutation and downstream selection.

  2. How sensitive are the main results to stricter evaluation metrics, particularly for IFBench and HoVer? Would the relative rankings remain similar?

  3. Since task evaluators still execute before their scores are masked (Appendix A.2), could the authors quantify the reduction in labeled feedback requirements and the additional inference cost introduced by pairwise judging? This would clarify the practical benefit of the proposed reward-substitution strategy.

1 comment
AuthorsOpenReview ↗

Response to Reviewer RHrp: Breadth, reward access and scoring verification

Useful evolution with less task-reward feedback

PairEvolve shows that useful evolution does not require task rewards at every search decision.

MethodTask-reward feedback during search
Full rewardGuides revision, acceptance and parent choice
PairEvolve-ValChooses which prompt to revise next
PairEvolve-EloNone; pairwise ratings choose the next parent

Both PairEvolve variants compare the parent and revised prompt’s answers to decide whether to accept the revision. After search, all methods choose the highest-validation-score prompt and test it on separate questions.

For example, GPT-5.4-mini HotpotQA Val/Elo reach 84.0/79.8 test F1, versus 29.2 initially and 77.1 with full reward. Both improve over the initial prompt in all three seeds; Val exceeds full reward in every seed.

1. W1 / Q1 — What do comparisons add to search?

LLM comparisons provide informative feedback. Table 2a uses constructed pairs: a control answer and a flawed alternative. Evaluators see the task and answers, without references or flaw annotations. Pairwise compares answers together; pointwise scores them separately, favoring the higher score.

Evaluating LLMTaskPairwisePointwise
DeepSeek V4 FlashHotpotQA94.7±1.190.8±1.6
DeepSeek V4 FlashHoVer89.8±1.684.2±1.8
Qwen3.8-27BHotpotQA97.8±0.896.5±1.1
Qwen3.8-27BHoVer94.6±1.192.8±1.4

Agreement with the designated better answer (%) ± standard error; thinking enabled, same valid pairs, half credit for ties. Pairwise is 39.8–47.8 points above chance (50%).

GPT-5.4-mini HotpotQA Elo’s selected prompts first received full-validation evaluation at iterations 327, 278 and 97 of 350: useful revisions continue to emerge late in search.

2. W2 — How broad and repeatable is the evidence?

96 runs span four tasks and two models: HotpotQA, HoVer, IFBench and PUPA; Qwen3-8B/GPT-5.4-mini use five/three seeds per method/task, with 30–50 test questions per task.

Mini HotpotQA Val exceeds full reward in all three seeds: +10.65, +5.86 and +4.17 F1. On mini IFBench, Val matches full reward’s mean 90.4 and improves over the initial prompt in every seed under the reported metric.

Beyond GEPA, ADRS and ShinkaEvolve apply comparisons to scheduling, circle packing, signal processing and prime counting: one run per configuration, with code-execution checks. The evidence spans prompt and program evolution.

3. Q3 — What reward access and inference does this require?

Elo uses task scores only for final selection. All methods use 30 validation questions with reference answers to select the final prompt. Full reward and Val also use these scores to choose parents during search. We reduce reward-guided decisions, not validation-set size.

Evolution cost. Comparisons reuse generated answers; Elo reuses judgments for ratings. Appendix F records 267,680 task calls/783.7M tokens and 26,100 validator calls/32.2M tokens, pooled over two tasks/four configurations, not per-method costs. Failed-attempt usage is unknown. Evaluators run before score masking, so reduced reward feedback does not establish compute savings.

Test-time cost. Each method deploys its selected prompt directly: no extra judge calls. Prompt/answer lengths can still change token use.

4. W2 / Q2 — How sensitive are the results to evaluation rules?

HoVer: Val accuracy rises from 38.4% to 45.2% under semantic verdict evaluation. To isolate scoring effects, we applied both rules to the same new Qwen3-8B thinking-on answers from fixed historical prompts: all five seeds, 50 common questions, 250 answers per method. Two LLM interpreters, blinded to reference labels, extract the intended verdict for every method.

MethodOriginal ruleSemantic evaluation
Full reward48.453.6
Val38.445.2
Elo42.842.4

Accuracy (%). Semantic evaluation accepts equivalent verdict wording.

IFBench and PUPA: no observed fallback effect in the audited evaluations. We observed zero fallback/default events in 1,144 IFBench constraint checks and 480 new PUPA judgments. All 588/588 auditable IFBench validation scores reproduced; 480/480 PUPA combined scores were unchanged between numeric parsers. PUPA uses fixed historical Qwen prompts with thinking enabled on 30 common questions. Neither new PUPA nor HoVer answers reconstruct historical test outputs.

Strict all-constraints IFBench scoring preserves aggregate method ordering on common retained validation questions across five Qwen seeds. We will correct attribution to THU-KEG IFBench (Peng et al.). Fallback handling leaves the checked IFBench/PUPA scores unchanged.

Reviewer_EZSNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies which uses of task rewards can be removed from artifact evolution while retaining labeled final selection. PairEvolve revises prompts without gold answers or task-score feedback and uses a frozen LLM to compare parent and child outputs. Val retains validation-based parent selection, whereas Elo samples parents using ratings updated from accepted comparisons. Experiments compare both configurations with unevolved prompts and full-reward GEPA on four tasks and two task models. Additional studies examine comparison formats, presentation biases, and tie handling. Program-evolution integrations apply the comparison interface while retaining execution checks and proxy scores.

Critical strengths
  1. The reward-pathway decomposition yields a concrete empirical result beyond simply removing evaluator feedback. On gpt-5.4-mini HotpotQA, Val and Elo reach 84.0 and 79.8, compared with 29.2 without evolution and 77.1 with full reward (Table 1). Both variants improve over the initial prompt across all three seeds (Table 6). This supports the feasibility of generating useful candidates without task rewards guiding local revisions, while clearly reserving labeled evaluation for final selection.

  2. The acceptance diagnostics identify a consequential interaction between order reconciliation and tie handling. Symmetric pairwise acceptance admits 99.6% of HotpotQA proposals and 99.3% of HoVer proposals, with endpoint scores close to unconditional acceptance (Tables 8 and 10). Appendix C.3 separates explicit ties from order disagreements. This provides an actionable finding: obtaining comparative judgments does not ensure that those judgments meaningfully filter an evolutionary search.

Critical weaknesses
  1. The validity of several primary task scores remains unresolved. Appendix A.1 states that unverifiable IFBench dictionary constraints default to passing and failed PUPA quality evaluations default to 0.5, without recorded fallback frequencies. HoVer uses substring matching, and Appendix D.4 documents earlier evolved prompts using INSUFFICIENTLY_SUPPORTED that were credited as SUPPORTED under the same parsing rule used in the primary study. These observations do not establish how much the primary results are affected, but sharing an evaluator across methods does not eliminate differential effects when methods produce different outputs. Consequently, the claimed improvements across tasks require an audit of whether score increases reflect valid task behavior rather than parser or fallback behavior.

  2. The primary study does not establish the added value of informative pairwise judgments over quality-blind filtering followed by labeled selection. In the controlled study, symmetric pairwise acceptance scores 83.0 versus 84.8 for no validator on HotpotQA, and 76.7 versus 77.3 on HoVer (Table 8). That experiment uses different models and acceptance rules from the primary configurations, so it neither validates nor disproves the usefulness of their comparators. Table 7 establishes that the primary configurations reject proposals, but rejection frequency alone does not establish selection quality. A matched primary control is therefore important for determining what the proposed comparator contributes beyond reward-blind mutation and downstream selection.

Questions for authors
  1. Do the improvements persist when the already selected primary checkpoints are rescored using exact HoVer final-label parsing and verified IFBench/PUPA evaluations? Please keep checkpoint selection fixed and report affected-output and evaluator-fallback counts by method.

  2. For a primary Val configuration on HotpotQA, how does its final score compare with random accept/reject decisions matched to its observed acceptance rate, holding the generator, mutation procedure, parent and final selectors, and proposal budget fixed? This would distinguish useful comparative feedback from quality-blind filtering.

2 comments
AuthorsOpenReview ↗

Response to Reviewer EZSN: Comparative feedback and scoring verification

Useful evolution with less task-reward feedback

PairEvolve shows that useful evolution does not require task rewards at every search decision.

MethodTask-reward feedback during search
Full rewardGuides revision, acceptance and parent choice
PairEvolve-ValChooses which prompt to revise next
PairEvolve-EloNone; pairwise ratings choose the next parent

Both PairEvolve variants compare the parent and revised prompt’s answers to decide whether to accept the revision. After search, all methods choose the highest-validation-score prompt and test it on separate questions.

For example, GPT-5.4-mini HotpotQA Val/Elo reach 84.0/79.8 test F1, versus 29.2 initially and 77.1 with full reward. Both improve over the initial prompt in all three seeds; Val exceeds full reward in every seed.

1. W2 / Q2 — What do comparisons contribute beyond quality-blind filtering?

LLM evaluators favor the designated better answer in 89.8–97.8% of clear-error comparisons. Table 2a uses constructed answer pairs: one control and one alternative with an annotated flaw. The evaluating LLM sees the task input and answers, without references or flaw annotations. Pairwise presents both answers together and asks which is better; pointwise scores each separately and favors the higher score.

Evaluating LLMTaskPairwisePointwise
DeepSeek V4 FlashHotpotQA94.7±1.190.8±1.6
DeepSeek V4 FlashHoVer89.8±1.684.2±1.8
Qwen3.8-27BHotpotQA97.8±0.896.5±1.1
Qwen3.8-27BHoVer94.6±1.192.8±1.4

Agreement with the designated better answer (%) ± standard error; thinking enabled. Both columns use the same valid pairs; ties receive half credit.

Pairwise agreement is 39.8–47.8 points above the 50% fair-choice reference. On Qwen3.8-27B's subtler, near-error HotpotQA pairs, pairwise agreement is 84.6% versus 76.6% pointwise, a paired gain of 8.0±2.2 points. These results establish informative judgments.

We also reconstructed all 96 primary search runs. For example, in mini HotpotQA, all three seeds show the following selected-prompt discovery times:

MethodSeed 42Seed 50Seed 51
Full reward3324188
Val206295334
Elo32727897

Iteration when the selected prompt was first evaluated on all 30 validation questions, out of 350. We choose the highest-validation-score prompt, breaking ties by earliest appearance; test scores are not used.

Our acceptance diagnostics identify how order reconciliation and tie handling affect admission pressure. The judge study establishes informative feedback; the trajectories show continued candidate improvement under reduced-feedback search.

2. W1 / Q1 — Do the scores reflect task performance rather than scorer behavior?

HoVer: Val accuracy rises from 38.4% to 45.2% under semantic verdict evaluation. To isolate scoring effects, we applied both rules to the same new Qwen3-8B thinking-on answers from fixed historical prompts: all five seeds, 50 common questions, 250 answers per method. Two LLM interpreters, blinded to reference labels, extract the intended verdict for every method.

MethodOriginal ruleSemantic evaluation
Full reward48.453.6
Val38.445.2
Elo42.842.4

Accuracy (%). Semantic evaluation accepts equivalent verdict wording.

IFBench and PUPA: no observed fallback effect in the audited evaluations. We observed zero fallback/default events in 1,144 IFBench constraint checks and 480 new PUPA judgments. All 588/588 auditable IFBench validation scores reproduced; 480/480 PUPA combined scores were unchanged between numeric parsers. PUPA uses fixed historical Qwen prompts with thinking enabled on 30 common questions. Neither new PUPA nor HoVer answers reconstruct historical test outputs.

Strict all-constraints IFBench scoring preserves aggregate method ordering on common retained validation questions across five Qwen seeds. We will correct attribution to THU-KEG IFBench (Peng et al.). Fallback handling leaves the checked IFBench/PUPA scores unchanged.

Thank you for the clarifications, reconstructed trajectories, and additional scoring checks. The distinction between reduced task-reward feedback during search and labeled final selection is clear, and the fallback checks provide useful evidence for the audited records. However, the two central questions in my review remain incompletely addressed.

First, the constructed-pair results establish informative judgments on those answer pairs. My question concerns their incremental value in the primary search configuration. Neither these results nor the selected-prompt discovery times provide the requested rate-matched random-acceptance control. Please compare at least one primary HotpotQA Val configuration against random acceptance at its observed acceptance rate, keeping the generator, mutation procedure, parent and final selectors, and proposal budget fixed.

Second, the new HoVer/PUPA evaluations and retained IFBench validation records partially address scoring reliability, but their relationship to the primary test results remains unclear. Please identify which already-selected primary prompts are covered and report corrected scores, affected-output counts, and fallback/default counts by method. Historical-output rescoring should be distinguished from regenerated-output evaluation, with the generation configuration and verdict-mapping rules specified. If historical outputs are unavailable, a clearly labeled evaluation of the same selected primary prompts under the original generation configuration would still be useful; checkpoint selection should remain fixed.

I continue to recognize the positive HotpotQA result and the value of studying reduced reward access. Given the remaining uncertainty about the comparator's contribution and the coverage of the scoring audit, I maintain my recommendation of 2 (Weak rejection).

Reviewer_UcyiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether self-evolving agents can improve their prompts with less task-reward feedback. They propose PairEvolve, which removes gold answers from mutation, but use a LLM as judge to compare answers generated from the two prompts, to determine what prompt is better. They have two variants, PairEvolve-Val and PairEvolve-Elo, with Val using labeled validation scores to select the next prompt to evolve, and elo used a maintained Elo rating to select. The method is evaluated on 4 task and two LLMs, to show that useful evolution is possible with fewer reward-dependent decisions.

Critical strengths
  1. The paper tries to an important practical challenge, that the task reward feedback can be limited in self-evolving agents. Tackling such a problem is valuable for making self-evolution more applicable to real-world tasks.
  2. The paper has systematic investigation of LLM validators, like including pairwise versus pointwise judgments, etc. These analyze offer useful insights for LLM-based evaluation.
Critical weaknesses

The paper demonstrates that useful prompts can be generated with reduced task-reward feedback. However, it does not convincingly establish that pairwise feedback guides the evolution process toward progressively better prompts, since the final prompt is selected using a labeled validation set from all generated candidates. It is possible that the best prompt appears early in the evolution process, while subsequent iterations contribute little or no improvement. Without analyzing how prompt quality changes throughout evolution, it is difficult to determine whether pairwise feedback provides effective guidance or whether the observed gains primarily result from generating many candidates and selecting the best through validation.

Questions for authors

Can the authors show how prompt quality changes throughout the evolution process and report at which iteration the final best prompt is discovered? It is unclear whether pairwise feedback effectively guides the search toward better prompts over time, or whether the best prompt often appears early in evolution, with subsequent iterations providing little additional improvement.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Ucyi: Search progress and comparative feedback

Useful evolution with less task-reward feedback

PairEvolve shows that useful evolution does not require task rewards at every search decision.

MethodTask-reward feedback during search
Full rewardGuides revision, acceptance and parent choice
PairEvolve-ValChooses which prompt to revise next
PairEvolve-EloNone; pairwise ratings choose the next parent

Both PairEvolve variants compare the parent and revised prompt’s answers to decide whether to accept the revision. After search, all methods choose the highest-validation-score prompt and test it on separate questions.

For example, GPT-5.4-mini HotpotQA Val/Elo reach 84.0/79.8 test F1, versus 29.2 initially and 77.1 with full reward. Both improve over the initial prompt in all three seeds; Val exceeds full reward in every seed.

1. W1 / Q1 — Does prompt quality improve, and when is the best prompt found?

Improvements continue beyond the initial gains, with selected prompts appearing as late as iteration 334 of 350. To answer the timing question, we reconstructed the validation trajectories and endpoints of all 96 primary evolved runs. This GPT-5.4-mini HotpotQA example includes all three seeds for every method.

MethodInitial25% of budget50%75%100%
Full reward31.0983.6884.5984.9384.93
Val30.1183.4385.6386.8286.93
Elo29.9881.1283.9783.9784.93

Median best-observed validation F1 (%) across seeds 42, 50 and 51; 30 validation questions and 350 attempted mutations.

Large initial gains are followed by smaller, intermittent improvements. The selected prompts were first fully validated at these iterations:

MethodSeed 42Seed 50Seed 51
Full reward3324188
Val206295334
Elo32727897

Each entry shows when the selected prompt was first evaluated on all 30 validation questions, out of 350 iterations. Selection uses the highest validation score, choosing the earliest prompt in a tie.

Test scores determine neither selection nor discovery timing; the table includes early and late selections. Val and Elo discover selected prompts as late as iterations 334/350 and 327/350: continued search contributes better candidates even when task rewards do not guide parent selection.

2. W1 — How informative is the feedback guiding search?

LLM evaluators favor the designated better answer in 89.8–97.8% of clear-error comparisons. Table 2a uses constructed answer pairs: one control and one alternative with an annotated flaw. The evaluating LLM sees the task input and answers, without references or flaw annotations. Pairwise presents both answers together and asks which is better; pointwise scores each separately and favors the higher score.

Evaluating LLMTaskPairwisePointwise
DeepSeek V4 FlashHotpotQA94.7±1.190.8±1.6
DeepSeek V4 FlashHoVer89.8±1.684.2±1.8
Qwen3.8-27BHotpotQA97.8±0.896.5±1.1
Qwen3.8-27BHoVer94.6±1.192.8±1.4

Agreement with the designated better answer (%) ± standard error; thinking enabled. Both columns use the same valid pairs; ties receive half credit.

A fair random choice has expected agreement of 50%. Pairwise agreement of 89.8–97.8% is therefore 39.8–47.8 points above chance, demonstrating substantial quality information for comparative selection. On subtler, near-error HotpotQA pairs, Qwen3.8-27B's pairwise agreement is 84.6% versus 76.6% pointwise, a paired gain of 8.0±2.2 points.

The trajectories establish continued candidate improvement, while the judge study establishes informative comparative feedback. Together with Elo's search mechanism and final performance, these findings support the central contribution: useful comparison-guided evolution without task-reward feedback during search.

Thank you for the detailed response and additional analyses. The trajectories and discovery timings address my question about whether later iterations can produce better candidates. However, it remains unclear whether pairwise feedback itself improves the search beyond generating more candidates and selecting the best through validation. After careful consideration, I have decided to maintain my original score.

Comments on the paper

1 comment

Dear Reviewers,

I hope this message finds you well.

I would like to remind you that we are currently in the rebuttal phase of the paper review process. During this stage, authors may respond to the comments and feedback you provided. If you receive any responses from the authors, please consider providing your feedback to ensure a smooth review process.

Thank you for your hard work and dedication throughout this review process. Your insights are invaluable in enhancing the quality of the submissions, and we truly appreciate your involvement.

If you have any questions or need further assistance, please do not hesitate to reach out.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.