acceptodds
Desk rejected at ICLR 2027

False Frontiers: Diagnosing and Mitigating Co-Cheating in Self-Evolving Search Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-evolving search agents can construct their own training curricula by jointly optimizing a proposer that generates questions and a solver that answers them. This closed loop introduces a failure mode that we call co-cheating: the proposer and solver increasingly agree on shared errors, so internal reward improves without a corresponding increase in external correctness. A post-hoc reference audit against source evidence shows that co-cheating becomes increasingly severe over successive rounds of self-evolution, with pseudo-label correctness stagnating or declining even as the in-loop training signal improves. The most direct mitigation is to verify each proposal before training. We therefore introduce multi-sample verification (MSV), which queries the same model used in self-evolution three times with the source and three times without it to determine task admission and replace unreliable pseudo-labels. MSV partially reduces false agreement but leaves substantial residual co-cheating and requires six additional labeler generations for every candidate. These limitations motivate CrossFit, our main method. It partitions the proposer's source documents into groups A and B: questions generated from A are scored by an auxiliary solver trained only on B, and vice versa. The resulting cross-fitted agreement determines proposer reward, preventing a same-source pseudo-label from being directly reproduced through the feedback solver while leaving the original solver's update rule unchanged. We evaluate both interventions by rerunning the complete self-evolution loop with Qwen3.5-4B and Qwen3.5-9B. After self-evolution, MSV reduces false-agreement mass from 6.1% to 5.7% on Qwen3.5-4B and from 8.8% to 7.2% on Qwen3.5-9B, whereas CrossFit reduces it to 3.0% and 3.7%, respectively. Replaying identical proposals with source-excluded feedback further reduces false agreement to 0.4% and 0.1%, isolating feedback ancestry from changes in the generated curriculum. Across seven downstream search benchmarks, CrossFit improves average performance over standard coupled self-evolution by 8.8 and 8.4 points and over Search-R1 by 8.7 and 7.8 points at 4B and 9B, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-Evolving Agents, Search Agents, Reward Hacking, Pseudo-Labeling, Cross-Fitting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_C2e7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies shared errors in self-evolving search agents, where a proposer generates questions and pseudo-labels and a solver supplies feedback for subsequent curricula. It diagnoses co-cheating through a post-hoc, source-grounded audit of labels and solver responses. The proposed CrossFit method partitions source documents and uses auxiliary solvers trained on complementary partitions to score proposals, while retaining all admitted tasks for main-solver training. Multi-sample verification provides a separate intervention on label quality. Experiments cover three evolution rounds, two Qwen3.5 model sizes, seven downstream benchmarks, and fixed-bank evaluator comparisons. The authors report reduced false agreement and improved downstream question answering.

Critical strengths
  1. CrossFit targets a specific feedback dependency rather than treating every pseudo-label error as interchangeable. Section 3.2 and Figure 4 exclude the evaluated source from the auxiliary solver's training while preserving the main solver's access to all admitted tasks. This provides a concrete way to limit direct recycling of source-derived mistakes without requiring an external truth oracle during training.

  2. The downstream results provide evidence beyond the audit metric. Table 1 reports gains of 8.8 and 8.4 percentage points over Dr. Zero on the same 1,325-question evaluation suite, with improvements on all seven benchmarks at both scales. Table 5 shows that the advantage grows between rounds 2 and 3. These results support the practical value of changing proposer feedback for the learned main solver.

Critical weaknesses
  1. The central co-cheating measurements lack independent validation. Section 2 and Appendix B use one model to construct references and judge labels and responses, but report no human-validation results or adjudication prompts that would establish the reliability of false-agreement judgments. Table 4 reports approximately 85–86% audit coverage, without specifying how unresolved cases enter each statistic, and similar coverage across treatments does not establish comparable judgment error rates. These omissions matter because the claimed reductions depend on distinguishing shared incorrect answers from valid alternatives and unsupported cases. A small blinded validation across treatments would materially strengthen this diagnosis.

  2. The fixed-bank experiment is insufficiently specified to verify its causal interpretation. Sections 5.2 and 5.3 do not identify the bank's originating run and round, its overlap with each evaluator's training questions, or the scoring rule for replay accuracy. For the 4B coupled control, Section 5.2 reports 68.7% solver truth and 88.1% replay accuracy without explaining how these quantities differ. The same-source, full-data, random-row, and half-budget controls are relevant, but the missing protocol details prevent checking whether the comparison isolates source ancestry rather than exact-question exposure or differences in evaluation. This limits the mechanism claim, independently of the downstream gains.

Questions for authors
  1. Could you provide a small blinded human validation of the audit's false-agreement judgments across treatments, with the observed disagreements and an explicit account of how unresolved cases enter the reported rates?

  2. Could you specify the fixed-bank construction and each evaluator's initialization and training overlap, and define replay accuracy relative to solver truth, so that the source-exclusion comparison in Figure 7 can be reconstructed?

9 comments
AuthorsOpenReview ↗

Response to Reviewer C2e7 (1/7): W1

Thank you for your time and for looking closely at how we computed the audit and replay scores. Your questions prompted a human re-review of the audit and a comparison of replay outcomes on the same questions.

W1. Can audit errors or unresolved cases explain the reduction in false agreement?

Response.{\color{green}\mathbf{Response.}} We checked whether the original model audit's judgments agree with human judgments. We re-reviewed 30 questions per treatment and scale, comparing the adopted label and five saved solver answers. A label–response pair is counted as false agreement when both share the same incorrect answer. We compare only pairs whose status is resolved by both the audit and the human review, using the same sample weights and denominator for both F estimates.

With human judgments as the reference, precision measures how many audit-flagged false agreements humans confirm; recall measures how many human-identified false agreements the audit detects. These metrics assess judgment reliability on the jointly resolved validation sample.

Model  Method  Audit F (%)  Human re-review F (%)  Weighted precision (%)  Weighted recall (%)
4BDr. Zero6.86.8100.0100.0
4BMSV5.75.495.3100.0
4BCrossFit2.52.496.4100.0
4BMSV + CrossFit  0.80.8100.0100.0
9BDr. Zero9.59.296.9100.0
9BMSV7.07.0100.0100.0
9BCrossFit2.72.7100.0100.0
9BMSV + CrossFit  1.51.5100.0100.0

Human judgments reproduce CrossFit's lower false-agreement rate: 6.8% versus 2.4% for Dr. Zero and CrossFit at 4B, and 9.2% versus 2.7% at 9B. Weighted precision is 95.3–100%, and recall is 100% on the jointly resolved pairs. This supports the audit's reliability and the CrossFit comparison on the human-reviewed sample.

These validation-sample rates are distinct from Table 4's averages over 43 training steps. Audit coverage is reported as J/E (Appendix B); unresolved cases are addressed in Q1 below.

AuthorsOpenReview ↗

Response to Reviewer C2e7 (2/7): W1 (continued)

Continuation of our response to Reviewer C2e7.

W1 (continued). Can audit errors or unresolved cases explain the reduction in false agreement?

Blinded human-review protocol. Three author annotators participated: R1 and R2 independently reviewed every sampled question, and R3 adjudicated disagreements. R1 and R2 each received the same anonymized question, source/evidence packet, adopted label, and five saved responses. Method, model scale, training round, reward, audit verdicts, and the other annotator's verdicts were hidden. Each annotator reconstructed the evidence-backed acceptable-answer set and classified the label and each response as correct, incorrect, or unresolved; decisions were locked before comparison. A complete decision consists of the label verdict and the five response verdicts. If any of the six is unresolved, the decision is treated as unresolved, while the case record retains the partial verdicts. When R1 and R2 agree exactly, that decision is final; otherwise, R3 independently reviews the same evidence packet and makes the final decision. R3's decision may remain unresolved. Two unresolved initial decisions count as exact agreement but remain unresolved.

Model  Method  R1–R2 exact agreement / 30  R1 and R2 both unresolved  Sent to R3 / 30  Final human decision unresolved / 30
4BDr. Zero19 / 30 (63.33%)011 / 30 (36.67%)0 / 30 (0.00%)
4BMSV21 / 30 (70.00%)09 / 30 (30.00%)2 / 30 (6.67%)
4BCrossFit22 / 30 (73.33%)08 / 30 (26.67%)0 / 30 (0.00%)
4BMSV + CrossFit  21 / 30 (70.00%)19 / 30 (30.00%)1 / 30 (3.33%)
9BDr. Zero20 / 30 (66.67%)110 / 30 (33.33%)2 / 30 (6.67%)
9BMSV20 / 30 (66.67%)010 / 30 (33.33%)1 / 30 (3.33%)
9BCrossFit19 / 30 (63.33%)011 / 30 (36.67%)2 / 30 (6.67%)
9BMSV + CrossFit  24 / 30 (80.00%)06 / 30 (20.00%)0 / 30 (0.00%)

Across all 240 questions, R1–R2 exact agreement is 166/240 (69.17%), including two both-unresolved cases; R3 adjudicates 74/240 (30.83%), and the final human decision remains unresolved for 8/240 (3.33%). Two final unresolved cases retain the initial both-unresolved decision; six remain unresolved after R3 adjudication. Among the 202 questions that both initial annotators resolve completely, agreement is 164/202 (81.19%). Final human-decision coverage is 232/240 (96.67%). The audit and the final human decisions jointly resolve 194 questions, or 970 pairs.

Audit instruction and outputs. The automated auditor and human annotators use the following evidence-based decision rule.

Given the question, source/evidence packet, adopted label, five saved responses and frozen label-match flags, establish the set of acceptable answers supported by the question and evidence. Record supporting passages and valid alternative answers. Judge the label and each response as correct, incorrect or unresolved, with an evidence-based reason. Supported aliases and valid alternatives count as correct; disagreement with a preferred answer string alone does not establish error. Retain unresolved when evidence is insufficient, contradictory or unable to disambiguate the answer. Preserve partial verdicts without guessing. Mark shared wrong only when the frozen matcher is positive and the label and response express the same incorrect answer. Mark lost credit only when the label is incorrect, the response is correct, and the matcher rejects it. Leave a predicate unresolved if its evidence is undecidable. Do not alter saved answers, match flags, reward, admission or updates; report matcher discrepancies explicitly. Return acceptable answers, supporting citations, six verdicts, five shared-wrong/lost-credit decisions and unresolved reasons.

The output record contains question_id, reader_id, acceptable_answers, evidence_citations, adopted_label, and responses. Each verdict contains status, reason, and evidence_ids; the five response entries also contain slot, frozen_match, shared_wrong, and lost_credit. status is correct, incorrect, or unresolved. Slots must be exactly 1–5. The aggregation applies the decision rules above.

AuthorsOpenReview ↗

Response to Reviewer C2e7 (3/7): W2

Continuation of our response to Reviewer C2e7.

W2. Does the fixed-bank comparison isolate source ancestry from exact-question exposure?

Response.{\color{green}\mathbf{Response.}} We use two complementary controls. First, fixed-bank replay keeps the evaluated questions, adopted labels, and matcher unchanged while varying the evaluator's training history. Figure 7 compares coupled and source-ID feedback on the same questions; the supplementary replay compares same-source and source-ID OOF auxiliary solvers on the same 3,000-question bank.

The table focuses on the 829 questions with wrong adopted labels. Five answers per question give 4,145 responses per evaluator. Wrong-label match counts responses repeating the incorrect label; reference-correct answer counts responses judged correct against the reference. Lower values in the first column and higher values in the second indicate improvement.

Model  Auxiliary evaluator  Wrong-label match  Reference-correct answer
4BSame-source946 / 4,145 (22.82%)  2,158 / 4,145 (52.06%)
4BSource-ID OOF65 / 4,145 (1.57%)3,072 / 4,145 (74.11%)
9BSame-source1,302 / 4,145 (31.41%)  1,996 / 4,145 (48.15%)
9BSource-ID OOF20 / 4,145 (0.48%)3,540 / 4,145 (85.40%)

Source-ID OOF reduces repetition of wrong labels while increasing correct answers on the same questions. Wrong-label matching falls from 22.82% to 1.57% at 4B and from 31.41% to 0.48% at 9B, while reference correctness rises from 52.06% to 74.11% and from 48.15% to 85.40%. Thus, the improvement includes recovering correct answers on questions whose adopted labels are wrong.

In the supplementary replay, exact-question-ID overlap is 0% for both evaluators; routed source-ID overlap is 100% for same-source and 0% for source-ID OOF. The following F rates use all 15,000 responses per evaluator.

Model  Evaluator  Exact-question overlap (%)  Routed source overlap (%)  F, all responses
4BSame-source auxiliary  0.00100.00946 / 15,000 (6.31%)
4BSource-ID OOF0.000.0065 / 15,000 (0.43%)
9BSame-source auxiliary  0.00100.001,302 / 15,000 (8.68%)
9BSource-ID OOF0.000.0020 / 15,000 (0.13%)

Second, the reward-label control fixes the training curriculum and changes the labels used to compute the main solver's training rewards. Each cell below reports adopted-label rewards / reference-label rewards. Evaluation still measures F against the original adopted labels and correctness against the reference, so changing the training reward labels does not change the evaluation definitions.

Model  Dr. Zero main-solver reward labels  F (%)  Solver truth (%)
4BAdopted / reference9.05 / 4.05  75.46 / 80.94
9BAdopted / reference7.27 / 3.02  79.56 / 84.80

Reference-label rewards lower F and improve reference correctness at both scales on the fixed curriculum. Together, these controls show that both evaluator training history and reward-label choice affect shared-error reproduction. The exact-question exposure details needed to interpret source ancestry separately are specified in Q2.

Reviewer_Gnk2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies and investigates co-cheating in data-free search agent training, a failure mode where internal agreement reward rises while true accuracy stagnates because both agents increasingly converge on shared, source-derived errors. To mitigate this issue, the authors compares two interventions: multi-sample verification (MSV), an admission-time filter that relabels candidates via consensus across three source-aware and three source-blind rollouts. They also propose CrossFit, their main method, which splits source documents into two folds to cross-score proposer rewards via out-of-fold auxiliary solvers, while keeping the main solver trained on all tasks. Experiments show that while MSV yields only modest gains at considerable sampling cost, CrossFit cuts round-3 false-agreement mass by more than half and boosts average performance across QA benchmarks.

Critical strengths
  1. Instead of relying on internal reward curves, the authors design a post-hoc audit grounded in source evidence to track true correctness. This exposes co-cheating, showing how disagreement between proposer and solver is gradually replaced by shared, source-derived mistakes.

  2. By proposing g MSV with CrossFit, the authors show that how the proposer model is scored matters. CrossFit requires no changes to the original frontier reward function and keeps the main solver trained on 100% of admitted questions, reducing co-cheating while improve solver performance.

  3. The proposed method consistently improves performance across two model scales and seven QA benchmarks.

Critical weaknesses
  1. The proposed method lacks novelty. While diagnosing co-cheating in search self-play is valuable, the proposed mitigations are direct adaptations of standard techniques.

  2. Substantial compute overhead without iso-compute baselines. Training two extra auxiliary solvers increases total solver updates and raises end-to-end H200-hour budgets over the Dr. Zero baseline.

Questions for authors

n/a

2 comments
AuthorsOpenReview ↗

Response to Reviewer Gnk2: W1, W2

Thank you for taking the time to assess the contribution and for emphasizing the need for a fair compute comparison. The extended Dr. Zero runs test how much longer training closes the performance gap.

W1. How does CrossFit advance beyond standard cross-fitting and verification?

Response.{\color{green}\mathbf{Response.}} Our contribution is to diagnose and test a feedback failure tied to shared source documents: a solver can learn an incorrect label from one question and repeat the error when scoring another question from the same document, returning that error as proposer reward. CrossFit applies the established exclusion principle of cross-fitting [1] to this feedback path. Each auxiliary solver scores sources excluded from its self-evolution training, while the main solver still trains on all admitted questions (Figure 4).

Verification checks label reliability at admission; CrossFit controls the evaluator's training history. A separate evaluator or a random question split can retain same-source supervision, as tested in Figure 7. In the supplementary replay below, both evaluators answer the same 3,000 questions with fixed labels: same-source auxiliary has trained on questions from the evaluated sources, whereas source-ID OOF has trained only on other sources.

Model  Same-source auxiliary F  Source-ID OOF F  Same-source solver truth  Source-ID OOF solver truth
4B6.31%0.43%69.19%77.14%
9B8.68%0.13%70.92%86.87%

Excluding source ancestry sharply reduces false agreement while improving reference correctness at both scales. Together with the random-question-fold ablation, these results support source documents as the relevant unit of exclusion; simply adding an auxiliary evaluator is insufficient when it retains same-source supervision. The full training runs also improve the main solver on all seven benchmarks at both scales (Table 1), connecting this feedback intervention to downstream gains.

[1] Double/Debiased Machine Learning for Treatment and Causal Parameters


W2. Could extra compute account for the downstream gains?

Response.{\color{green}\mathbf{Response.}} We tested whether longer Dr. Zero training could recover CrossFit's gains under the same reserved H200-hour budget caps: 650 hours at 4B and 854 hours at 9B. We continued Dr. Zero and evaluated its last checkpoint within each cap on the original 1,325-question, seven-benchmark suite (Appendix B). The extended runs reached 125 and 130 main-solver updates, respectively, compared with 75 for CrossFit. The table reports both the budget caps and the reserved hours used at the selected checkpoints.

Metric  4B Dr. Zero continued  4B CrossFit  9B Dr. Zero continued  9B CrossFit
Reserved H200-hour cap650650854854
Reserved H200-hours at checkpoint  631.64650 (Table 3)853.76854 (Table 3)
Main solver updates1257513075
NQ (%)42.545.552.058.0
TriviaQA (%)68.073.074.076.5
PopQA (%)33.541.542.045.5
HotpotQA (%)39.046.040.549.0
2WikiMQA (%)49.057.535.542.5
MuSiQue (%)15.524.519.525.5
Bamboogle (%)45.653.653.661.6
Macro Cover-EM (%)41.948.845.351.2
Micro Cover-EM (%)41.748.544.850.6

CrossFit retains higher performance on all seven benchmarks at both scales under the matched budget caps. Its macro Cover-EM is 48.8% versus 41.9% at 4B (+6.9 percentage points) and 51.2% versus 45.3% at 9B (+5.9 percentage points). Longer training improves Dr. Zero but leaves a substantial gap, supporting a benefit from changing feedback provenance beyond extending coupled training.

AuthorsOpenReview ↗

Looking forward to your feedback

Thank you again for your thoughtful review. We have tried our best to address your concerns in our response: how CrossFit goes beyond standard cross-fitting and verification, supported by the same-source versus source-ID OOF replay, and the compute comparison with Dr. Zero continued under the same H200-hour budget caps.

As the discussion period is drawing to a close, we would be grateful to hear whether this response addresses your concerns. If any questions remain, we would be glad to discuss them further.

Reviewer_hbacOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a co-cheating phenomenon in the proposer-solver-based data generation framework developed in prior work. Specifically, the proposers and solvers may increasingly agree on incorrectly labeled questions, leading to misleading reward feedback rather than actual improvement. In this paper, the authors study this phenomenon and propose MSV and CrossFit to mitigate this issue. Specifically, MSV tries to calibrate the correctness of the label with source exclusion, and CrossFit isolates the source documents used for training the proposer and solver. Empirical results and controlled studies demonstrate the effectiveness of the proposed CrossFit algorithm in reducing co-cheating.

Critical strengths
  1. The paper presents an interesting diagnosis of co-cheating in self-evolving search agents. The proposed CrossFit is simple and intuitive. Source-level exclusion provides a reasonable way to mitigate feedback contamination.
  2. The empirical results seem strong. The fixed-bank replay and data-splitting ablations provide convincing evidence for the importance of feedback provenance, with consistent downstream improvements across tasks and model scales.
Critical weaknesses
  1. The mechanism behind co-cheating could be further understood. In particular, the relative contributions of proposer reward optimization and solver self-confirmation are not entirely clear.
  2. The relationship between false agreement and the actual proposer reward could be analyzed more directly, especially since the frontier reward is non-monotone in agreement.
Questions for authors
  1. The paper suggests that same-source false agreement can lead to misleading proposer feedback. However, the frontier reward is not monotone in agreement, so increasing false agreement does not necessarily increase proposer reward. Can the authors provide a more direct analysis of the relationship of two scores, or how they are driving the training dynamics in a correlated way?
  2. Since the proposer and solver are updated in separate phases, can the authors further distinguish their contributions to co-cheating? For example, evaluating the proposer before and after its updates using a common frozen solver, and evaluating the solver on a fixed question bank. I think that could help understand whether the increase in false agreement comes mainly from proposer selection or solver self-confirmation.
  3. CrossFit addresses same-source feedback contamination, but still uses the original difficulty-based reward. I wonder whether the proposer could still favor ambiguous or low-quality questions that receive high difficulty rewards. A small empirical analysis of question quality under CrossFit would help understand the scope of this improvement.
  4. Could similar co-cheating issues arise in other data generation frameworks, such as SPICE (Liu et al., 2025) and INFUSER (Chen et al., 2026)? SPICE uses corpus-grounded self-play with a different frontier reward and question validation, while INFUSER uses an influence-guided generator reward to directly optimize external learning utility rather than difficulty. I would appreciate some further discussion of these papers.
7 comments
AuthorsOpenReview ↗

Response to Reviewer hbac (1/4): W1, W2

Thank you for taking the time to review our work and for asking how shared errors earn training credit. Your suggestion to separate the proposer and solver updates guided our controlled comparison.

W1. What drives co-cheating: proposer optimization or solver self-confirmation?

Response.{\color{green}\mathbf{Response.}} We evaluated the proposer and solver before and after their respective updates. The comparison concerns the Round 2 proposer and solver updates, starting from their shared Round 1 endpoint. The pre-update and post-update proposers each generated a bank of 1,536 questions. We then evaluated both versions of the solver on each bank, giving four combinations in total.

To measure the effect of the proposer update, we kept the pre-update solver fixed and compared its false-agreement rate (F) on the two banks. F increased on the post-update proposer bank. To measure the effect of the solver update, we kept each bank’s questions and adopted labels fixed and compared F before and after updating the solver. On the post-update proposer bank, this solver update produced a larger increase in F than the proposer update alone. The solver update also increased F more on the post-update bank than on the pre-update bank; this difference is the interaction reported below. All changes in F are absolute differences in percentage points.

Model  Method  Proposer update with pre-update solver  Solver update on post-update bank  Interaction
4BDr. Zero3.78% → 5.09% (+1.32 pp)5.09% → 9.05% (+3.96 pp)+1.21 pp
4BCrossFit3.78% → 4.41% (+0.64 pp)4.41% → 7.60% (+3.19 pp)+0.68 pp
9BDr. Zero2.43% → 3.32% (+0.89 pp)3.32% → 7.27% (+3.95 pp)+1.12 pp
9BCrossFit2.43% → 3.15% (+0.72 pp)3.15% → 5.83% (+2.68 pp)+0.57 pp

Differences are computed from unrounded values.

These results show that both updates contribute to co-cheating, with a larger increase in false agreement following the solver update. The positive interaction further indicates that solver training reinforces shared errors more strongly on the updated proposer's questions. At both model scales, CrossFit reduces the increase in F associated with each update and weakens their interaction, supporting source-excluded feedback as a way to limit this reinforcement.


W2. False agreement and proposer reward have a non-monotone relationship.

Response.{\color{green}\mathbf{Response.}} We agree that an increase in false agreement (F) does not necessarily increase proposer reward. The frontier reward is non-monotone: zero, one, two, and five matching answers receive rewards of 0, 1, 0.75, and 0, respectively (Section 3.2; Dr. Zero [1]). We therefore measured the reward contribution of shared errors directly. Keeping the same questions and five saved solver answers fixed, we recomputed the reward in two ways:

  • Remove false matches: stop counting matches identified by the audit as shared errors, leaving all other match decisions unchanged.

  • Score against the reference: count matches to the audited reference answer instead of the adopted label. This also credits correct answers that the original label failed to recognize.

For each calculation, the table reports original reward − recomputed reward, averaged over fully adjudicated Round 3 questions. A positive value means the original scoring assigned more reward. These are reward-score differences, not percentage-point changes in F.

Model  Method  Fully adjudicated questions / total  Reward gap: remove false matches  Reward gap: score against reference
4BDr. Zero2,971 / 3,456+0.152+0.056
4BCrossFit2,971 / 3,456+0.100−0.0003
9BDr. Zero2,985 / 3,456+0.172+0.062
9BCrossFit2,969 / 3,456+0.082+0.014

Removing false matches lowers mean reward in all four runs, confirming that shared errors earn proposer credit under the non-monotone frontier objective. CrossFit reduces this reward contribution from 0.152 to 0.100 at 4B and from 0.172 to 0.082 at 9B. It also reduces the magnitude of the mean reward gap relative to reference scoring at both scales, with a near-zero gap at 4B. These results show that CrossFit limits the reward assigned to shared errors while retaining the original frontier reward function.

[1] Dr. Zero: Self-Evolving Search Agents without Training Data

AuthorsOpenReview ↗

Response to Reviewer hbac (2/4): Q1

Continuation of our response to Reviewer hbac.

Q1. Does false agreement translate into a positive proposer training signal?

Response.{\color{green}\mathbf{Response.}} We examined whether questions with incorrect adopted labels receive both frontier reward and a positive proposer training signal over the three evolution rounds. These are distinct quantities: a question can receive positive reward but still have negative advantage if its reward is below the corresponding task-bucket baseline. We therefore report two measures:

  • Wrong-label share of frontier reward: the reward awarded to questions with confirmed incorrect labels, divided by the total reward in the measured set. This measures how much of the reward goes to wrong-label questions, not how many questions have wrong labels.

  • Wrong-label selections with positive advantage: among selected questions with confirmed incorrect labels and all five solver answers adjudicated, the percentage whose reconstructed task-bucket advantage is positive. Positive advantage indicates a favorable relative training signal in the reconstructed objective; this is not a direct measurement of parameter updates.

The table reports both percentages for Rounds 1, 2, and 3. The advantage comparison had the same direction under both tested bucket layouts.

Model  Method  Wrong-label share of frontier reward, R1 / R2 / R3 (%)  Wrong-label selections with positive reconstructed advantage, R1 / R2 / R3 (%)
4BDr. Zero4.8 / 21.8 / 43.83.6 / 27.9 / 64.2
4BCrossFit4.8 / 13.3 / 25.43.6 / 20.3 / 56.3
9BDr. Zero2.1 / 23.6 / 55.04.1 / 29.9 / 72.2
9BCrossFit2.1 / 11.5 / 19.44.1 / 19.0 / 63.2

Under Dr. Zero, wrong-label questions receive a growing share of reward and increasingly obtain positive reconstructed advantages across rounds. By Round 3, they account for 43.8% and 55.0% of frontier reward at 4B and 9B, respectively. CrossFit lowers these shares to 25.4% and 19.4%, and reduces the fraction of selected wrong-label questions with positive reconstructed advantage from 64.2% to 56.3% at 4B and from 72.2% to 63.2% at 9B. These results support the link between shared errors and positive proposer training signals, and show that CrossFit reduces both the reward share and the prevalence of positive reconstructed advantages associated with wrong-label questions. Residual positive advantages remain, so the mitigation is partial.

AuthorsOpenReview ↗

Response to Reviewer hbac (3/4): Q2

Continuation of our response to Reviewer hbac.

Q2. Can you isolate the two update phases and test solver self-confirmation?

Response.{\color{green}\mathbf{Response.}} The proposer-only comparison is reported in W1. Here we isolate the solver update by fixing the post-update proposer bank and comparing three conditions:

  • Frozen solver: the solver before its update.

  • Adopted-label update: the solver updated using rewards computed against the original adopted labels.

  • Reference-label update: the solver updated on the same training questions using rewards computed against the audited reference labels.

The two update conditions differ in the labels used to compute training rewards. At evaluation, all three conditions use the same questions. We measure F against the original adopted labels in every condition, and measure solver correctness against the audited reference. Thus, changing the training reward labels does not change the definition of F. Each table cell lists the three conditions in the order frozen / adopted-label update / reference-label update.

Model  Method  F (%): frozen / adopted-label update / reference-label update  Reference correctness (%): frozen / adopted-label update / reference-label update
4BDr. Zero5.09 / 9.05 / 4.0577.62 / 75.46 / 80.94
4BCrossFit4.41 / 7.60 / 3.5878.82 / 77.54 / 81.88
9BDr. Zero3.32 / 7.27 / 3.0282.80 / 79.56 / 84.80
9BCrossFit3.15 / 5.83 / 2.8483.50 / 81.39 / 85.31

Across all four runs, updating with adopted-label rewards increases F and lowers reference correctness relative to the frozen solver. Updating with reference-label rewards reverses both directions: F falls and reference correctness improves. This controlled comparison supports solver self-confirmation: training rewards based on incorrect labels can reinforce shared errors on a fixed question bank. CrossFit has a smaller increase in F under adopted-label updates than Dr. Zero at both scales: +3.19 versus +3.96 percentage points at 4B, and +2.68 versus +3.95 at 9B.

We next examined where the increase in F under the adopted-label solver update occurs. We divided the evaluation bank into three groups of 512 questions according to their source's exposure history: sources used in the current solver update, sources used only in earlier updates, and sources never used in solver training. For each group, ΔF is the false-agreement rate after the solver update minus its rate before the update, measured in percentage points.

Model  Method  ΔF on sources seen this round  Seen only earlier  Never seen
4BDr. Zero+10.31 pp+0.78 pp+0.78 pp
4BCrossFit+8.32 pp+0.63 pp+0.63 pp
9BDr. Zero+10.86 pp+0.94 pp+0.04 pp
9BCrossFit+7.19 pp+0.70 pp+0.16 pp

The increase in F is concentrated on sources used in the current update; the increases for earlier-only and never-seen sources are all below one percentage point. CrossFit reduces this largest, current-source increase from +10.31 to +8.32 percentage points at 4B and from +10.86 to +7.19 at 9B. This concentration is consistent with shared-error reinforcement tied to recent source exposure, and the smaller increases under CrossFit support the value of controlling source ancestry in proposer feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.