acceptodds
Under review as a conference paper at ICLR 2027

Beyond Correct Labels: Feedback, Efficiency, and Coverage

Abstract

Mathematical self-evolution requires supervision that improves answer accuracy while preserving the breadth of problems a model can solve. We propose an oracle-based mathematical co-evolution framework that constructs training targets through program execution and screens their agreement with problem statements. To audit the target–grader–policy interface, we define Active Error Exposure (AEE) as the joint probability of an incorrect target and a nonconstant task-reward group, and pair it with replay diagnostics for lost reference distinctions. The first round raises pass@1 from 36.4% to 64.6% on MATH-500 and from 17.0% to 28.5% on a fixed 500-question Olympiad subset, while retaining high pass@256 coverage. Across three recorded R2 pairs, pass@256 gains over an Agent0 reproduction are 2.4–4.2 points on MATH and 9.2–11.0 on Olympiad; native 8K/16K Olympiad checks do not remove the gap. A label-only intervention further shows a training-side result: changing only conflict targets changes delivered group-relative feedback: E conflict groups are mostly all-positive, whereas M groups are mostly all-negative. The downstream effect of that intervention remains unidentified under the available grading versions. In a native 16K Olympiad check, Oracle-based reaches 64.15% pass@32 at 786 tokens per response, versus 56.87% for Agent0 at 9,905 tokens, a descriptive 12.6-fold realized-token contrast. For Agent0, extra decoding is net negative: 276 decisions are rescued but 409 change from positive to negative from its 4K prefix to 16K.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-evolving reasoning models, self-play, curriculum generation, execution-grounded supervision, reward and feedback analysis, group-relative advantage, LLM agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zbWYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies supervision in two rounds of mathematical co-evolution using Qwen3-4B-Base. Its execution-grounded framework generates problem-program pairs, executes programs to obtain targets, and filters them using fresh statement-only implementations before ADPO training. It introduces Active Error Exposure, the joint occurrence of an incorrect target and a nonconstant reward group, alongside paired reference replay. Experiments report higher finite-budget coverage and shorter outputs than an Agent0 reproduction on MATH-500, an Olympiad subset, and AIME25, with longer-output checks on Olympiad. Additional probes examine target substitution and representation-sensitive grading. The target-only training experiment reports altered feedback, while its downstream effect remains unresolved.

Critical strengths
  1. Table 1 reports positive R2 pass@256 gaps on MATH and Olympiad in all three recorded pairs under both extraction rules. The native 16K experiment further reports Olympiad pass@32 of 64.15% versus 56.87%, at mean output lengths of 786 versus 9,905 tokens (Tables 19-21). This same-stack native comparison supports a practically meaningful coverage and output-cost advantage for the evaluated checkpoints, not merely an improvement in pass@1.

  2. The fixed-policy, same-response replay demonstrates that collapsed rewards can erase distinctions present under reference rewards. In Table 4, low-reach targets yield five mixed groups versus 41 for reachable targets, and turn 35 of 40 source-mixed groups all-negative (Section 4.5.3). Appendix H's equivalent-target example makes the distinction between semantic errors and representation-sensitive grading particularly useful.

Critical weaknesses
  1. The interpretation of grader-measured coverage as mathematical coverage needs a targeted validity check. Appendix H reports seven versus zero accepted responses for mathematically equivalent target encodings, whereas the main benchmark comparisons retain mathruler grades (Appendices F, K, and L). First-box and last-box sensitivity tests vary extraction, not the underlying equivalence decisions, and the saved-record audits do not establish their semantic accuracy. Given the different output behaviors across methods, differential grading errors remain an unresolved explanation for part of the coverage gap. A blinded audit of answers on questions covered by only one method would help assess this concern.

  2. AEE's empirical value beyond marginal error and collapse rates remains insufficiently demonstrated. Equations (1) and (6) define the statistic and its conditional-expectation decomposition, but the results do not report joint target-error/activity estimates for the main training runs. Table 4 measures grader reachability, while Table 23 demonstrates reward saturation in the target-only intervention, rather than a comparison showing what the joint statistic reveals beyond those marginals. These findings support the feedback observations, but leave the added diagnostic contribution underdeveloped. A bounded analysis of the existing reviewed groups could strengthen this contribution without requiring proof that AEE causes downstream gains.

Questions for authors
  1. On a blinded, stratified sample of s2 benchmark questions covered by only one method, can independent mathematical checking verify the accepted answers and inspect rejected equivalent answers from the other method? Reporting the resulting corrections would help determine whether grading asymmetry materially affects the coverage advantage.

  2. Can the existing reviewed training groups support a joint table of target correctness and reward activity, with AEE, annotation coverage, and unresolved cases reported explicitly? Within that selected sample, what diagnostic conclusion differs from one based only on marginal target-error and collapse rates?

2 comments
AuthorsOpenReview ↗

Response to Reviewer zbWY: Coverage Audit and Joint AEE Analysis

Thank you for your detailed assessment of the same-stack coverage and output-cost comparisons and the fixed-policy replay results. Following your suggestions, we have reviewed candidates from all one-sided coverage questions and assembled a joint analysis of target correctness and reward activity.

Q1

We have reviewed 811 candidate responses spanning all 74 questions covered by only one method in the s2 comparison. These comprise 66 questions originally classified as covered only by our method and 8 covered only by Agent0. The review checks accepted answers and examines potentially overlooked correct responses from the other method.

For newly reviewed candidates, complete responses with method identities concealed were examined in two separate contexts of the same model, and the findings were combined with existing review records. We retain both a legacy-compatible criterion and a stricter criterion requiring explicit confirmation that the final answer was not subsequently retracted.

The review identifies concrete corrections to coverage assignments. Under the legacy-compatible criterion, correct baseline responses were found on 10 questions originally classified as covered only by our method; the stricter criterion confirms 9 of these. No additional jointly covered questions were confirmed in the reverse direction.

To quantify the effect of these identified corrections, we perform a conditional sensitivity calculation: reclassify these 10 questions as covered by both methods, keeping all other original grading decisions unchanged.

Category within the original 74 one-sided questionsOriginal gradingAfter specified corrections
Covered only by our method6656
Covered only by Agent088
Covered by both010
Difference in one-sided counts (ours minus Agent0)5848

Under this conditional calculation, the one-sided coverage difference remains in our favor after incorporating the identified shared-coverage corrections. The table makes the effect of these false negatives explicit. The review covers 811 candidates from the full pool of 37,888 responses; the remaining original grades are retained in this calculation. The revision will report the candidate-selection rules, model-assisted review procedure, review scope, and unresolved cases alongside this table.

The audit highlights a broader methodological challenge for self-evolution pipelines that rely on automated grading: mathematically equivalent answers can receive different scores, affecting both training rewards and measured coverage. This makes semantic-equivalence assessment and feedback diagnostics important directions for further research. We will discuss this shared challenge alongside the concrete grading asymmetries identified in our audit.

Q2

We have aligned target-correctness review results with the original reward records and constructed the requested joint analysis. Of 72 saved training groups, 42 have resolved target-correctness labels under the current model-assisted review. Annotation coverage and AEE within the resolved-label subset are:

MethodResolved / saved groupsUnresolved groupsIncorrect and active groupsResolved-subset AEE
Agent013/3219646.15%
Our method29/401126.90%

AEE uses each method's resolved-label group count as its denominator. Within this subset, our method exhibits lower active error exposure. The joint counts show how target correctness and feedback activity intersect:

MethodTarget statusActive rewardsCollapsed rewards
Agent0Correct51
Agent0Incorrect61
Our methodCorrect320
Our methodIncorrect24

Of Agent0's 7 resolved incorrect-target groups, 6 have active rewards; of our method's 6 resolved incorrect-target groups, 2 have active rewards. The two methods have similar numbers of incorrect targets in this subset, distributed across different feedback states. The joint table locates which incorrect targets still produce task-reward distinctions used to differentiate responses.

The additional information is also visible in the pooled counts: the 42 resolved groups contain 13 incorrect-target groups and 16 active-reward groups. These marginal counts allow their intersection to range from 0 to 13 groups; the group-level linkage identifies 8. AEE connects target quality with observed reward structure, locating supervision issues in specific feedback states. These groups can then be examined through reference replay to determine how the reward distinctions change.

The revision will include these joint counts, annotation coverage, unresolved cases, and subset AEE, presenting the findings as diagnostics within the resolved-label subset. This makes the practical information added by the joint statistic directly inspectable.

Thank you for the detailed responses and additional experiments. It helps clarify some of my concerns. I will keep my current score.

Reviewer_P1vjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes execution-grounded self-training for math reasoning, improving performance while keeping broad problem coverage and pass@k performance. It also introduces diagnostics that distinguish misleading rewards from missing learning signals.

Critical strengths
  1. The method improves pass@1 while largely preserving pass@256, which seems stronger than traditional RL methods that cause collapsing.
Critical weaknesses
  1. Experiments are only performed on one backbone (Qwen3-4B). Any other similar-sized non-qwen model could help further prove author's point.

  2. The discussion and generalizability of the method is limited to just math reasoning.

Questions for authors
  1. Does additional rounds still provide improvement?
2 comments
AuthorsOpenReview ↗

Response to Reviewer P1vj: Generality and Additional Co-evolution Rounds

Thank you for recognizing our improvements in single-sample accuracy while preserving high-sampling-budget coverage. We address your concerns about model generality, task scope, and additional co-evolution rounds below.

1. Model generality

We have added training and evaluation on Llama-3.1-8B-Instruct, extending our empirical study to another model family. Starting from the same Llama initialization, we conducted a target-only comparison using program-execution answers and model-voted answers, with each arm completing 20 training steps.

We evaluated the initialization and both trained checkpoints under a common setup on MATH-500 and a fixed 500-question Olympiad subset. Each model generated 32 responses per question, yielding 96,000 evaluation responses in total.

Execution-target training improves MATH pass@1 by more than 10 percentage points over initialization under a conservative comparison based on the current reviewed results. Voting-target training also improves performance. These results provide additional training evidence in another model family and show that execution-derived targets can support effective mathematical reasoning training.

The revision will include the setup and findings, explicitly identifying Llama as an 8B instruction-tuned model and presenting this supplementary comparison alongside the original Qwen3-4B base-model experiments.

2. Task generality

The framework applies to tasks for which programs can produce deterministic answers whose consistency with the task descriptions can be checked. Mathematics provides a suitable setting for constructing executable targets and studying how target quality affects training feedback. The same workflow offers concrete extensions:

  • Physics: given specified initial conditions, physical models, and parameters, analytical computation or numerical programs can produce reference answers. Numerical tasks require explicit precision and tolerance specifications.
  • Coding: program outputs, deterministic input-output tests, or reference implementations can provide supervision.

For tasks with deterministic, comparable answers, voting-based and execution-based supervision construct targets through model-answer aggregation and program execution, respectively. Both can follow the task-generation pipeline; our execution-based route generates a task and an executable solution procedure, obtains the target through execution, and checks consistency with the task description.

The feedback diagnostics also extend to tasks using group-relative rewards. With verifiable reference answers, AEE characterizes the overlap between incorrect targets and reward activity, while fixed-response replay examines how grading targets affect reward distinctions. This provides a practical route for studying supervision quality and feedback structure in additional domains.

Our current empirical results concern mathematical tasks. The revision will expand the discussion of these application settings, their verification requirements, and the corresponding extension workflow.

3. Additional co-evolution rounds

The completed R1-to-R2 results show that an additional round can improve single-sample accuracy while broadly maintaining high multi-sample coverage. In the historical evaluations with a 2,048-token output budget, Oracle R1 and the first R2 run yield:

BenchmarkR1 pass@1R2 pass@1R1 pass@256R2 pass@256
MATH-50064.55%66.94%97.0%97.2%
Olympiad28.02%29.69%76.2%76.6%

The other recorded R2 runs also show higher pass@1, with small coverage fluctuations. A complementary AIME25 comparison using the same evaluation stack and question-level sampling seeds shows pass@1 increasing from 4.47% to 5.35%, with pass@256 remaining at 53.33%. AIME25 was used for training-time validation. We will present the round-wise results with their evaluation settings and training-resumption configurations.

We are also pursuing a third-round supplementary experiment. Curriculum training has been completed for both arms, with data preparation, executor training, and common evaluation still to be completed.

Mechanistically, additional rounds offer gains when the updated curriculum supplies learnable tasks suited to the current executor. Program execution and consistency checks construct their targets, and within-group reward differences provide task feedback. Existing records show increasingly saturated reward groups as training progresses, motivating an expectation of smaller accuracy gains in later rounds, with broadly stable but fluctuating coverage. Further coverage expansion depends on introducing learnable problem types that the executor has not yet mastered. This is our mechanism-based expectation for R3; the completed two-round comparison provides the current empirical evidence.

AuthorsOpenReview ↗

Additional Results on Third-Round Co-Evolution

Additional results on further co-evolution rounds. In our original response, we noted that the benefit of additional rounds depends on whether the new curriculum continues to provide learnable tasks matched to the current executor. Following up on the third-round experiment mentioned there, we now provide results from a completed, fixed-10-update Oracle R3 experiment.

Starting from the R2 executor, we performed 10 optimizer updates on 128 fixed training examples constructed using the third-round curriculum, with a single random seed, and evaluated the pre-specified step-10 checkpoint. We compared R2 and R3 under the same protocol on a fixed subset of 100 MATH and 100 OlympiadBench questions, with eight responses per question. Results under the original grader are shown below, with R2 on the same evaluation subset serving as the reference for this comparison.

BenchmarkMetricR2R3 (10 updates)Change (pp)
MATHpass@160.625%61.375%+0.750
MATHpass@879.000%77.000%−2.000
OlympiadBenchpass@130.000%29.250%−0.750
OlympiadBenchpass@850.000%51.000%+1.000

The short-run third-round results show small, mixed changes: MATH pass@1 increases by 0.75 percentage points and OlympiadBench pass@8 by 1 percentage point, while the other two metrics decrease slightly. The R1-to-R2 comparison in our original response showed accuracy gains, whereas this R3 comparison shows no consistent further improvement across benchmarks and metrics.

This brings us back to the central question in our original response: whether the new curriculum still offers useful learning opportunities for the current model, rather than simply adding another round of training. Under the mechanism we discussed, further gains depend on whether new tasks expose capabilities the model has not yet mastered while producing differentiated reward feedback. The focus of further co-evolution is therefore to continually provide such learnable tasks so that curriculum updates translate into performance gains, rather than expecting every additional round to improve performance automatically.

Reviewer_qznUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a mathematical self-evolution framework that uses program execution and independent consistency checks to construct training targets. It also examines how target errors and grading failures affect group-relative feedback. Experiments show better high-budget coverage and shorter outputs than an Agent0 reproduction, although the controlled target-only experiments do not establish a downstream performance benefit.

Critical strengths

The paper highlights how incorrect targets can either introduce misleading reward differences or erase useful ones. This distinction provides a useful way to examine supervision failures that label accuracy alone would miss.

Critical weaknesses
  • The main comparison changes target sources, task construction, filtering, and difficulty feedback together, so it is unclear what causes the gains. The target-only experiment leaves downstream performance unresolved, weakening the connection between the proposed analysis and the empirical improvements.

  • AEE’s practical value is not sufficiently demonstrated. The paper does not show that it predicts learning outcomes beyond existing error and reward-collapse statistics, while representation-sensitive grading makes semantic errors difficult to separate from scoring failures. This limits the evidence for the central diagnostic contribution.

Questions for authors
  • Do any existing controlled results isolate the effect of target source from changes in task construction, filtering, and difficulty feedback?

  • Is there evidence that AEE predicts subsequent performance or coverage changes beyond target-error and reward-collapse rates?

2 comments
AuthorsOpenReview ↗

Response to Reviewer qznU: Target-Source Controls and the Diagnostic Value of AEE

Thank you for recognizing the distinction between misleading reward differences and the loss of useful reward distinctions. Your questions help us sharpen the presentation of our controlled experiments and the practical value of AEE. We address them below.

Q1

Yes. Our target-only comparison isolates the effect of target source on the group-relative feedback delivered during training. We will present its design and findings more prominently in the revision.

In the Qwen comparison (§4.5.3 and Appendix M.1), both arms use the same model initialization, training problems and their order, training configuration, and budget. Only the target source changes on conflicting problems:

  • Execution-target arm (E): the target is the output of the program associated with each problem.
  • Voting-target arm (M): model-generated answers are clustered by numerical value, and the answer represented by the largest cluster is used as the target.

Problems on which the two sources agree retain the same target. Fixing the training set and curriculum schedule allows us to examine the contribution of target source under otherwise shared training conditions.

The intervention produces a clear difference in the observed reward structure. Across 2,542 conflict-group exposures per arm, E produces 2,435 all-positive groups and 107 mixed-reward groups, while M produces 2,537 all-negative groups and 5 mixed-reward groups. Since group-relative training uses within-group reward differences to distinguish responses, these results directly show how target source affects both reward acceptance and the availability of differentiated task feedback.

A complementary fixed-policy, same-response replay experiment examines the direct scoring effect of target substitution. Holding model outputs fixed, changing only the grading target turns many previously mixed-reward groups into all-negative groups. The two experiments provide complementary evidence: replay isolates how target substitution changes the rewards assigned to identical responses, while the training comparison shows the feedback patterns that arise during actual learning.

We have also completed target-only training and evaluation on Llama. Execution-target training improves MATH performance over initialization, extending the training evidence to another model family. We will present the downstream comparisons with their grading-sensitivity analyses alongside the training-side findings.

Together, these controlled results establish an independent effect of target source on training feedback structure in the evaluated setting. The revision will bring the experimental design, quantitative feedback results, and replay analysis together, making the role of target design within the overall method easier to assess.

Q2

Our current empirical evidence demonstrates the additional joint diagnostic information supplied by AEE. Target-error rates describe target quality, and reward-collapse rates describe the availability of within-group reward differences. AEE connects these properties at the level of individual training groups, identifying where incorrect targets coincide with active feedback.

We aligned the target-correctness review with the original reward records group by group. Among the 42 training groups with resolved target-correctness labels under the model-assisted review:

  • 13 groups have incorrect targets.
  • 16 groups have active rewards.
  • 8 groups have both incorrect targets and active rewards.

The two marginal counts alone allow their intersection to range from 0 to 13 groups; the joint analysis identifies 8. AEE therefore supplies measurable additional information: where incorrect targets occur in groups that still produce within-group reward distinctions, and how frequently this exposure occurs.

This information guides the next diagnostic step. For incorrect-target groups with active rewards, reference replay examines how the training rewards distinguish responses relative to reference feedback. For incorrect-target groups with collapsed rewards, replay examines whether useful reference distinctions have been erased. Our fixed-policy, same-response experiments show concretely how target substitution can turn mixed-reward groups into all-negative groups.

AEE locates the overlap between target errors and reward activity, while reference replay reveals how reward distinctions change. Together, they connect target quality to the feedback actually delivered to the learner and identify specific groups for closer analysis.

The revision will include the full joint table, annotation coverage, and unresolved cases, with the findings presented as feedback diagnostics within the saved sample. This will make AEE’s additional information and its practical use in analyzing supervision more concrete.

Thank you for the clear response. It helps clarify some of my concerns. I will keep my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.