- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies supervision in two rounds of mathematical co-evolution using Qwen3-4B-Base. Its execution-grounded framework generates problem-program pairs, executes programs to obtain targets, and filters them using fresh statement-only implementations before ADPO training. It introduces Active Error Exposure, the joint occurrence of an incorrect target and a nonconstant reward group, alongside paired reference replay. Experiments report higher finite-budget coverage and shorter outputs than an Agent0 reproduction on MATH-500, an Olympiad subset, and AIME25, with longer-output checks on Olympiad. Additional probes examine target substitution and representation-sensitive grading. The target-only training experiment reports altered feedback, while its downstream effect remains unresolved.
Critical strengths
-
Table 1 reports positive R2 pass@256 gaps on MATH and Olympiad in all three recorded pairs under both extraction rules. The native 16K experiment further reports Olympiad pass@32 of 64.15% versus 56.87%, at mean output lengths of 786 versus 9,905 tokens (Tables 19-21). This same-stack native comparison supports a practically meaningful coverage and output-cost advantage for the evaluated checkpoints, not merely an improvement in pass@1.
-
The fixed-policy, same-response replay demonstrates that collapsed rewards can erase distinctions present under reference rewards. In Table 4, low-reach targets yield five mixed groups versus 41 for reachable targets, and turn 35 of 40 source-mixed groups all-negative (Section 4.5.3). Appendix H's equivalent-target example makes the distinction between semantic errors and representation-sensitive grading particularly useful.
Critical weaknesses
-
The interpretation of grader-measured coverage as mathematical coverage needs a targeted validity check. Appendix H reports seven versus zero accepted responses for mathematically equivalent target encodings, whereas the main benchmark comparisons retain mathruler grades (Appendices F, K, and L). First-box and last-box sensitivity tests vary extraction, not the underlying equivalence decisions, and the saved-record audits do not establish their semantic accuracy. Given the different output behaviors across methods, differential grading errors remain an unresolved explanation for part of the coverage gap. A blinded audit of answers on questions covered by only one method would help assess this concern.
-
AEE's empirical value beyond marginal error and collapse rates remains insufficiently demonstrated. Equations (1) and (6) define the statistic and its conditional-expectation decomposition, but the results do not report joint target-error/activity estimates for the main training runs. Table 4 measures grader reachability, while Table 23 demonstrates reward saturation in the target-only intervention, rather than a comparison showing what the joint statistic reveals beyond those marginals. These findings support the feedback observations, but leave the added diagnostic contribution underdeveloped. A bounded analysis of the existing reviewed groups could strengthen this contribution without requiring proof that AEE causes downstream gains.
Questions for authors
-
On a blinded, stratified sample of s2 benchmark questions covered by only one method, can independent mathematical checking verify the accepted answers and inspect rejected equivalent answers from the other method? Reporting the resulting corrections would help determine whether grading asymmetry materially affects the coverage advantage.
-
Can the existing reviewed training groups support a joint table of target correctness and reward activity, with AEE, annotation coverage, and unresolved cases reported explicitly? Within that selected sample, what diagnostic conclusion differs from one based only on marginal target-error and collapse rates?
2 comments
Thank you for the detailed responses and additional experiments. It helps clarify some of my concerns. I will keep my current score.
Response to Reviewer zbWY: Coverage Audit and Joint AEE Analysis
Thank you for your detailed assessment of the same-stack coverage and output-cost comparisons and the fixed-policy replay results. Following your suggestions, we have reviewed candidates from all one-sided coverage questions and assembled a joint analysis of target correctness and reward activity.
Q1
We have reviewed 811 candidate responses spanning all 74 questions covered by only one method in the s2 comparison. These comprise 66 questions originally classified as covered only by our method and 8 covered only by Agent0. The review checks accepted answers and examines potentially overlooked correct responses from the other method.
For newly reviewed candidates, complete responses with method identities concealed were examined in two separate contexts of the same model, and the findings were combined with existing review records. We retain both a legacy-compatible criterion and a stricter criterion requiring explicit confirmation that the final answer was not subsequently retracted.
The review identifies concrete corrections to coverage assignments. Under the legacy-compatible criterion, correct baseline responses were found on 10 questions originally classified as covered only by our method; the stricter criterion confirms 9 of these. No additional jointly covered questions were confirmed in the reverse direction.
To quantify the effect of these identified corrections, we perform a conditional sensitivity calculation: reclassify these 10 questions as covered by both methods, keeping all other original grading decisions unchanged.
Under this conditional calculation, the one-sided coverage difference remains in our favor after incorporating the identified shared-coverage corrections. The table makes the effect of these false negatives explicit. The review covers 811 candidates from the full pool of 37,888 responses; the remaining original grades are retained in this calculation. The revision will report the candidate-selection rules, model-assisted review procedure, review scope, and unresolved cases alongside this table.
The audit highlights a broader methodological challenge for self-evolution pipelines that rely on automated grading: mathematically equivalent answers can receive different scores, affecting both training rewards and measured coverage. This makes semantic-equivalence assessment and feedback diagnostics important directions for further research. We will discuss this shared challenge alongside the concrete grading asymmetries identified in our audit.
Q2
We have aligned target-correctness review results with the original reward records and constructed the requested joint analysis. Of 72 saved training groups, 42 have resolved target-correctness labels under the current model-assisted review. Annotation coverage and AEE within the resolved-label subset are:
AEE uses each method's resolved-label group count as its denominator. Within this subset, our method exhibits lower active error exposure. The joint counts show how target correctness and feedback activity intersect:
Of Agent0's 7 resolved incorrect-target groups, 6 have active rewards; of our method's 6 resolved incorrect-target groups, 2 have active rewards. The two methods have similar numbers of incorrect targets in this subset, distributed across different feedback states. The joint table locates which incorrect targets still produce task-reward distinctions used to differentiate responses.
The additional information is also visible in the pooled counts: the 42 resolved groups contain 13 incorrect-target groups and 16 active-reward groups. These marginal counts allow their intersection to range from 0 to 13 groups; the group-level linkage identifies 8. AEE connects target quality with observed reward structure, locating supervision issues in specific feedback states. These groups can then be examined through reference replay to determine how the reward distinctions change.
The revision will include these joint counts, annotation coverage, unresolved cases, and subset AEE, presenting the findings as diagnostics within the resolved-label subset. This makes the practical information added by the joint statistic directly inspectable.