- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates reference-relative mathematical novelty in math-pretrained and post-trained models from the DeepSeek and Qwen families. Using 400 CreativeMath problems with supplied reference solutions, it finds that the tested post-trained models achieve higher automated correctness acceptance but lower accepted-and-novel response rates. The paper further examines token likelihood, high-entropy segment coverage, and departures from tokenwise argmax predictions through self-scoring and cross-scoring. These analyses show that several diagnostic comparisons depend on the scorer, likelihood threshold, and outcome denominator. The work is observational and does not isolate the causal effect of RLVR.
Critical strengths
Strengths
-
A meaningful evaluation question. Distinguishing answer correctness from the ability to produce alternative valid solutions is important. The paper examines a dimension of mathematical reasoning that accuracy alone does not capture.
-
Useful cross-scoring design. Scoring the same responses with multiple checkpoints helps distinguish properties of the generated text from the probability distributions assigned by particular scorers. This is more informative than relying exclusively on self-scoring.
-
Transparent sensitivity analysis. The paper explicitly reports cases where changing the outcome denominator, likelihood threshold, or scorer reverses a comparison. It also acknowledges that token likelihood is not calibrated confidence and that entropy windows do not necessarily correspond to semantic reasoning branches.
Critical weaknesses
-
The validity of the central outcome measure is insufficiently established. The correctness prompt asks whether a response reaches the same result as the references, rather than explicitly verifying the mathematical reasoning. Since reference solutions are provided to the generator, invalid reasoning that reproduces the reference answer may pass this check. Meanwhile, the novelty rubric allows differences in assumptions or complexity, which need not constitute a valid alternative solution to the original problem. Without expert validation and judge-agreement analysis, the reliability of the accepted-and-novel labels remains uncertain.
-
The checkpoint comparisons do not isolate the role of RLVR. The Math–RLVR comparisons jointly reflect SFT, reinforcement learning, and other checkpoint differences. DS-Inst is available but evaluated only as a scorer, leaving a particularly useful SFT-only generation baseline untested. Furthermore, the described training pipelines include learned reward signals, limiting the scope of conclusions about verifiable-reward training specifically. Although the paper acknowledges these limitations, the evidence answers a narrower question than the title suggests.
-
Response retention and unequal sampling complicate the behavioral comparison. Responses reaching the 1,024-token limit or attempting API calls are regenerated, with up to 20 retries. This procedure can alter the distribution of retained solutions differently across models. The reported rates therefore do not establish performance over unfiltered attempts or under a matched generation budget. Results with larger token limits and explicit accounting for all attempts are needed.
-
The statistical evidence is primarily descriptive. Multiple responses share the same problem, and problems with more retained responses receive greater weight. The paper does not provide problem-level paired uncertainty estimates for the main behavioral differences. Consequently, the robustness of these differences across problems and sampling variation is unclear.
-
The probability diagnostics provide limited explanatory evidence. High observed-token likelihood, low off-argmax ratios, and relative entropy-window coverage do not directly measure mathematical novelty. The sensitivity analyses are informative, but they also show that these statistics do not provide a stable, unified explanation for the novelty gap. The paper lacks semantic validation or targeted interventions connecting these diagnostics to alternative solution strategies. Using “overconfidence” in the title remains potentially misleading despite the operational qualification in the text.
Questions for authors
-
Can you provide a blinded expert audit of generated solutions, checking both mathematical validity and substantive novelty under the original problem assumptions? How frequently do the automated labels disagree with expert judgments?
-
How robust are the main results to stricter correctness prompts, requiring both judges to agree on novelty, and excluding novelty judgments based solely on changed assumptions or increased complexity?
-
Why is DS-Inst not evaluated as a generator? Does the novelty decline occur primarily between Math and SFT, or between SFT and RL?
-
Do the behavioral differences persist when all initial attempts are included, generation budgets are matched, and the output-token limit is increased? Please report model-specific rejection and retry rates.
-
Can you report equally weighted, paired problem-level comparisons with confidence intervals that account for multiple responses per problem?
-
Do the token diagnostics predict novelty after controlling for correctness, response length, formatting, and problem identity? Can a targeted intervention based on these diagnostics improve the rate of valid alternative solutions?