acceptodds
Withdrawn from ICLR 2027

Can RLVR Models Think More Creatively? A Study of Overconfidence and High Entropy Segment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advances in Large Language Models (LLMs) are often attributed to improved reasoning abilities through Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR). Although RLVR models perform strongly on mathematical benchmarks, their ability to generate creative solutions remains an open question. In this work, we compare RLVR models and math-pretrained models from the DeepSeek and Qwen families on mathematical problem solving with supplied reference solutions. RLVR models yield fewer responses judged both correct and novel, while their self-scoring high lower-tail token likelihood (HL) rates exceed those of the Math models by 15.85 and 74.70 percentage points, respectively. HL operationalizes token-level Overconfidence (OC) in this study, rather than calibration error. Motivated by the hypothesis that SFT and policy optimization reshape token probabilities, we analyze High Entropy Segments (HES), defined as windows whose mean token entropy exceeds the response mean. Cross-scoring reveals scorer-dependent HES coverage and overlap, while the association between HL and novelty depends on the outcome denominator and likelihood threshold. Furthermore, RLVR models have lower off-argmax ratios under self-scoring, accompanied by a higher fraction of positions with top-token probability at least 0.80. These observational findings characterize the tested RLVR models and suggest directions for improving creative generation without isolating causal training effects.

Primary area:
generative models
Keywords:
LLM, RLVR, Reasoning Models, Reinforcement Learning, Policy Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_P7vZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates reference-relative mathematical novelty in math-pretrained and post-trained models from the DeepSeek and Qwen families. Using 400 CreativeMath problems with supplied reference solutions, it finds that the tested post-trained models achieve higher automated correctness acceptance but lower accepted-and-novel response rates. The paper further examines token likelihood, high-entropy segment coverage, and departures from tokenwise argmax predictions through self-scoring and cross-scoring. These analyses show that several diagnostic comparisons depend on the scorer, likelihood threshold, and outcome denominator. The work is observational and does not isolate the causal effect of RLVR.

Critical strengths

Strengths

  1. A meaningful evaluation question. Distinguishing answer correctness from the ability to produce alternative valid solutions is important. The paper examines a dimension of mathematical reasoning that accuracy alone does not capture.

  2. Useful cross-scoring design. Scoring the same responses with multiple checkpoints helps distinguish properties of the generated text from the probability distributions assigned by particular scorers. This is more informative than relying exclusively on self-scoring.

  3. Transparent sensitivity analysis. The paper explicitly reports cases where changing the outcome denominator, likelihood threshold, or scorer reverses a comparison. It also acknowledges that token likelihood is not calibrated confidence and that entropy windows do not necessarily correspond to semantic reasoning branches.

Critical weaknesses
  1. The validity of the central outcome measure is insufficiently established. The correctness prompt asks whether a response reaches the same result as the references, rather than explicitly verifying the mathematical reasoning. Since reference solutions are provided to the generator, invalid reasoning that reproduces the reference answer may pass this check. Meanwhile, the novelty rubric allows differences in assumptions or complexity, which need not constitute a valid alternative solution to the original problem. Without expert validation and judge-agreement analysis, the reliability of the accepted-and-novel labels remains uncertain.

  2. The checkpoint comparisons do not isolate the role of RLVR. The Math–RLVR comparisons jointly reflect SFT, reinforcement learning, and other checkpoint differences. DS-Inst is available but evaluated only as a scorer, leaving a particularly useful SFT-only generation baseline untested. Furthermore, the described training pipelines include learned reward signals, limiting the scope of conclusions about verifiable-reward training specifically. Although the paper acknowledges these limitations, the evidence answers a narrower question than the title suggests.

  3. Response retention and unequal sampling complicate the behavioral comparison. Responses reaching the 1,024-token limit or attempting API calls are regenerated, with up to 20 retries. This procedure can alter the distribution of retained solutions differently across models. The reported rates therefore do not establish performance over unfiltered attempts or under a matched generation budget. Results with larger token limits and explicit accounting for all attempts are needed.

  4. The statistical evidence is primarily descriptive. Multiple responses share the same problem, and problems with more retained responses receive greater weight. The paper does not provide problem-level paired uncertainty estimates for the main behavioral differences. Consequently, the robustness of these differences across problems and sampling variation is unclear.

  5. The probability diagnostics provide limited explanatory evidence. High observed-token likelihood, low off-argmax ratios, and relative entropy-window coverage do not directly measure mathematical novelty. The sensitivity analyses are informative, but they also show that these statistics do not provide a stable, unified explanation for the novelty gap. The paper lacks semantic validation or targeted interventions connecting these diagnostics to alternative solution strategies. Using “overconfidence” in the title remains potentially misleading despite the operational qualification in the text.

Questions for authors
  1. Can you provide a blinded expert audit of generated solutions, checking both mathematical validity and substantive novelty under the original problem assumptions? How frequently do the automated labels disagree with expert judgments?

  2. How robust are the main results to stricter correctness prompts, requiring both judges to agree on novelty, and excluding novelty judgments based solely on changed assumptions or increased complexity?

  3. Why is DS-Inst not evaluated as a generator? Does the novelty decline occur primarily between Math and SFT, or between SFT and RL?

  4. Do the behavioral differences persist when all initial attempts are included, generation budgets are matched, and the output-token limit is increased? Please report model-specific rejection and retry rates.

  5. Can you report equally weighted, paired problem-level comparisons with confidence intervals that account for multiple responses per problem?

  6. Do the token diagnostics predict novelty after controlling for correctness, response length, formatting, and problem identity? Can a targeted intervention based on these diagnostics improve the rate of valid alternative solutions?

Reviewer_YafSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether LLMs trained with Reinforcement Learning with Verifiable Rewards (RLVR) can generate more creative solutions in mathematical problem solving, and what relationship this capability has with the token probability distribution characteristics of the models. Based on the CreativeMath benchmark, the authors select four 7B models from the DeepSeek and Qwen model families as generators and conduct experiments on 400 mathematical problems. For each problem, at most two reference solutions are provided, and the models are required to generate different solution methods, with Gemini-1.5-Pro and o4-mini used as judges for evaluation. The experimental results show that in both model families, RLVR models exhibit a higher correct acceptance rate than math-pretrained models, but the rate of correct and novel generations instead decreases. The authors also conduct further analyses to provide additional insights. Overall, this paper provides an empirical study on the relationship among mathematical reasoning correctness, the reference-relative novelty, and token probability distributions, and through further analysis experiments, reveals the limitations of relying solely on self-scoring probability profiles to explain model creative performance.

Critical strengths
  • This paper investigates whether improving the correctness of mathematical problem solving through RLVR is accompanied by an enhanced ability of the model to generate different solution approaches, which is an interesting and valuable question. Current evaluations of mathematical reasoning typically emphasize the accuracy of final answers, whereas this paper attempts to distinguish solution diversity from correctness. This research direction helps to more comprehensively understand the capability boundaries of mathematical reasoning models.
  • Results on a variety of experimental settings and models further strengthen the conclusions. This paper observes consistent core phenomena across two different model families, and its conclusions have certain cross-model support.
  • This paper introduces multiple complementary diagnostic metrics, providing a richer analytical perspective than a single average entropy. These metrics describe the model's probabilistic behavior from different angles, helping to enhance understanding of the conclusions.
Critical weaknesses
  • The reliability of the evaluation mechanism used in the paper has not yet been fully validated, which may affect the paper's core conclusions. The authors adopt a lenient judgment rule for novelty. They use two judges to assess correctness, but only required at least one judge model to recognize novelty in order to classify a response as Crea. Additionally, the current paper also lacks quantitative validation of the reliability of the judge model's results. This paper used Gemini-1.5-Pro and o4-mini, which may have a considerable performance gap compared with the latest frontier models.
  • This paper takes the differences between the RLVR model and the Math model as its main research subject, but the differences between these two types of models are not limited to RLVR training. Taking DS-Inst as an example, although the authors examine DS-Inst, they don't use it as a generative model for creativity evaluation. Therefore, this paper can't determine whether the observed decline in novelty mainly occurs in the SFT stage or in the subsequent RL stage.
  • The definition of HES mainly captures the relative entropy distribution, and there is still a lack of evidence proving that it has a substantive connection with mathematical creativity or reasoning branching. In a single response, if the average token entropy of a window is higher than the overall average entropy of that response, then that window will be marked as a high entropy segment. However, HES does not indicate that a certain segment of reasoning has absolutely high uncertainty, but only shows that its average entropy is higher relative to the overall mean of the same response. Even if the absolute entropy levels of two models differ significantly, they may still have similar HES coverage.
  • The results indicate that HL doesn't exhibit cross-model consistency in the current experiments. For example, in Table 3, the Cre of the HL group in DS-RLVR is lower than that of the non-HL group, whereas the Cre of the HL group in QW-RLVR is instead higher than that of the non-HL group. Additionally, this difference in results is also clearly affected by the choice of threshold. However, no stable connection has been established between the current results and the conclusions, which weakens the reliability of the conclusions in this paper.
Questions for authors

Please refer to the weaknesses above.

Reviewer_MKTTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper compares on the creativity of language models during each stage of training (pre-trained, mid-trained, and RLVR'ed).

Critical strengths
  • The paper studies an interesting and important topic of the creativity of RLVR models.
Critical weaknesses
  • The paper is not written well. Very hard to understand what the authors is trying to convey. I had couple of questions simply when reading the abstract. There are also too few citations that allows the readers to put their work into context. To be more specific: I don't understand what does "self-scoring high lower-tail token likelihood" mean in the abstract. In the intro, the authors wrote: "...despite higher correctness-acceptance rates" can be simply written as "despite higher overall correctness rate".

  • The core of the studies relies on the "novel judge" being accurate, but there is no study on the choice / ablations on how did the authors designed the novelty judge. They treated the 2 judges response (gemini 1.5 pro / o4 mini) as groundtruth. I would suggest try to do simple human annotation on a fixed set, or use a stronger model to see if the judges are correct. If the core piece of the study is hand-wavy, then it is hard to build upon it.

  • The author presents a lot of analysis, but didn't provide main takeaways, nor thought about the implications or how can the community adopt their analysis.

Questions for authors

suggestions: what is "self-scoring high lower-tail token" in the abstract? it is not clearly defined nor a commonly used term in the literature. I would suggest defining it first.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.