acceptodds
Under review as a conference paper at ICLR 2027

Capability Self-Assessment in Large Language Models

Abstract

The ability to recognize one’s own limitations and decide whether to solve a problem or seek help is fundamental to reliable intelligent systems. Yet modern large language models tend to overestimate their competence and attempt queries they cannot solve. We study how models can learn Capability Self-Assessment (CSA) while preserving their problem-solving ability. Because the same model both assesses and solves queries, self-assessment training can change the capability it must recognize. We systematically compare supervised fine-tuning (SFT) and reinforcement learning (RL), evaluating each trained model’s self-assessment against its updated capability and measuring how well it retains its solving ability. Across model families, scales, and domains, RL provides the strongest assessment–retention trade-off. We also find that CSA transfers to the out-of-distribution settings we evaluate, and demonstrate its practical value for local–cloud routing at inference time and targeted data selection during training.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Capability Self-Assessment, Large Language Models, Capability Awareness, Model Delegation, Self-Knowledge

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WpnKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

In this paper, the author investigates Capability Self-Assessment (CSA), a task that aims to predict whether a large language model (LLM) can correctly solve a given query before generating a response. To achieve this, the authors adopt a straightforward approach to constructing labeled training data and apply two established post-training techniques, supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR), to train the model for CSA. Through empirical comparisons, the authors conclude that RLVR achieves better self-assessment performance while more effectively preserving the model's original problem-solving capabilities.

Critical strengths
  1. The paper is well-written and clearly organized. The proposed data curation strategy and training pipeline are straightforward and easy to follow.

  2. The paper conducts extensive experiments across different model families, scales, and datasets, providing empirical evidence for the generalizability of its findings across the evaluated settings.

Critical weaknesses
  1. The technical novelty of this work is limited. The paper does not introduce any fundamentally new training methods but instead directly applies existing techniques, such as SFT and RLVR, to the CSA task. Moreover, the task of predicting a model's own correctness or capability has already been extensively explored in prior research on uncertainty estimation and confidence prediction. The main experimental conclusions are also largely expected and provide limited new insights into the underlying problem.

  2. The motivation for directly fine-tuning the original task-performing LLM to conduct CSA is not sufficiently justified. Since CSA is essentially formulated as a binary classification problem, a simpler and smaller auxiliary model could potentially perform the same task without modifying the original LLM. Such a design would completely preserve the original model's problem-solving capabilities while potentially achieving comparable CSA performance. Therefore, the paper's extensive investigation into capability retention appears to address a problem introduced by its own methodological choice. A more convincing justification or comparison with lightweight standalone predictors is needed to establish the necessity and practical advantages of the proposed approach.

  3. Several important details of the data curation process remain unclear. For example, when constructing rationale-augmented training samples for SFT, how do the authors ensure that the generated rationales are logically consistent with the corresponding CSA labels? This concern is particularly relevant to the teacher-generated rationale setting, where a stronger model generates rationales conditioned on labels derived from the original model's performance. The resulting rationales may not faithfully reflect the original model's actual capabilities.

Additionally, it is unclear whether the authors apply any filtering to retain only examples for which the original model makes correct CSA predictions. If such filtering is performed, it could introduce selection bias, as the training process may primarily reinforce knowledge or assessment patterns already captured by the original model rather than improve its ability to recognize previously unknown capability limitations. Further clarification of the data filtering and quality-control procedures would help assess the validity of the training process and experimental conclusions.

Questions for authors
  1. Why should capability retention be considered a central challenge of CSA rather than a consequence of the chosen implementation? If the CSA predictor and the problem-solving model were separated, what fundamental limitation would prevent this alternative from achieving the same objective?

  2. What new insights do the experimental findings provide beyond the expected differences between SFT and regularized RL training?

4 comments
AuthorsOpenReview ↗

Response to Reviewer WpnK (1/2): separate predictor, novelty

Thank you for the review. Your main soundness concern is answered by a comparison that's already in the paper, which we should have made easier to find.

Q1, W2: Why train the model itself instead of a small separate predictor? It's the right question, and the paper tests it (Table 3, and Section 4.3, "Why retention matters"). Two options in Table 3 leave the model untouched:

  • a probe, which is a classifier on the model's own frozen representation of the question: 0.70 / 0.71 M-F1 on Math (Qwen3-1.7B / 4B) and 0.55 / 0.56 on Science;
  • LoRA adapters: 0.73 / 0.62 on Math and 0.51 / 0.54 on Science, keeping 99–100% of capability.

Training the model itself with RLVR gets 0.84 / 0.85 on Math and 0.66 / 0.67 on Science.

So a separate predictor works, just less well: the better of the two is 0.11–0.14 M-F1 behind RLVR in every setting. The most accurate assessor turned out to be the model itself. So keeping its abilities intact isn't a problem we created. It's something we have to check. It's also simpler to deploy, since one model does both jobs.

W1: No new training method. That's correct, and intended. The paper is an analysis and doesn't claim a new method. Our contributions are an evaluation of self-assessment together with its effect on the model, a comparison of training-free methods, SFT and RL, and two uses of the result. We used standard SFT and GRPO on purpose, so the comparison reflects the training signal and not a new algorithm. The DFW warm-up is only an implementation detail, and we don't claim it as a contribution.

Q2: What's new beyond "regularized RL beats SFT"? The general response goes through this. In short:

  • training the model changes what it's trying to predict, so the usual evaluation can make a model that's getting worse look better.
  • what it learns is about itself rather than about general difficulty, even on a domain it never saw.
  • what it can solve stays the same, query by query.
AuthorsOpenReview ↗

Response to Reviewer WpnK (2/2): rationales, data filtering

W3: Do the SFT rationales actually match the labels? They argue for their label by construction, since each one is written with the label given. We used two kinds on purpose. A stronger teacher writes better rationales, but they may not reflect what the target model can actually do. Self-written rationales are closer to the model's own view but lower in quality. Having both lets us see whether that trade-off matters, and in Table 2 neither consistently wins: each has the higher M-F1 in 6 of the 12 settings. We didn't check whether the rationales reflect the model's real reasoning, and the teacher's can't, because a different model writes them. That's a real limitation of training on rationales. It's also part of why we include label-only SFT, which has no rationale, and RLVR, where only the final decision is rewarded and the reasoning is never supervised.

W4: Is the training data filtered toward examples the model already gets right? No. No method filters the training data. Every method trains on the same full training set with the same labels. Task-mixed SFT adds solving examples on top, but nothing is removed. For the rationale variants, we write one rationale for each possible label and keep the one that matches, so no question is dropped. The one difference is RLVR's warm-up (DFW). It does pick a subset, but by a different rule: it keeps questions where the model's 16 sampled decisions disagree, not questions where it's correct. It uses that subset only for a short warm-up (42–152 steps) and then trains on everything.

In fact, that subset leans the opposite way from what you describe. On Math, 62–70% of it is labeled DELEGATE, compared with 20–34% of the full set, so it focuses on limits the model doesn't yet recognize. And SFT trained on the same subset first (DFW-matched SFT, Table 3) still falls short of RLVR: 0.69 / 0.71 vs. 0.84 / 0.85 on Math.

Thanks again. We hope this clears up the soundness concerns, and we'd be grateful if you'd consider it in your final assessment.

We appreciate the authors' detailed response. The additional clarifications regarding data construction and experimental settings have adequately addressed my concerns about the experimental setup. However, my primary concern regarding the paper's contributions remains. The current work focuses primarily on empirically comparing two training paradigms, SFT and RL, against several baselines, without providing a sufficiently insightful analytical framework or explanation of the underlying mechanisms that drive their performance differences. Developing a stronger theoretical motivation or well-grounded hypotheses, supported by comprehensive empirical analyses that investigate the fundamental differences between these training signals, would substantially strengthen the paper's contributions.

AuthorsOpenReview ↗

Further Clarification of Our Core Contribution and CSA Formulation

We thank the reviewer for acknowledging our clarifications. We respectfully emphasize that our primary contribution is not merely an empirical comparison between SFT and RL, but the proposed formulation of Capability Self-Assessment (CSA) as a joint learning problem that explicitly considers capability preservation (Section 3.1, Eq. 1).

Unlike conventional self-assessment approaches, which focus primarily on improving assessment accuracy, our formulation highlights a fundamental challenge: when the same model performs both self-assessment and problem-solving, training the assessment policy can alter the very capability being assessed. Consequently, improved assessment accuracy against fixed labels does not necessarily indicate better self-awareness. This motivates our formulation of CSA as optimizing assessment quality subject to preserving the model's underlying problem-solving capability.

From this perspective, comparing SFT and RLVR is not the goal itself, but rather a systematic investigation of how different training signals satisfy this joint objective. Our analyses of training dynamics (Appendix D.3), capability-preserving baselines (Table 3), and query-level capability stability (Section 4.3) provide empirical evidence supporting this formulation.

We agree that a deeper theoretical understanding of these training signals would further strengthen the work. Nevertheless, we believe identifying and formalizing the coupling between self-assessment learning and capability preservation constitutes an important contribution, beyond a conventional empirical comparison of training paradigms. We also believe this contribution is consistent with ICLR's emphasis on identifying meaningful research problems and advancing relevant knowledge through rigorous empirical investigation.

Reviewer_JHkcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies Capability Self-Assessment: training a model to decide, before answering, whether to solve a query itself or delegate it to a stronger model. Labels come from sampling the initial model 5 times per query. The paper compares three SFT variants and RLVR, and evaluates each trained model against freshly re-probed labels, along with how much solving ability it keeps. The main finding is that RLVR gives the best assessment while keeping solving ability, across Qwen3 and OLMo2 models on math and science. The paper also shows transfer to other domains and two applications: local–cloud routing and data selection.

Critical strengths

The evaluation is careful. Judging each trained model against its own re-probed capability is the right setup, and the paper adds solid checks: label churn compared with sampling noise, within-source permutation tests, and a comparison with OLMoE to show the assessment is model-specific.

Critical weaknesses
  1. The main finding is mostly expected, and the “retention” problem looks partly created by weak SFT baselines. Training a model to predict whether it can answer before answering was already done with P(IK) [1], and RL-trained confidence and help-seeking exist too. That RL forgets less than SFT is also known [2]. On top of that, RLVR here uses a very small learning rate, KL, and only tens to about a hundred steps, so it barely changes the model. Meanwhile SFT_Label loses 90%+ of math ability after one epoch on about 2,000 examples, which looks more like format overfitting than a real cost of learning self-assessment. The extra baselines in Table 3 help, but the gaps to LoRA and task-mixed SFT are smaller and only tested on two small models.
  2. RLVR is not clearly better than strong baselines, and the main text shows the best case. Figure 3 only shows Qwen3-4B Math, where RLVR (0.80) slightly beats self-consistency (0.78). But in Table 16, on Science, self-consistency and semantic entropy reach 0.75–0.79, while RLVR gets 0.64–0.69. On math, a large part of the score comes from telling GSM8K, MATH500 and AIME apart: the within-source shuffle null is already 0.68–0.85, and RLVR beats it by only 3–10 points. The cost advantage of not generating answers is real, but the paper should present this as a cost–accuracy trade-off, not as RLVR being the best method.

Reference: [1] Kadavath, S., et al. Language Models (Mostly) Know What They Know. arXiv:2207.05221, 2022. [2] Shenfeld, I., Pari, J., and Agrawal, P. RL’s Razor: Why Online Reinforcement Learning Forgets Less. ICLR 2026.

Questions for authors
  1. Why does SFT_Label lose almost all math ability after one epoch on about 2,000 examples? If this is format overfitting, is capability retention really the core difficulty of this problem?
  2. Can you report all model–domain settings against self-consistency and semantic entropy in the main text, and compare them on accuracy versus cost?
3 comments
AuthorsOpenReview ↗

Response to Reviewer JHkc (1/2): retention and SFT baselines

Thank you for the careful reading, and for noting the re-measured labels and the controls.

Q1: Why does label-only SFT lose almost all its math ability? Is retention really the hard part? We agree it's mostly format collapse. Label-only SFT trains every weight (learning rate 1e-5) to output a single token, and the model learns to say almost nothing. We included it because it's the recipe that earlier work on trained abstention uses [1, 2], not to argue that self-assessment itself costs capability.

And no, retention isn't the hard part. It's a precondition. LoRA SFT keeps 99–100% of capability but only reaches 0.51–0.73 M-F1, so the hard part is telling solvable questions from unsolvable ones. Retention matters for a different reason. Any change to the model changes the labels it's trying to predict, so checking against old labels can make a model that's getting worse look like it's getting better (general response, point 1). Gentler methods also change the model. SFT with a smaller learning rate keeps 72.4% of capability on Qwen3-1.7B Math and 84.5–88.6% on Science, and SFT with a KL penalty keeps 90.4–92.0% on Science.

W1: "RLVR barely changes the model." Yes, and that's what we find interesting. A small update takes Qwen3-4B Math from 0.51 to 0.80 M-F1, while what the model can solve stays the same query by query, within sampling noise. SFT with the same budget doesn't get there. In Table 3, every method gets the same batch size and number of updates (31 on Math, 47 on Science), the smaller-learning-rate SFT uses RLVR's own rate of 1e-6, and one variant adds a KL penalty. No SFT variant goes above 0.75 on Math or 0.65 on Science, while RLVR gets 0.84–0.85 and 0.66–0.67.

You're right that Table 3 covers two models. It does cover both Math and Science, with several SFT variants and three runs each. Running it for every model isn't feasible with our compute, but if there's a specific model and task you'd like to see, we'd be happy to add it.

AuthorsOpenReview ↗

Response to Reviewer JHkc (2/2): strong baselines, cost, benchmark mix

W2, Q2: Show every setting against self-consistency and semantic entropy, as accuracy vs. cost. Good suggestion. We'll move the full comparison (Table 16) into the main text and present it as a trade-off. Here it is for all four settings. Each cell is M-F1, with tokens per question in parentheses. Semantic entropy is within 0.03 of self-consistency everywhere, so we only show self-consistency.

SettingRLVRSelf-consistency, 3 samples5 samples10 samplesBest single-answer method
Qwen3-4B Math0.80 (151)0.78 (2,255)0.78 (3,799)0.74 (7,395)0.76
Qwen3-4B Science0.64 (185)0.76 (1,707)0.75 (2,847)0.77 (5,763)0.63
OLMo2-7B Math0.89 (402)0.81 (1,470)0.81 (2,417)0.84 (4,774)0.72
OLMo2-7B Science0.69 (633)0.71 (1,448)0.76 (2,388)0.79 (4,804)0.68

In short: on Math, RLVR is the most accurate and generates 3.7–14.9× fewer tokens than self-consistency with three samples. On Science, the sampling methods are more accurate by 0.07–0.13 with 5 or 10 samples, but they use 3.8–31× as many tokens and have to write full answers before deciding. Cost can go lower still. If the model writes its decision before its analysis, it can stop after a few tokens as shown in our ablation (Table 14). We'll describe RLVR as the strongest method that decides before answering, not the strongest overall. We don't treat RLVR as our contribution, so we have no problem with costlier baselines matching or beating it in some settings. That isn't the point of the paper.

There's a reason self-consistency does better on Science and not on Math. The Science label asks "are at least three of five answers correct?", and self-consistency asks "do the answers agree?" With ten options, agreeing and being correct go together closely, so self-consistency is almost a second run of the labeling process, just without the answer key. On Math the label is "any of five correct", which agreement doesn't capture, so RLVR comes out ahead. We were aware of this when we set up the study, and we kept two different labeling rules on purpose. In an analysis paper, it's useful to see how each method behaves under different definitions of "can solve", and this contrast is one example of what that shows.

W2: How much of the Math M-F1 is just telling GSM8K, MATH500 and AIME apart? We tested exactly this (Section 4.4). We kept how often each model delegates on GSM8K, MATH500 and AIME, but randomly reshuffled which questions inside each benchmark it delegated. This shuffled version already knows which benchmark is harder. It just can't tell questions apart within a benchmark. All six models beat it, by 3.4–9.6 M-F1 points. So the decisions aren't only about which benchmark a question comes from. They pick out the right questions within the same benchmark.

Other results don't involve mixing benchmarks at all:

  • On Omni-MATH, a single benchmark, M-F1 is 0.67, and the self-solve rate drops from 90.7% on the easiest problems to 5.8% on the hardest.
  • On code, the real decisions also clearly beat the same within-benchmark shuffle, and on LiveCodeBench the self-solve rate drops with every difficulty tier.
  • Among problems that OLMoE finds equally hard, CSA still tells apart the ones its own model can solve from the ones it can't.

We'll also add a per-benchmark breakdown to the paper.

Thanks again. We hope our response address your concerns, and we'd be grateful if you'd consider it in your final assessment.


References

[1] Zhang et al. R-Tuning: Instructing Large Language Models to Say "I Don't Know". NAACL 2024.

[2] Kapoor et al. Large Language Models Must Be Taught to Know What They Don't Know. NeurIPS 2024.

Thank you for the detailed response. The budget-matched SFT comparison and the plan to present Table 16 as a cost–accuracy trade-off address my soundness concerns. My main concern remains novelty. Most of the findings in the general response either follow from RLVR changing the model very little or confirm the evaluation protocol, which I already listed as a strength. I will keep the score.

Reviewer_f5cxOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether an LLM can learn to judge whether it can solve a question before answering. If it can’t, it should pass it to a stronger model. The authors call it Capability Self-Assessment (CSA). Since language models are usually overconfident, they attempt questions they can’t really solve, leading to made-up answers and wasted compute. In this case, the same model judges and solves. If you train it to get better at judging, the training might change how well it solves. So the paper measures two things together: how good the model’s solve or delegate decisions are, and whether the model is still as good at actually solving problems after training. Comparing supervised fine-tuning (SFT) with RL, the paper finds that RL can teach a model to judge, before answering, whether it can solve a question without making it worse at solving, and that judgment is useful for routing hard questions to a bigger model and for choosing what to train on.

Critical strengths
  1. The evaluation accounts for the fact that the model being judged changes during training. When the same model both assesses and solves, training it to self-assess can change what it is able to solve. The paper handles this directly.

  2. The evidence of RL’s advantage rules out the obvious alternative explanations. My natural concern was that RL wins only because SFT breaks the model, that it learns “this benchmark is hard,” or that it learns generic difficulty. The paper tests each of these. That makes the main claim credible beyond a single table.

  3. It is efficient. The decision is made before answering, in about 151 tokens per query. Answer-based methods like self-consistency need five full answers.

Critical weaknesses
  1. The advantage over simple, training-free baselines is only in math; in science, the paper shows those baselines win (Table 16)

  2. Among the trained methods, the science advantage is also very thin (ties for Qwen3-4B and 8B, and a loss for 14B in Table 2), limiting the method’s generalizability.

  3. Limited novelty, and the central finding is largely expected. In the paper’s own Table 1, P(IK) is also trained, assessed before answering, and uses the same model for both roles. It differs only in not re-evaluating solving ability afterward.

  4. RL-trained abstention and confidence already exist “RL preserves ability better than SFT” is a known general result.

  5. The contribution comes down to a careful evaluation confirming a known RL-vs-SFT pattern in a new task. The retention-aware evaluation is valuable (I list it as the paper’s main strength), but to me it doesn’t add enough new method or insight for ICLR.

Questions for authors
  1. Novelty. Earlier work already trains the same model to assess before answering (P(IK). RL-trained confidence and help-seeking also exist, and the finding that RL forgets less than SFT is established. What do you find that is specific to self-assessment and not already predicted by these results?

2, Run-to-run variance. Is Table 2 from a single training run? If so, what is the variance, given that the three-seed RLVR runs in Table 3 score 0.84–0.85 on the same math setting where Table 2 reports 0.80?

  1. Label definition. The math label counts a question as self-solvable if any one of five attempts is correct, while deployment allows a single attempt. Does RLVR’s advantage in Table 2, and in the routing results, hold with a stricter label (estimated single-attempt success, or majority of five) across all models, not just Qwen3-1.7B on science?

  2. Science baselines. In Table 16, self-consistency beats RLVR on science for Qwen3-4B (0.75 vs 0.64) and OLMo2-7B (0.76 vs 0.69). Is the claim that RL gives the strongest assessment meant to cover only math, or only a cost-matched comparison?

3 comments
AuthorsOpenReview ↗

Response to Reviewer f5cx (1/2): novelty, P(IK), variance

Thank you for the detailed review and the concrete questions.

Q1, W3–W5: What's new, and how is this different from P(IK)? You're right that the paper adds no new method, and we don't claim one. It's an analysis, so the real question is whether the findings are new. Please read the general response which lists four that we don't think follow from earlier results on forgetting.

On P(IK) [1]: we agree it's the closest earlier work. It's trained, it decides before answering, and it uses the same model. Our baselines already cover this kind of predictor:

  • The query probe in Table 3 is a P(IK)-style classifier trained on the frozen model's representation of the question. It gets 0.70 / 0.71 M-F1 on Math (Qwen3-1.7B / 4B) and 0.55 / 0.56 on Science. RLVR gets 0.84 / 0.85 and 0.66 / 0.67.
  • Table 16 also includes read-outs of the model's own confidence before answering (P(Yes) and internal confidence), as well as Kadavath et al.'s P(True) [1]. All of them score below RLVR in all four settings.

The bigger difference is how it's evaluated. P(IK) is checked against labels from the model before training. Point 1 of the general response shows why that can mislead when the model being assessed is also the one being trained. We'll update Table 1 to make this clear.

Q2: Is Table 2 a single run? Why is it 0.80 there and 0.84–0.85 in Table 3? Yes, Table 2 is one training run per cell. Training several seeds for every cell (six models up to 14B, two domains, four training methods) is beyond our compute budget. That's why we ran Table 3 as a separate study. It compares RLVR with several SFT variants, with three runs each, on both Math and Science, for two models. Its setup is also different: 300 training questions are held out for validation, and the batch is fixed at 64 questions (31 updates on Math). So 0.80 and 0.84–0.85 come from different runs under different setups, and we only compare methods within the same table. Within Table 3, RLVR's M-F1 varies by only 0.01–0.02 across runs, and it stays ahead of every SFT variant.

Decoding noise is also small: re-running the Table 2 checkpoints with three decoding seeds changes M-F1 by at most 0.8 points (Appendix D.2.3).

AuthorsOpenReview ↗

Response to Reviewer f5cx (2/2): labels, Science

Q3: Does RLVR still win with a stricter label? That's a fair point: "any of five correct" is not the same as succeeding in one try. Several results already use stricter or different labels:

  • All Science results in Table 2, for all six models, use majority of five: at least three correct, with the options shuffled each time.
  • The code transfer and model-specificity results use success rates from 32 samples per problem.
  • The routing extension (Appendix D.12.2) trains and tests on single-try success rates, and RLVR is the strongest deployable method there.
  • The label ablation (Appendix D.2.1) trains with 1, 3 and 5 correct out of 5, and tests all three on single-try labels.

Repeating this for every model and task in Table 2 isn't feasible with our compute. If there's a specific model and task you'd like to see under a stricter label, we'd be happy to add it.

Q4, W1: On Science, self-consistency beats RLVR. Does the claim cover Science? Yes. In general, RLVR is the strongest of the trained methods, and of the methods that decide without writing an answer first. On Science, self-consistency and semantic entropy are more accurate. We don't treat RLVR as our contribution, so we have no problem with costlier baselines matching or beating it in some settings. That isn't the point of the paper. Here are all four settings, as accuracy vs. cost. Each cell is M-F1, with tokens per question in parentheses. Semantic entropy is within 0.03 of self-consistency everywhere, so we only show self-consistency.

SettingRLVRSelf-consistency, 3 samples5 samples10 samples
Qwen3-4B Math0.80 (151)0.78 (2,255)0.78 (3,799)0.74 (7,395)
Qwen3-4B Science0.64 (185)0.76 (1,707)0.75 (2,847)0.77 (5,763)
OLMo2-7B Math0.89 (402)0.81 (1,470)0.81 (2,417)0.84 (4,774)
OLMo2-7B Science0.69 (633)0.71 (1,448)0.76 (2,388)0.79 (4,804)

There's a reason this happens on Science and not on Math. The Science label asks "are at least three of five answers correct?", and self-consistency asks "do the answers agree?" With ten options, agreeing and being correct go together closely, so self-consistency is almost a second run of the labeling process, just without the answer key. On Math the label is "any of five correct", which agreement doesn't capture, and RLVR comes out ahead: 0.80 vs. 0.78 for Qwen3-4B, and 0.89 vs. 0.81–0.84 for OLMo2-7B. We were aware of this when we set up the study, and we kept two different labeling rules on purpose. In an analysis paper, it's useful to see how each method behaves under different definitions of "can solve", and this contrast is one example of what that shows. Self-consistency also has to write out full solutions before deciding, which is wasted work for every question it then delegates.

W2: Among the trained methods, the Science margins are thin. Agreed. On Science in Table 2, RLVR beats the best SFT variant for Qwen3-1.7B and both OLMo2 models, ties for Qwen3-4B and 8B, and is lower for Qwen3-14B (0.57 vs. 0.61). Where it ties or loses, the SFT models have lost 5–13% of their solving accuracy, and RLVR has lost none. In Table 3, with three runs and stronger SFT baselines, RLVR gets 0.66 / 0.67 against at most 0.60 / 0.65. We'll revise the Science claim to say that RLVR matches the best SFT variant in most settings and is the only method that keeps solving accuracy intact.

Thanks again. We hope this helps, and we'd be grateful if you'd consider it in your final assessment.

References

[1] Kadavath et al. Language Models (Mostly) Know What They Know. arXiv:2207.05221, 2022.

Reviewer_f5cxOpenReview ↗

Thank you

Thank you for the detailed responses. They addressed my main concerns, and I am raising my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: what this paper is, and what's new in it

Thank you all for the careful reviews, and for recognizing the evaluation: checking each trained model against its own capability after training (JHkc), the controls that rule out simpler explanations (f5cx), the range of models and domains (WpnK), and the low cost of deciding before answering (f5cx).

All three reviews share one main concern: the paper doesn't propose a new method, and "RL forgets less than SFT" is already known. We'd like to clear this up.

This is an analysis paper, not a method paper. None of our three contributions (lines 70-78) is a new method. What we propose is a framework for studying self-assessment. The labels come from the model's own attempts, the model decides before answering, and every trained model is checked against labels measured again after training, along with how much of its solving ability it keeps. This is what makes findings 1 and 3 below visible. An evaluation against fixed labels would miss them. Within this framework, we compare training-free methods, SFT and RL, and show two uses of the result. We used standard SFT and GRPO on purpose, so that any difference comes from the training signal and not from a new algorithm. The warm-up stage (DFW) is only an implementation detail that gives GRPO something to learn from.

"RL forgets less than SFT" isn't our contribution either. It's one observation along the way, and we cite RL's Razor [1] for it. The right question for an analysis paper is whether it teaches us something new about the problem. We think it does, in four ways.

  1. Training the model changes the thing it's trying to predict. The labels say what this model can solve, so if training changes what the model can solve, the labels go stale. On Qwen3-4B Science, label-only SFT keeps improving against its training labels (M-F1 0.60 → 0.67) while it gets worse against labels measured again after training (0.42 → 0.31), and its accuracy drops from 0.67 to 0.22. Judged the usual way, against fixed labels as with P(IK) [2], this model looks like it's improving. RL's Razor tells us RL forgets less. It doesn't tell us that the training metric and the real target can move apart.

  2. What the model learns is about itself, not about difficulty in general. This is the finding we care about most. Prior work mostly uses self-knowledge as a safeguard against hallucination, and for that a general sense of "this looks hard" may be enough. We care about the boundary of what this particular model can do, because that is what a model needs in order to decide what to learn next and to improve itself. A generic difficulty signal can't provide that: it would point every model at the same "hard" examples, whatever each one already knows. On 722 coding problems, a domain it never trained on, each model's decisions follow its own success rate far more closely than another model's (OLMoE; partial correlation 0.22–0.32 vs. 0.04–0.09). A rule that delegates the problems OLMoE finds hardest, based on 32 executed solutions per problem, does worse than CSA by 0.05–0.06 M-F1, even though CSA only reads the problem text. This is what makes it useful for data-efficient training and, longer term, for recursive self-improvement, which we're working on as a follow-up to this paper. As Section 5.2 shows, picking training data by what the model would delegate is the best or tied-best selection method at every budget, and for Qwen3-4B, training on 5% of the data chosen this way beats training on all of it (0.57 vs. 0.48).

  3. What the model can solve stays the same, query by query. Across 4,848 query–policy pairs, 4.5% of labels change after training, while 5.4% would change anyway just from resampling the same model. In math, none of the 1,960 per-query tests shows a significant change.

  4. "I can't solve this" doesn't mean "a stronger model can". Out of every 100 Science questions, the small model gets about 49 wrong. The large model fixes about 30 of those. On the other 19, both models are wrong, so sending them to the large model costs a cloud call and gains nothing. A model trained only to predict "will I fail?" would send all 49. This matters because delegation isn't free: what we want is a model that asks for help when help is worth it. When we instead reward the outcome of each choice (a correct answer, minus the cost of a cloud call), the model learns to keep questions the stronger model would also fail. It beats every deployable baseline at the same number of cloud calls (Appendix D.12.2).

In the revision, the abstract and introduction say clearly that this is an analysis, and they lead with these findings.

References

[1] Shenfeld, Pari, Agrawal. RL's Razor: Why Online Reinforcement Learning Forgets Less. ICLR 2026.

[2] Kadavath et al. Language Models (Mostly) Know What They Know. arXiv:2207.05221, 2022.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.