- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the degradation of LLMs' metacognitive abilities during reinforcement learning. In particular, although RLVR can improve reasoning accuracy, it may weaken the model's ability to recognize insufficient evidence and abstain from answering when necessary. To address this issue, the authors propose Cognitive Pairwise Training (CPT), which introduces reasoning quality comparison as a mid-training objective before downstream task optimization. Specifically, the authors construct high-quality pairwise reasoning comparison data through multi-model sampling, three debiased reasoning trace pairing strategies, and self-consistent teacher labeling. The model is then trained through supervised fine-tuning to compare the relative quality of different reasoning traces. Experiments are conducted across five model scales and three model families, covering mathematical reasoning, AbstentionBench, and conflicting-information RAG tasks. The results show that CPT improves the trade-off between reasoning accuracy and abstention ability, while significantly reducing the degradation of abstention performance caused by subsequent RL.
Critical strengths
-
This paper proposes shifting metacognitive training from traditional output-level abstention supervision to a more fundamental task of reasoning quality discrimination. The motivation is reasonable: if a model can better distinguish between reliable and unreliable reasoning, this ability may transfer to its own reasoning process and improve its robustness during subsequent reinforcement learning.
-
The experiments are relatively comprehensive. The authors evaluate CPT across different model scales and model families, and compare it with several baselines, including standard SFT+RL, data-matched SFT, DPO+RL, and abstention-aware RL.
Critical weaknesses
-
The supervision used in CPT includes not only relative quality labels for reasoning traces, but also detailed comparative analyses generated by a strong teacher model (Qwen3-235B). Therefore, the improvements may partly come from distilling high-quality reasoning from the teacher, rather than from pairwise comparison itself. In particular, the paper lacks controlled experiments that can clearly separate the contributions of pairwise comparison, teacher-generated rationales, and additional supervision tokens.
-
The main hypothesis of this paper is that the ability to compare external reasoning traces can transfer to the model's own reasoning process and improve its metacognitive abilities. However, the main evaluation metrics are still focused on final mathematical accuracy and Abstention F1. Although abstention is an important aspect of metacognition, a higher Abstention F1 does not directly show that the model has become better at monitoring the reliability of its own reasoning.
Questions for authors
-
Could the authors provide more controlled experiments to distinguish at least the following two types of supervision?
- Pairwise Label Only: Use the same pairs of reasoning traces, but train the model to predict only the quality comparison labels, without the teacher-generated rationales.
- Teacher Rationale SFT: Use the same problems and teacher model to generate reasoning quality analyses, but without requiring the model to compare two reasoning traces. The number of training tokens and computational budget should be matched as closely as possible.
These two experiments would help clarify whether the effectiveness of CPT mainly comes from the pairwise comparison task, the teacher-generated reasoning analyses, or a combination of both.
-
When controlling for final answer correctness, can CPT models identify errors in their own generated reasoning traces more accurately than SFT+RL models? For example, given a reasoning trace generated by the model itself, the model could be asked to predict whether the reasoning is correct or reliable. The authors could then evaluate the accuracy or calibration of these predictions.
-
On AbstentionBench, does the improvement in abstention performance also come with a better ability to identify answerable questions? Specifically, could the authors report the abstention rate, accuracy on answerable questions, and false abstention rate? This would help rule out the possibility that the improvement mainly comes from a change in the model's overall tendency to answer or abstain.
8 comments
New experiments about Pointwise/Pairwise Mid-training ablation
We thank the reviewers for suggesting a teacher-rationale control. We added six training runs across Qwen3-4B-Base and Qwen3-14B-Base, comparing Math SFT, CPT followed by Math SFT, and pointwise teacher-rationale SFT followed by Math SFT.
The pointwise teacher receives one reasoning trace, its problem, and the reference answer, and produces a quality analysis and a good/bad judgment without seeing a second trace. We also apply a 4-of-6 self-consistency filter, retaining examples only when at least four of six teacher judgments agree. Both mid-training datasets contain 70,352 examples from the same source pool and use Qwen3-235B-A22B-Instruct-2507 as teacher. Their supervised response-token totals are 88.01M versus 88.22M, differing by approximately 0.23% (excluding EOS).
Most settings follow the paper. The two changes are the use of LLaMA Factory for SFT and Qwen3.8-Flash-Next with xhigh thinking for LLM-as-a-judge evaluation. These changes may affect absolute scores relative to the original paper; all six runs use the same updated setup. These experiments evaluate the models after Math SFT, before RL.
Mathematical reasoning results (%)
| Model | MATH500 | Olympiad | MinervaMath | AMC22 | AMC23 | AIME24 | AIME25 | Easy5 | Hard2 | Avg7 |
|---|---|---|---|---|---|---|---|---|---|---|
| 4B Math only | 83.70 | 52.45 | 61.03 | 50.29 | 60.94 | 25.42 | 24.58 | 61.68 | 25.00 | 51.20 |
| 4B CPT + Math | 86.60 | 54.38 | 59.01 | 53.49 | 67.81 | 27.08 | 27.50 | 64.26 | 27.29 | 53.70 |
| 4B Rationale + Math | 86.30 | 55.49 | 58.46 | 55.23 | 71.88 | 28.33 | 25.42 | 65.47 | 26.88 | 54.44 |
| 14B Math only | 92.40 | 67.51 | 68.75 | 68.02 | 85.94 | 48.33 | 38.75 | 76.52 | 43.54 | 67.10 |
| 14B CPT + Math | 93.70 | 69.58 | 71.51 | 72.09 | 90.62 | 55.42 | 42.08 | 79.50 | 48.75 | 70.72 |
| 14B Rationale + Math | 93.50 | 68.92 | 68.57 | 72.38 | 89.06 | 52.92 | 38.75 | 78.49 | 45.83 | 69.16 |
Both mid-training approaches improve mathematical performance over Math SFT alone: pointwise rationale SFT performs better at 4B, while CPT performs better at 14B.
AbstentionBench results under the Normal Prompt (%)
Here, the positive class means that a question requires abstention. TP and FN correspond to questions requiring abstention; FP and TN correspond to answerable questions.
- Precision / Pre = TP / (TP + FP): the proportion of predicted abstentions that are appropriate.
- Recall = TP / (TP + FN): the proportion of questions requiring abstention on which the model abstains.
- F1 = 2TP / (2TP + FP + FN).
- Abs% = (TP + FP) / N: the overall abstention rate.
- False abstention rate / FAR = FP / (FP + TN): the proportion of answerable questions on which the model incorrectly abstains.
- Acc_Ans: accuracy on answerable questions with reference answers, counting abstentions as incorrect.
| Model | F1 ↑ | Abs% | Pre ↑ | Recall ↑ | FAR ↓ | Acc_Ans ↑ |
|---|---|---|---|---|---|---|
| 4B Math only | 56.96 | 22.11 | 88.53 | 41.98 | 4.74 | 71.16 |
| 4B CPT + Math | 58.11 | 22.77 | 88.56 | 43.24 | 4.87 | 74.19 |
| 4B Rationale + Math | 54.08 | 20.29 | 89.20 | 38.80 | 4.11 | 70.43 |
| 14B Math only | 61.84 | 24.77 | 89.15 | 47.33 | 5.03 | 79.53 |
| 14B CPT + Math | 65.13 | 26.49 | 89.91 | 51.06 | 5.00 | 81.90 |
| 14B Rationale + Math | 63.46 | 25.39 | 90.01 | 49.01 | 4.74 | 78.76 |
CPT exceeds pointwise rationale SFT in abstention F1 by 4.03 points at 4B and 1.67 points at 14B, while improving answerable-question accuracy by 3.76 and 3.14 points, respectively. Although pointwise rationale SFT improves mathematics at both scales, it does not consistently improve abstention F1 and reduces answerable-question accuracy relative to Math SFT alone at both scales. CPT improves both metrics over Math SFT alone at both scales. CPT has slightly higher FAR than the pointwise control, but its higher Acc_Ans shows that its gains are not limited to refusing more questions.
With source data, teacher identity, example count, and supervised response-token volume controlled, these results support an additional benefit from comparative supervision, beyond additional teacher-generated reasoning supervision alone.
Emergency: Follow-up Before the Discussion Deadline
Hello, as the author–reviewer discussion period is coming to an end, may we ask whether your main concerns have been addressed? If not, would you nevertheless consider raising your score?
W1: First, we conducted a comparison with SFT-80K using the same seed pool and number of problems. Our method uses fewer tokens because comparative analyses are generally shorter than directly generated solutions. This demonstrates the advantage of the pairwise comparison task over direct solution generation under a comparable compute budget.
In fact, we conducted related experiments early in this project. Training with teacher labels alone using RL, as you suggested, did not improve either mathematical reasoning or abstention performance and even slightly reduced mathematical performance. We attribute this to the inability to provide an effective process-level reward signal.
We agree that the Teacher Rationale SFT experiment is necessary, as it tests a core prior underlying our method: pairwise comparison is more robust than pointwise estimation. Although substantial evidence already supports this prior in reward modeling (lines 74–86), we will conduct this experiment and promptly upload the results. Thank you for pointing this out.
W2: We provide an analysis comparing the reasoning quality of CPT and SFT under matched final-answer correctness in Appendix F.3. Both teacher-model judgments and word-frequency-based statistics show clear advantages for our method.
We agree that abstention F1 is not a direct measure of self-monitoring. However, direct evaluation presents an important challenge: self-evaluation performance is coupled with task competence, trace difficulty, and error type. Evaluating each model on its own generations introduces differences in the evaluated contexts, whereas evaluating both models on identical traces does not fully capture self-monitoring of their own reasoning. To reduce confounding from final-answer correctness, our analysis in Appendix F.3 compares traces under matched correctness conditions.
Q1: See W1.
Q2: See W2.
Q3:
We found an intermediate analysis file from our original experiments comparing 14B CPT+RL with SFT+RL. Unfortunately, additional raw results were not retained. If further experiments are needed, we can rerun the evaluations (Please Let Us Know).
The recovered analysis identifies three categories of cases where CPT+RL succeeds while SFT+RL fails:
These cases show benefits in both appropriate abstention and correct answering. In particular, the latter two categories demonstrate improvements on answerable questions, including cases where CPT+RL avoids incorrect abstention by SFT+RL. This supports the view that the benefit is not simply an increased tendency to abstain.
Our main experiments initially focused on mathematical performance and abstention. We later added answerable-question accuracy in Figure 6 and Table 32 to check for over-abstention, showing that CPT maintains stronger performance on answerable questions than the compared baselines. However, the complete outputs needed to compute this metric for all main-experiment models were not retained. Abs.% is reported in Table 3, and the false abstention rate can be calculated.
Thank you for your time.