- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors first observe that GRPO training significantly reduces the recall of uncertain categories, such as Uncertain, Maybe, and Not Enough Information, across multiple tasks, even as overall classification accuracy improves. This phenomenon consistently occurs in tasks with rare uncertain categories, but is not observed in control tasks with more frequent middle categories. These findings suggest that improvements in overall accuracy can mask substantial degradation in the model's ability to recognize specific categories. To understand this phenomenon, the authors conduct a series of controlled experiments and identify two main contributing factors. First, the frequency of the middle category plays an important role in determining its recall after RLVR training. Second, the authors argue that the two-sided structure of the middle category makes it more vulnerable than the extreme categories. Based on these observations, the authors attribute the collapse of the middle category to the combined effects of class rarity and its two-sided structure. To address this issue, the authors propose CURB, which modifies the model's output representation. Instead of directly predicting one of several ordered categories, CURB decomposes the classification task into multiple one-sided threshold decisions. For example, given three categories, False, Uncertain, and True, the model separately determines whether the true label is at least Uncertain and whether it is at least True. The final category is then derived from these binary decisions. In this way, the middle category is no longer treated as a direct prediction target bounded by two thresholds. Experiments show that CURB substantially improves middle-category recall after reinforcement learning without changing the original class distribution.
Critical strengths
I think the flow of this paper is clear and easy to follow. Although the final method is simple, the author spend two section with couple of experiments to show why the design method in this way. The most interesting finding is that it is not only due to the occcurance of "uncertain" in training data, it is due to the middle boundary is two sided and two sided is harder to predict in nature.
Based on these findings, the authors propose a simple yet well-motivated solution that replaces direct category prediction with multiple one-sided threshold decisions. Overall, I appreciate how the method is motivated by the empirical findings rather than introduced as an isolated design choice.
Critical weaknesses
-
Based on the experiments in Section 3.2, the authors claim that the middle category is more likely to collapse due to its two-sided boundary structure. However, I think these results only show that the middle category suffers a larger recall drop, but do not directly prove that this is caused by the two-sided boundaries. Other categories, such as Likely Pathogenic and Likely Benign, also have two-sided boundaries, but they do not show such severe collapse. I think this needs further explanation.
-
The CURB and standard GRPO methods have different performance after SFT warm-up. For example, in Biomedical QA, the recall of the standard method drops from 24.2% to 6.1%, while CURB drops from 48.5% to 27.3%. This makes me question whether the improvement is mainly due to the format of how the question is asked, rather than CURB's ability to mitigate RLVR collapse. In fact, CURB has a larger absolute recall drop in this case.
-
In Section 3.3, the authors show that standard SFT achieves both higher middle-category recall and higher overall accuracy than RLVR on Financial Fact-checking. If this is the case, why do we still need RLVR for these tasks? I would like to see a direct comparison between SFT-only and SFT+GRPO using the same CURB output format, training data, and comparable training budgets.
Questions for authors
See weakness.
4 comments
Response to Reviewer qGTW (2/2)
(Continued from our previous comment, which covers W1 and W2.)
[W3] SFT-only vs. SFT+GRPO in the CURB format — the matched-budget comparison, and why RLVR
…why do we still need RLVR? I would like to see a direct comparison between SFT-only and SFT+GRPO using the same CURB output format, training data, and comparable training budgets.
On §3.3. That SFT run is a control: it shows the collapse comes from the reward objective, not from the rare label. It was not a statement about which method to deploy, and you are right that the §1 sentence calling RLVR "preferred over supervised fine-tuning" overstated; we will change it.
The comparison you asked for. We ran it entirely within the CURB output format and on the same pool. SFT-only trains on the pool from the base model; SFT+GRPO does a supervised cold-start on a disjoint cold-start set, then 400 GRPO steps on the pool. We count budget as training examples seen (dataset size × epochs): SFT+GRPO's budget is its cold-start plus the GRPO stage (≈2,600–3,100 examples), which is about 2 epochs of SFT on the larger pools (financial, genetics) and ≈6 on the small PubMedQA pool. We report SFT-only at that matched budget, SFT+GRPO, and SFT-only at 12 epochs (middle recall / overall accuracy; full curves in the appendix):
| CURB — middle recall / overall accuracy | SFT-only (matched budget) | SFT+GRPO(CURB) | SFT-only (12 ep) |
|---|---|---|---|
| financial (3-class) | 0.377 / 0.711 | 0.384 / 0.689 | 0.687 / 0.691 |
| biomedical / PubMedQA (3-class) | 0.495 / 0.644 | 0.202 / 0.712 | 0.404 / 0.644 |
| genetics (5-class) | 0.321 / 0.412 | 0.225 / 0.498 | 0.518 / 0.528 |
(SFT-only: 3 seeds. SFT+GRPO: 5 seeds on financial and genetics, 3 on biomedical.)
At the matched budget, financial is on par (0.377 vs 0.384); on genetics and biomedical SFT keeps more of the middle (0.321 vs 0.225; 0.495 vs 0.202) while SFT+GRPO is the more accurate (0.498 vs 0.412; 0.712 vs 0.644) — the accuracy↔middle trade-off, now at a matched budget. With more SFT epochs the middle rises further on financial and genetics (on biomedical it is highest at 6 epochs), and on genetics SFT also becomes the more accurate. SFT-only keeps the middle as well as SFT+GRPO or better throughout, so for label-complete tasks we do not claim RLVR beats SFT, and we will make this explicit in revised §3.3.
Why RLVR, then? The table shows its appeal: at a matched budget it is more accurate than SFT on two of the three tasks (2 points lower on financial). It is also:
- a widely applied recipe for such tasks (Yaldiz et al. 2026; Lee et al. 2026; He et al. 2025);
- the natural option when supervision is a verifier of the final verdict, not target outputs.
Our contribution is about what that accuracy hides. Standard single-label GRPO has a previously unreported failure on ordinal tasks — it collapses the middle while accuracy rises (financial 0.325→0.149; biomedical 0.242→0.000, i.e. 0 of 33 middle items in every seed). We diagnose it (the two-cutoff structure) and reduce it at the output (CURB): vs. single-label GRPO, financial 0.384 vs 0.149, biomedical 0.202 vs 0.000. A gap to matched-budget SFT remains on genetics and biomedical.
What the new results leave standing. (i) The phenomenon: under standard single-label RLVR the rare middle collapses while accuracy rises, in every re-run and seed (biomedical: 0 of 33 in all three), and accuracy alone does not reveal it. (ii) The cause: at matched rarity all two-sided tiers collapse and the one-sided ends do not (W1). (iii) Where the remedy must act: graded rewards leave the collapse in place (Table 7), while changing the output reduces it and, on financial, reverses the order (W2). To our knowledge none of these three has been reported for RLVR.
We will add the budget table and curves and the training-cost column. If a further control would be more convincing, we are happy to run it.
Thank you for the responses. These have addressed some of my concerns. Some limitations remain, including differences in initial performance across output formats and incomplete matching of training data and computational cost in the SFT comparison. However, I find the observed failure mode interesting and valuable. Even without demonstrating a general advantage over SFT, the diagnosis and analysis remain meaningful contributions. I am therefore decide to increase my score.
Answer to Reviewer qGTW
Thank you for raising your score and for the careful review. We are glad that you find our findings valuable. We will keep the remaining limitations in mind as we improve the paper.
Response to Reviewer qGTW (1/2)
Thank you for the careful and specific review. Each weakness led to a few new experiments, so our response is split into two comments: W1 and W2 here, W3 in the next one.
[W1] Does the larger drop prove the two-sided boundary is the cause? (the LP/LB counterexample)
What the paper claims. The claim is the combination: a category collapses when it is rare and two-sided (Sec. 3), not two-sidedness alone. Table 3 is a controlled test of this, not only an observation of a larger drop: it fixes the minority share at 12% and changes only which category is the minority, on two tasks. Under this claim Likely Pathogenic is not expected to collapse in the natural pool, where it is common (22%).
Where you are right. Table 3 compared the middle only with the two ends, and the paper did not explain why the other two-sided tiers survive. We ran the missing control: each tier in turn at the same 12% share under standard single-label RLVR (one shared cold-start; not CURB):
At equal rarity every two-sided tier collapses (−59% to −88%) and the one-sided ends lose far less (−13% to −18%). This is the direct test of the cause that Table 3 lacked. Likely Benign is as rare as VUS in the natural pool (14% each); it rose in the submission's run from a low start, and in a separate run from a class-uniform cold-start it also falls (0.677→0.418, 3 seeds). The comparison is across classes, so it is a controlled test rather than a formal proof; the full table goes into the revised PDF, and Sec. 3.2 will state the rare-and-two-sided condition for every interior tier.
[W2] Is CURB's gain decided at the SFT warm-up (its format's higher base), or does the RL stage matter? (the "larger absolute drop")
Table 8 tests a second multiple-question format that still asks about the middle directly: one yes/no per class ("is it low?", "is it the middle?", "is it high?"). This format starts even higher than the single label on biomedical QA (0.364 vs 0.242) yet still collapses to 0.061. Only CURB's one-sided cutoff questions ("is it at least the middle?", "is it at least high?"), where the middle is never asked about but read off as the gap, keep it. So the cure is not reformatting per se (any format that still names the two-sided middle collapses).
Where you are right. The two formats start at different levels after warm-up, and the paper did not separate the starting point from the RL stage. Part of the gain is the starting point: the threshold format reads the middle higher after warm-up, on average (though not in every seed). But the gain is not only the starting point:
The second row is a new control for your concern: both formats' warm-ups trained on the same class-balanced data. CURB still starts somewhat higher (0.292 vs 0.244), but it loses less and ends about twice as high. On biomedical QA we re-ran both formats with 3 seeds at a fixed step (the submission had single runs, 0.061 and 0.273): the single label keeps none of the 33 middle items in any seed; CURB keeps about 7. CURB's absolute drop is indeed larger there (0.283 vs 0.242): it starts twice as high, and the single label cannot fall below zero.
On financial the single label starts higher, and RL reverses the order.
CURB starts lower and ends higher, so this gain comes from the RL stage, not from the starting point. (Table 6 gives CURB 0.246→0.321 from 3 runs, the same direction.) We will reword "restores the middle" in §4 to "reduces the collapse" and update the biomedical rows of Tables 1, 6 and 8.