- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies RLVR for generative classification, where an LLM emits a class label as a short string. The authors argue that this regime has low action entropy and few distinct reward values, which exposes two failure modes in GRPO-family optimizers. Early in training, the elementwise min in PPO’s clipped objective selects the unclipped term for negative-advantage, high-ratio (NAHR) samples, which the authors argue causes instability. Later in training, groups receive uniform rewards and mean-centered advantages vanish. CARE addresses these with two hyperparameter-free changes. First, it replaces the per-sample min with a single batch-level choice of whichever batch-mean surrogate has smaller magnitude. Second, it normalizes advantages over the whole batch and falls back to the raw reward when the batch has zero reward variance. The authors also claim that the batch-level selection recovers entropy without an entropy bonus. Trained on RAGTruth QA with Mistral-7B, CARE reaches 81.90 macro-F1 versus 70.44 for GRPO, transfers to seven out-of-distribution hallucination sets, reaches 64.40 on ANLI versus 59.20 for the strongest baseline, and composes with DAPO and DR.GRPO.
Critical strengths
- Relevant setting. RL for short-output classification (hallucination detection, moderation, intent classification) is practically important and receives little attention compared with reasoning tasks.
- Simple and composable. Both modifications are a few lines of code, add no hyperparameters, and can be layered on top of existing optimizers.
- Reasonably broad evaluation. The paper uses three seeds in-distribution, seven OOD sets, a second backbone (LLaMA-3.1-8B), a multi-class benchmark, and a component ablation. It also includes a useful comparison showing that SFT is strong in-distribution but transfers poorly.
- Good ethics discussion. The paper considers asymmetric error costs, calibration, and deployment risks.
Critical weaknesses
- Key related methods are missing. Several methods target exactly the zero-variance or advantage-collapse problem and are neither discussed nor compared: RL-ZVP, NGRPO, AVSPO, EDGE-GRPO, LENS, and GRESO. Dual-clip PPO is missing for the surrogate side.
- The empirical case is weaker than presented. In-distribution, SFT (79.77) and TRULY-PPO (79.12) are within about two standard deviations of CARE (81.90 ± 1.25). All OOD results are single-seed, and SFT’s OOD collapse may reflect tuning or overfitting rather than an inherent limitation. Established hallucination detectors such as MiniCheck, AlignScore, and LettuceDetect are not compared.
- The NAHR diagnosis misreads PPO. For a sample with negative advantage and a ratio above , selecting the unclipped term is the intended pessimistic lower bound. The resulting gradient lowers the probability of a bad action whose probability has risen, which is a corrective update rather than an “escape route.” The only real issue is that its magnitude is unbounded, which is exactly what Dual-clip PPO addresses. Dual-clip PPO is neither cited nor compared.
- Mod 1 is not actually conservative. It compares the magnitudes of batch means. Because advantages are mean-centered, positive and negative terms largely cancel, so this comparison does not measure how aggressive an update is and is likely noise-dominated. Whenever the unclipped branch is selected, clipping is disabled for the entire batch, including positive-advantage samples with large ratios. Section 5’s entropy-recovery argument also relies on enabling gradients beyond the trust region (Q3), which contradicts the “conservative” framing.
Questions for authors
see above
2 comments
Response to Reviewer Yq4P. Part [2/2]
W3: The NAHR gradient
We agree: the min is PPO's intended pessimistic bound, and the NAHR gradient is corrective in direction. Our concern is only its size, as you note. Corrective updates also do not need NAHR samples: every in-range negative-advantage sample gives one, and CARE keeps all in-range samples active in every batch. By "escape route" we mean that Q2 is the one region whose term grows with r without limit (Proposition 1); Section 3 now states this in the updated manuscript.
Dual-Clip PPO bounds this term with a fixed constant c, so the NAHR sample keeps its full gradient for 1+ε < r < c regardless of the batch; Q3 stays clipped and uniform groups still give zero gradient. CARE uses a dynamic clip instead: the NAHR sample keeps its full gradient until it would push the batch objective past its clipped value. With ε = 0.2, A = (+1, +1, −1, −1) and in-range ratios 1.15, 1.15, 0.85, this happens at r = 1.7; with 1.1, 1.1, 0.9 at r = 1.4. On Q2 samples, sample-level Mod 1 equals Dual-Clip PPO at c = 1+ε. Section 2 and Table 1 now include it (c = 3, CARE's configuration). At the same configuration and seed, it reaches 70.29, against 56.38 for GRPO, 73.35 for sample-level Mod 1 and 74.97 for batch-level Mod 1 (Tables 2 and G.1). Over three seeds it reaches 72.52 in-distribution, 2.1 points above GRPO and 9.4 below CARE.
W4: In what sense Mod 1 is conservative
Let s1 and s2 be the unclipped and clipped batch means. Eq. 6 keeps s1 only when |s1| ≤ |s2|, that is, while s1 lies in CARE's gradient-active band [−|s2|, +|s2|].
- The applied batch objective is therefore never larger in magnitude than the fully clipped one, in every batch. This is the sense of "conservative": a bound on the batch objective, not a cap on each sample's gradient. Section 3 now states this scope, and Section 5.2 gives the full rule.
- Positive-advantage samples with large ratios are active only if inside this bound. When s2 > 0, such a sample shifts s1 out of the band and switches the batch to s2.
- On cancellation: when the means cancel and |s2| is small, the band is narrow, so the batch keeps s1 only if out-of-range samples change it by little. Proposition 3 shows that the batch decision has lower variance than the per-sample one.
- Opening Q3 is intended and also happens inside this bound. It is the route to entropy recovery (Section 5), which GRPO's clipping never allows.
We once again thank you for your valuable comments! If you have any further questions, please feel free to let us know and we will do our utmost to respond. If you find our replies satisfactory, we kindly ask that you consider updating your score accordingly.
Response to Reviewer Yq4P. Part [1/2]
We thank you for your careful review. We are glad you found the setting practically important, the two changes simple and composable, the evaluation reasonably broad, and the ethics discussion useful. Your questions on the NAHR gradient and on the Mod 1 sharpened our own understanding of CARE. They led us to state Mod 1's scope precisely in Section 3. We respond below.
W1: Related methods
Thank you for the pointers; we read each paper. RL-ZVP, NGRPO, AVSPO and EDGE-GRPO reshape the advantage of uniform groups (entropy terms, virtual samples, injected reference answers), and LENS cleans prompts. GRESO, like DAPO's dynamic sampling (a baseline in Table 1), discards zero-variance prompts, but predicts them before rollout to save compute. All address only the zero-variance failure (our Failure Mode 2), are designed for long chain-of-thought math reasoning, and most add hyperparameters to tune. Mod 2's fallback adds no hyperparameter, rollout or reference answer. None changes which surrogate term is kept, so none addresses the NAHR update that Mod 1 targets. They are therefore complementary to CARE and could, in principle, be combined with it, as DAPO and Dr. GRPO are in Table 1.
W2: Empirical case
In-distribution, we agree that SFT and TRULY-PPO are close, and the paper says so (Section 4.2). TRULY-PPO, the closest prior repair to surrogate selection, recovering most of the gap supports our diagnosis; CARE stays ahead without its two extra hyperparameters.
Our claim on SFT is about transfer, and transfer was the reason we developed CARE: on our production data, SFT did not carry over to new domains. SFT uses the same backbone, LoRA setup, optimizer, epochs and batch size as CARE (Appendix F), so the objective is the only difference, and its 79.77 in-distribution shows it is well fit. Out of distribution it falls to 36.93 and 37.46 on Internal QA, our financial-services data, where CARE reaches 64.88 and 61.26, and it trails CARE on all seven OOD sets. Fitting one fixed target in this way is the known limitation of SFT relative to RL post-training (Chu et al., 2025). These entries are single-seed, but gaps of over 20 points are far larger than the in-distribution seed spread.
MiniCheck, AlignScore and LettuceDetect are dedicated detectors with their own architectures and training data; Appendix A already discusses MiniCheck and AlignScore. CARE is an RL objective for generative classification (see also ANLI, 64.40 vs 59.20), and our study holds the model and data fixed to compare objectives.