acceptodds
Under review as a conference paper at ICLR 2027

CARE: CONSERVATIVE POLICY OPTIMIZATION FOR RLVR IN GENERATIVE CLASSIFICATION

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become the predominant approach for post-training large language models on tasks where output correctness can be algorithmically verified, and GRPO-family optimizers (GRPO, DAPO, DR.GRPO), all of which inherit PPO’s clipped surrogate, dominate this space. Their design assumptions of high-entropy action spaces and well-dispersed rewards are inherited from multi-step mathematical reasoning and open-ended generation. Generative classification, where the policy emits a class label directly as a short generated string, satisfies neither: the action distribution is low-entropy, and while every rollout is scored, the reward takes only a handful of distinct values. We identify two temporally distinct breakdowns of this clipped surrogate under these conditions. Early in training, elementwise surrogate minimization selects the unclipped branch for negative-advantage high-ratio (NAHR) samples, admitting exactly the aggressive updates that clipping exists to suppress. Later, as accuracy rises, prompt groups become uniform and mean-centered advantages vanish, starving the policy of gradient signal. We propose Conservative Advantage Ratio Enhancement (CARE), which addresses both with two modifications and no new hyperparameters. First, the elementwise minimum between clipped and unclipped surrogates is replaced by a single batch-aggregated magnitude comparison, committing the whole update to the less aggressive surrogate and closing the per-sample escape route. Second, advantage normalization is applied conditionally, falling back to the raw reward when the batch carries no reward variance. We further show that the batch-level conservative surrogate selection enables entropy recovery in collapsed-policy regimes without an explicit entropy bonus. Trained on RAGTruth QA for hallucination classification, CARE surpasses GRPO by absolute +11.5 points macro-F1 on in distribution test set over three seeds, and transfers to OOD hallucination classification on RAGTruth (Summarization, Data2Text), HaluEval (QA, Dialogue, Summarization), and Internal QA (Easy, Hard). CARE attains 64.40 macro-F1 on the multi-class ANLI benchmark against 59.20 for the strongest baseline.

Primary area:
reinforcement learning
Keywords:
Natural Language Processing, Generative Classification, Reinforcement Learning with Verifiable Rewards, Hallucination Detection, Language Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_Yq4POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies RLVR for generative classification, where an LLM emits a class label as a short string. The authors argue that this regime has low action entropy and few distinct reward values, which exposes two failure modes in GRPO-family optimizers. Early in training, the elementwise min in PPO’s clipped objective selects the unclipped term for negative-advantage, high-ratio (NAHR) samples, which the authors argue causes instability. Later in training, groups receive uniform rewards and mean-centered advantages vanish. CARE addresses these with two hyperparameter-free changes. First, it replaces the per-sample min with a single batch-level choice of whichever batch-mean surrogate has smaller magnitude. Second, it normalizes advantages over the whole batch and falls back to the raw reward when the batch has zero reward variance. The authors also claim that the batch-level selection recovers entropy without an entropy bonus. Trained on RAGTruth QA with Mistral-7B, CARE reaches 81.90 macro-F1 versus 70.44 for GRPO, transfers to seven out-of-distribution hallucination sets, reaches 64.40 on ANLI versus 59.20 for the strongest baseline, and composes with DAPO and DR.GRPO.

Critical strengths
  1. Relevant setting. RL for short-output classification (hallucination detection, moderation, intent classification) is practically important and receives little attention compared with reasoning tasks.
  2. Simple and composable. Both modifications are a few lines of code, add no hyperparameters, and can be layered on top of existing optimizers.
  3. Reasonably broad evaluation. The paper uses three seeds in-distribution, seven OOD sets, a second backbone (LLaMA-3.1-8B), a multi-class benchmark, and a component ablation. It also includes a useful comparison showing that SFT is strong in-distribution but transfers poorly.
  4. Good ethics discussion. The paper considers asymmetric error costs, calibration, and deployment risks.
Critical weaknesses
  1. Key related methods are missing. Several methods target exactly the zero-variance or advantage-collapse problem and are neither discussed nor compared: RL-ZVP, NGRPO, AVSPO, EDGE-GRPO, LENS, and GRESO. Dual-clip PPO is missing for the surrogate side.
  2. The empirical case is weaker than presented. In-distribution, SFT (79.77) and TRULY-PPO (79.12) are within about two standard deviations of CARE (81.90 ± 1.25). All OOD results are single-seed, and SFT’s OOD collapse may reflect tuning or overfitting rather than an inherent limitation. Established hallucination detectors such as MiniCheck, AlignScore, and LettuceDetect are not compared.
  3. The NAHR diagnosis misreads PPO. For a sample with negative advantage and a ratio above 1+ϵ1+\epsilon, selecting the unclipped term is the intended pessimistic lower bound. The resulting gradient lowers the probability of a bad action whose probability has risen, which is a corrective update rather than an “escape route.” The only real issue is that its magnitude is unbounded, which is exactly what Dual-clip PPO addresses. Dual-clip PPO is neither cited nor compared.
  4. Mod 1 is not actually conservative. It compares the magnitudes of batch means. Because advantages are mean-centered, positive and negative terms largely cancel, so this comparison does not measure how aggressive an update is and is likely noise-dominated. Whenever the unclipped branch is selected, clipping is disabled for the entire batch, including positive-advantage samples with large ratios. Section 5’s entropy-recovery argument also relies on enabling gradients beyond the trust region (Q3), which contradicts the “conservative” framing.
Questions for authors

see above

2 comments
AuthorsOpenReview ↗

Response to Reviewer Yq4P. Part [1/2]

We thank you for your careful review. We are glad you found the setting practically important, the two changes simple and composable, the evaluation reasonably broad, and the ethics discussion useful. Your questions on the NAHR gradient and on the Mod 1 sharpened our own understanding of CARE. They led us to state Mod 1's scope precisely in Section 3. We respond below.

W1: Related methods

Thank you for the pointers; we read each paper. RL-ZVP, NGRPO, AVSPO and EDGE-GRPO reshape the advantage of uniform groups (entropy terms, virtual samples, injected reference answers), and LENS cleans prompts. GRESO, like DAPO's dynamic sampling (a baseline in Table 1), discards zero-variance prompts, but predicts them before rollout to save compute. All address only the zero-variance failure (our Failure Mode 2), are designed for long chain-of-thought math reasoning, and most add hyperparameters to tune. Mod 2's fallback adds no hyperparameter, rollout or reference answer. None changes which surrogate term is kept, so none addresses the NAHR update that Mod 1 targets. They are therefore complementary to CARE and could, in principle, be combined with it, as DAPO and Dr. GRPO are in Table 1.


W2: Empirical case

In-distribution, we agree that SFT and TRULY-PPO are close, and the paper says so (Section 4.2). TRULY-PPO, the closest prior repair to surrogate selection, recovering most of the gap supports our diagnosis; CARE stays ahead without its two extra hyperparameters.

Our claim on SFT is about transfer, and transfer was the reason we developed CARE: on our production data, SFT did not carry over to new domains. SFT uses the same backbone, LoRA setup, optimizer, epochs and batch size as CARE (Appendix F), so the objective is the only difference, and its 79.77 in-distribution shows it is well fit. Out of distribution it falls to 36.93 and 37.46 on Internal QA, our financial-services data, where CARE reaches 64.88 and 61.26, and it trails CARE on all seven OOD sets. Fitting one fixed target in this way is the known limitation of SFT relative to RL post-training (Chu et al., 2025). These entries are single-seed, but gaps of over 20 points are far larger than the in-distribution seed spread.

MiniCheck, AlignScore and LettuceDetect are dedicated detectors with their own architectures and training data; Appendix A already discusses MiniCheck and AlignScore. CARE is an RL objective for generative classification (see also ANLI, 64.40 vs 59.20), and our study holds the model and data fixed to compare objectives.

AuthorsOpenReview ↗

Response to Reviewer Yq4P. Part [2/2]

W3: The NAHR gradient

We agree: the min is PPO's intended pessimistic bound, and the NAHR gradient is corrective in direction. Our concern is only its size, as you note. Corrective updates also do not need NAHR samples: every in-range negative-advantage sample gives one, and CARE keeps all in-range samples active in every batch. By "escape route" we mean that Q2 is the one region whose term grows with r without limit (Proposition 1); Section 3 now states this in the updated manuscript.

Dual-Clip PPO bounds this term with a fixed constant c, so the NAHR sample keeps its full gradient for 1+ε < r < c regardless of the batch; Q3 stays clipped and uniform groups still give zero gradient. CARE uses a dynamic clip instead: the NAHR sample keeps its full gradient until it would push the batch objective past its clipped value. With ε = 0.2, A = (+1, +1, −1, −1) and in-range ratios 1.15, 1.15, 0.85, this happens at r = 1.7; with 1.1, 1.1, 0.9 at r = 1.4. On Q2 samples, sample-level Mod 1 equals Dual-Clip PPO at c = 1+ε. Section 2 and Table 1 now include it (c = 3, CARE's configuration). At the same configuration and seed, it reaches 70.29, against 56.38 for GRPO, 73.35 for sample-level Mod 1 and 74.97 for batch-level Mod 1 (Tables 2 and G.1). Over three seeds it reaches 72.52 in-distribution, 2.1 points above GRPO and 9.4 below CARE.


W4: In what sense Mod 1 is conservative

Let s1 and s2 be the unclipped and clipped batch means. Eq. 6 keeps s1 only when |s1| ≤ |s2|, that is, while s1 lies in CARE's gradient-active band [−|s2|, +|s2|].

  1. The applied batch objective is therefore never larger in magnitude than the fully clipped one, in every batch. This is the sense of "conservative": a bound on the batch objective, not a cap on each sample's gradient. Section 3 now states this scope, and Section 5.2 gives the full rule.
  2. Positive-advantage samples with large ratios are active only if inside this bound. When s2 > 0, such a sample shifts s1 out of the band and switches the batch to s2.
  3. On cancellation: when the means cancel and |s2| is small, the band is narrow, so the batch keeps s1 only if out-of-range samples change it by little. Proposition 3 shows that the batch decision has lower variance than the per-sample one.
  4. Opening Q3 is intended and also happens inside this bound. It is the route to entropy recovery (Section 5), which GRPO's clipping never allows.

We once again thank you for your valuable comments! If you have any further questions, please feel free to let us know and we will do our utmost to respond. If you find our replies satisfactory, we kindly ask that you consider updating your score accordingly.

Reviewer_cd6aOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates RLVR on classification tasks and find two key failure modes: 1) early-training instability in the branch that elementwise minimum selects for NAHR samples (Proposition 1) and 2) late-training collapse in the probability that a group is uniform. The authors proposed two key mechanisms: 1) Batch-level conservative surrogate selection (Mod 1) closes the NAHR escape route and 2) conditional advantage computation (Mod 2) preserves gradient flow on zero reward variance batches. Experiments show consistent improvement compared to strong prior RLVR apporaches.

Critical strengths
  1. The paper is clear and easy to follow.
  2. The paper is insightful which provides deep analysis on failure modes of existing RLVR approach on classification tasks.
  3. The experiments are strong and extensive.
Critical weaknesses
  1. The selected foundation models is old and limited. For example, the authors only select on foundation models, Mistral 7B Instruct v0.3. I suggest the authors add more experiments on broader model family.
  2. Lack of some baselines, such as R1-RE, which also explore RLVR on classification tasks.
Questions for authors

See above

1 comment
AuthorsOpenReview ↗

Response to Reviewer cd6a

We appreciate your valuable time and insights, and thank you for finding the paper clear, insightful, and empirically strong. We respond to both points below.

W1: On the choice and diversity of foundation models

We respectfully note that the paper already reports a second backbone from a different model family. Besides Mistral 7B Instruct v0.3, we replicate the full RAGTruth QA training and all eight evaluations on LLaMA 3.1-8B (Section 4.3, "Seeds and backbones"; Table G.3 in Appendix G). The two models differ in pretraining data and tokenizer (32K vs. 128K vocabulary).

The LLaMA results mirror the Mistral ones. CARE improves over GRPO on all eight evaluations, including +5.94 Macro-F1 in-distribution (79.56 vs. 73.62). A CARE variant is the best method in seven of eight columns. The one exception is HaluEval QA, where Dr. GRPO leads by 0.47 points.

On the model choice itself, we offer two clarifications.

  1. CARE is an objective-level change. Both failure modes follow from the bounded label set and the few-valued reward, not from the backbone (Propositions 1 and 2).
  2. Deployment constraints guided the selection. Our target setting is self-hosted 7 to 13B models in regulated domains (Section 1). Our institution's model-governance policy restricts which open-weight models can run in our compute environment. Within these constraints, we report results on two widely used model families, Mistral and LLaMA.

W2: On the comparison with R1-RE

Thank you for the pointer. R1-RE is a training framework for relation extraction rather than a new policy-optimization algorithm. Its contributions are an annotation-guideline prompt and a multi-stage format and accuracy reward. The policy itself is optimized with standard GRPO, which is already our primary baseline (Table 1).

R1-RE also works in a different regime. The model reasons in a long chain of thought before answering, and its responses grow from about 200 tokens to roughly 500 and 1,000 tokens on its two datasets. CARE targets generative classification, where the label is emitted directly, and changes only the optimization objective.

Our experiments therefore hold the prompt, reward, data, and backbone fixed, so that the objective is the only variable. The matching comparisons are other objectives: the GRPO family (GRPO, DAPO, Dr. GRPO), which shares the template we analyze; REINFORCE++, which uses batch-level pooling; TRULY-PPO, the closest prior repair to surrogate selection; and SFT as the supervised reference (Table 1).


We once again thank you for your valuable review! If you have any further questions, please feel free to let us know and we will do our utmost to respond. If you find our replies satisfactory, we kindly ask that you consider updating your score and confidence accordingly.

Reviewer_7m8FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies RLVR for generative classification and attributes optimization difficulties to asymmetric surrogate clipping and vanishing group-relative advantages. It proposes CARE, combining batch level surrogate selection based on absolute magnitude with batch normalized advantages that revert to raw rewards when reward variance is zero. On RAGTruth QA, CARE achieves 81.90 macro-F1 versus GRPO’s 70.44 across three seeds. The paper also reports cross domain evaluations, component ablations, and additional backbone and multiclass results.

Critical strengths
  1. Short-label generative classification is a useful application of RLVR, and uniform reward groups present a concrete optimization difficulty. CARE introduces two compact changes without additional tunable hyperparameters, making the approach straightforward to incorporate into existing training pipelines.
  2. Table 1 includes GRPO family methods, SFT, REINFORCE++, and TRULY-PPO. CARE’s three-seed improvement and its results across several distribution shifts provide meaningful evidence of practical utility beyond comparison with GRPO alone.
Critical weaknesses
  1. Equation 6 compares absolute batch-mean surrogate values. Opposite contributions can cancel, allowing the unclipped branch to be selected even when the batch contains high ratio samples. Smaller objective magnitude also does not imply a smaller gradient or policy update. Consequently, the claim that CARE closes the NAHR escape route, and the clipping diagrams used to explain entropy recovery, are not generally established for the actual batch-level algorithm. This weakens the central mechanistic justification.
  2. Mod 2 simultaneously changes normalization from per-prompt groups to the full batch and introduces the zero-variance fallback, so that Table 2 does not separate their contributions under identical settings. Moreover, full CARE exceeds Mod 2 alone by only 0.92 macro-F1 in a single-seed comparison. The available evidence therefore does not establish how much improvement comes specifically from the fallback or whether Mod 1 adds a reproducible benefit.
Questions for authors
  1. Under what conditions does Equation 6 suppress NAHR gradients in batches containing mixed sign advantages? Please reconcile the batch level rule with the per-sample clipping regions in Figures 2 and 4.
  2. With normalization pooling and all training settings fixed, what improvement comes from the zero variance fallback alone? Does the additional benefit of Mod 1 persist across seeds?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 7m8F. Part [1/2]

We thank you for your careful and technical review. We appreciate your recognition that uniform reward groups are a concrete optimization difficulty, and that CARE's two compact, hyperparameter-free changes are easy to add to existing pipelines. We are also glad you found the breadth of baselines, the three-seed gains, and the distribution-shift results meaningful evidence of practical utility. Your questions on the batch-level rule led us to derive its exact form, which explains the mechanism of Mod 1 more clearly and more precisely than our submission did. We respond below.

W1 and Question 1: What the batch-level rule does, exactly

We agree that Mod 1 is not conservative for every individual sample, and that our phrase "closes the NAHR escape route" should be stated precisely. Below we explain what Eq. 6 does from the definitions. The revised manuscript adds this account as Section 5.2 (Figure 5), with the derivation and worked examples in Appendix D.4 (Figure D.1). In short: GRPO lets negative-advantage high-ratio samples (Q2, NAHR) keep their unclipped term, so they can enlarge the objective without limit (Proposition 1). Mod 1 closes this route with a dynamic clip. The batch objective never exceeds its fully clipped value in magnitude, and Q2 samples lose their gradient once they would push it past that point.

Setup. Outside the clip range [1−ε, 1+ε], the ratio-advantage plane has four quadrants: Q1 (r > 1+ε, A > 0), Q2 or NAHR (r > 1+ε, A < 0), Q3 (r < 1−ε, A < 0) and Q4 (r < 1−ε, A > 0). For each sample, GRPO forms the unclipped term u = r·A and the clipped term c = clip(r)·A, and keeps min(u, c). This zeroes the gradient in Q1 and Q3 and keeps it in Q2 and Q4. CARE's batch-level Mod 1 instead averages each term over the M samples of a batch, s1 = mean(u) and s2 = mean(c), and keeps whichever of the two is closer to zero, for the whole batch (Eq. 6). So in a given batch, all out-of-range samples are either active (CARE keeps s1: the "unclipped branch") or have zero gradient (CARE keeps s2: the "clipped branch"). In-range samples, whose ratio lies in [1−ε, 1+ε], are always active, even when the batch as a whole takes the clipped branch.

Sample-level Mod 1: clip above the range, keep below it. For one sample, the unclipped term r·A is larger in size than the clipped term when r > 1+ε, and smaller when r < 1−ε, whatever the sign of A. CARE keeps the term that is smaller in size, so it clips Q1 and Q2 and keeps Q3 and Q4 active (Figures 2b and 4b). GRPO's min keeps the lower value instead. With A > 0 this is the same choice. With A < 0 it is the opposite one: GRPO keeps the growing term r·A in Q2, which is the escape route, and clips Q3.

Batch-level Mod 1: the gradient-active band. Eq. 6 can be understood in the following points (Figure 5).

  1. Where s1 starts. An in-range sample adds the same term to s1 and s2. So with no out-of-range samples, s1 = s2. Each out-of-range sample shifts s1 away from s2 by (r − clip(r))·A/M.
  2. Which way each quadrant sample shifts s1. The direction is fixed: Q1 and Q3 shift s1 to the right (they increase it), and Q2 and Q4 shift it to the left (they decrease it).
  3. When CARE keeps s1. Eq. 6 keeps s1 exactly when |s1| ≤ |s2|, that is, when s1 lies in CARE's gradient-active band, from −|s2| to +|s2|. Since s1 starts at s2, on one edge of this band, a net shift outward clips the batch. A net shift inward keeps s1, unless it is large enough to pass the opposite edge.

Whether "inward" means right or left depends only on the sign of s2:

  • s2 > 0: Q2 and Q4 shift s1 inward and are kept; Q1 and Q3 shift it outward and are clipped.
  • s2 < 0: the reverse. Uniform all-wrong batches under the Mod 2 fallback are always of this kind, since every advantage there is negative.
AuthorsOpenReview ↗

Response to Reviewer 7m8F. Part [2/2]

W1 and Question 1 (continued)

Three consequences address your W1.

  1. Q2: a dynamic clip. Lets take an example with ε = 0.2, M = 4 and A = (+1, +1, −1, −1): the two correct samples at r = 1.15 and one wrong sample at r = 0.85 are in range, and the other wrong sample is in Q2. Here s2 = (1.15 + 1.15 − 0.85 − 1.2)/4 = 0.0625 > 0. Since s2 is the mean of the clipped terms, the Q2 sample's ratio is replaced by its clipped value 1.2 in this sum. So s2 > 0, and the Q2 sample shifts s1 inward. It keeps its full corrective gradient, as in GRPO, up to r = 1.7, where s1 reaches −0.0625 = −s2. Beyond that, the whole batch is clipped. Figure D.1 plots the gradient on this sample against its ratio: under GRPO it grows without limit, sample-level Mod 1 cuts it at 1+ε, and CARE follows GRPO up to the batch's own threshold and then cuts it. With in-range ratios 1.1, 1.1 and 0.9 instead, the threshold is 1.4. Appendix D.4 works through both batches step by step. The clip on Q2 is always present; only its position depends on the batch. When s2 ≤ 0, a Q2 sample shifts s1 outward and is clipped as soon as it leaves the range, as in sample-level Mod 1.
  2. Objective bound. CARE keeps s1 only inside the band, so in every batch the applied objective is never larger in magnitude than the fully clipped one, which is itself at most (1+ε)·mean|A|. GRPO has no such bound (Proposition 1). In the example above at r = 3, GRPO's objective is −0.3875 and keeps growing with r, while CARE's stays at the clipped value 0.0625.
  3. Q3: opened in all-wrong batches. In a uniform all-wrong batch, s2 < 0, so s1 starts at the left edge of the band (Figure 5b). Q3 samples shift s1 right, into the band, and Q2 samples shift it left, out of it. So the batch keeps s1, and Q3 stays active, unless the Q2 samples shift s1 further left than the Q3 samples shift it right. (s1 cannot pass the right edge here, since every advantage is negative.) In particular, an all-wrong batch with no Q2 sample always keeps Q3 active, which GRPO never does. Appendix D.4 works through an example. This is the route to the entropy recovery in Figure 3 (Section 5).

Scope. As you point out, this is a guarantee on the batch objective, which is the quantity Eq. 6 controls, rather than a separate cap on each sample's gradient. Section 3 now states this scope explicitly.

Other changes to the paper. The abstract, introduction, Section 3 and related work now state that Mod 1 closes the Q2 (NAHR) escape route through a dynamic, batch-dependent clip. Section 3 defines "escape route". The captions of Figures 2 and 4 note that they show sample-level Mod 1.

We are grateful for this question. Working through it gave us a cleaner account of when Q2 and Q3 samples are active, and the revised paper is clearer because of it.

Empirical evidence. Under identical settings, Mod 1 alone lifts GRPO from 56.38 to 74.97, and the batch rule outperforms sample-level Mod 1 at 73.35 (Table 2; Table G.1).


W2 and Question 2: Pooling, the fallback, and Mod 1

The paper treats the two parts of Mod 2 separately. Mod 2 pools the advantage over the global batch, the same pooling used by REINFORCE++ (Section 2). Pooling alone, however, cannot remove the failure, since any mean-centered advantage vanishes once every reward in the pool coincides (Section 3.3). Section 6 notes that batch-level pooling trades per-prompt difficulty calibration for statistical stability, and proposes hybrid normalization as future work.

Two existing results bear on your question. First, REINFORCE++ is batch pooling without the fallback or Mod 1. Over three seeds it reaches 70.68 ± 1.03 against 70.44 ± 0.66 for GRPO. Second, Figure 1b shows the fallback firing on more than half of the batches after step 250 (Appendix D). Without it, those batches carry exactly zero gradient.

On the contribution of Mod 1: on top of Mod 2 it adds +0.92 (Table 2). Under the same configuration, Mod 1 alone lifts GRPO from 56.38 (Table G.1) to 74.97 (Table 2), a gain of +18.59. The two modifications address different failures. Mod 2 restores a learning signal when rewards are uniform but leaves the clipping rule unchanged. Mod 1 changes the clipping rule: it bounds Q2 (NAHR) updates and lets Q3 samples move. They are also most active in different phases of training: Mod 1 early, and Mod 2 later (Figure 1b). In addition, Mod 1 complements the fallback: in the uniform all-wrong batches that the fallback handles, Mod 1 can keep Q3 samples active (point 3 above), which GRPO's clipping never allows.


We once again thank you for your valuable comments! If you have any further questions, please feel free to let us know and we will do our utmost to respond. If you find our replies satisfactory, we kindly ask that you consider updating your score accordingly.

Thank you for the detailed response. The revised batch-level analysis addresses most of my concern in W1, while W2 remains open. I would need an ablation that toggles the zero-variance fallback while holding the normalization pool and all other training settings fixed. I would also like to see full CARE and Mod 2 alone compared across the same seeds to assess whether Mod 1 provides a reproducible additional benefit. After considering your response and the other reviewers’ assessments, I will keep the original score.

Reviewer_UdhKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies RLVR for generative classification, where the model outputs a short class label. It identifies two problems with GRPO-style optimizers in this setting: negative-advantage, high-ratio (NAHR) samples can cause aggressive updates, while advantages vanish when all rewards in a group are identical. The authors propose CARE, which selects between clipped and unclipped surrogates at the batch level and uses raw rewards when the batch has no reward variance. On RAGTruth QA, CARE improves macro-F1 by 11.5 points over GRPO. The paper addresses an interesting problem and proposes a simple, practical solution with strong initial results. Although the claims about conservative updates and entropy recovery need further validation, I find the approach promising and believe it makes a useful contribution to RLVR for generative classification. I therefore lean towards acceptance.

Critical strengths

The problem is clear and well motivated. The paper explains why low action entropy and limited reward diversity can hurt GRPO-style training. The analysis of NAHR samples and uniform groups is useful. In addition, the method is simple and practical. It requires relatively small changes to standard GRPO and introduces no new hyperparameters. The in-distribution gains are strong across three seeds, with encouraging results on a second backbone, OOD datasets, and ANLI.

Critical weaknesses
  • The batch-level selection may not be conservative for individual samples. Since the branch is selected using aggregate magnitudes, contributions from different samples can cancel each other out. It is therefore unclear whether this mechanism consistently controls NAHR samples. Per-sample gradient or KL analysis, as well as a comparison with Dual-Clip PPO, would strengthen the claim.
  • The experimental evidence is limited in some areas. The OOD results and ablation use a single seed, while the entropy recovery result also needs stronger validation. Different baseline configurations make it difficult to attribute the improvements solely to CARE. Moreover, Mod 2 alone achieves 79.49, compared with 80.41 for the full method, suggesting that the additional contribution of Mod 1 is relatively small.
Questions for authors
  • How frequently is each surrogate branch selected during training, and how much NAHR contribution is present in those batches?
  • How does CARE compare with GRPO using Dual-Clip PPO?
  • Is entropy recovery consistent across multiple seeds, and how does it compare with an explicit entropy bonus?
  • What is the batch size, and how frequently does the raw-reward fallback in Mod 2 activate?
2 comments
AuthorsOpenReview ↗

Response to Reviewer UdhK. Part [1/2]

We sincerely thank you for your careful and encouraging review. We are glad you found the problem clear and well motivated, and the analysis of NAHR samples and uniform groups useful. We also appreciate your recognition that CARE is simple and practical, needs only small changes to GRPO, and adds no new hyperparameters. Thank you as well for noting the strong three-seed gains and the encouraging results on a second backbone, OOD data, and ANLI. Your questions helped us state Mod 1's guarantee more precisely and position it against Dual-Clip PPO. We respond below.

W1: Batch-level selection You are right that Mod 1 does not guarantee conservatism for every individual sample. Its guarantee is at the batch level: the applied objective always takes the smaller magnitude of the two batch-mean surrogates (Eq. 6), so it never exceeds (1+ε) times the mean advantage magnitude.

This is by design. For an NAHR sample, the unclipped gradient is corrective: it lowers the probability of an action that was wrongly reinforced. The problem in GRPO is that nothing bounds it (Proposition 1). Sample-level Mod 1 removes this gradient for every NAHR sample. Batch-level Mod 1 can keep it, but not without limit. Let s1 and s2 be the unclipped and clipped batch means. CARE keeps s1 only while it lies in its gradient-active band [−|s2|, +|s2|]. A growing NAHR ratio keeps decreasing s1 and leaves s2 unchanged, so, with the rest of the batch fixed, it always pushes s1 out of the band and switches the whole batch to the clipped surrogate. Mod 1 thus closes the NAHR escape route with a dynamic clip whose position is set by the batch, not a fixed constant. Applied to each sample (sample-level Mod 1), the same rule gives the regions of Figures 2b and 4b. Section 5.2 (Figure 5) and Appendix D.4 in the updated manuscript now illustrate this. Figure D.1 gives the per-sample gradient analytically; we have not measured it empirically.

Mod 1 is first a selection rule, and granularity is a separate choice that we ablate (Table 2): 73.35 per sample versus 74.97 per batch. Proposition 3 (Appendix C) shows that the batch-level decision also has lower variance.


Question 2: Dual-Clip PPO

Thank you for the pointer. For negative advantages, Dual-Clip PPO (Ye et al., 2020) adds a floor cA to the objective, so an NAHR sample loses its gradient once r > c. It differs from CARE in three ways. Its ceiling is a fixed hyperparameter, so NAHR samples with 1+ε < r < c keep their full gradient regardless of the batch. Q3 stays clipped as in GRPO, closing the entropy-recovery route of Section 5. Uniform groups still give zero gradient (Failure Mode 2). On Q2 samples, sample-level Mod 1 equals Dual-Clip PPO at c = 1+ε. Section 2 and Table 1 of updated manuscript now include it (c = 3, CARE's configuration). At the same configuration and seed, it reaches 70.29, against 56.38 for GRPO, 73.35 using sample-level Mod 1 and 74.97 using batch-level Mod 1 (Tables 2 and G.1). Over three seeds it reaches 72.52 in-distribution, 2.1 points above GRPO and 9.4 below CARE.


AuthorsOpenReview ↗

Response to Reviewer UdhK. Part [2/2]

Questions 1 and 4: Activation frequency and batch size

Figure 1b tracks the activation of both modifications. Mod 1 counts as active on a batch when its selection differs from GRPO's elementwise min (now stated in Appendix D.2). It is most active early, when NAHR samples are most frequent (Appendix D.1), and then decays. The Mod 2 fallback first fires near step 150 and is active on over half of all batches by step 250. Each batch has 16 samples per GPU (Table F.1).


W2: Seeds, configurations, and Mod 1

Seeds. In-distribution, CARE reaches 81.90 ± 1.25 over three seeds against 70.44 ± 0.66 for GRPO. The 11.5-point gap is far larger than the seed spread.

Configurations. Under CARE's exact configuration, GRPO, Dr. GRPO, and DAPO collapse (56.38, 58.24, 56.26; Table G.1), while CARE reaches 80.41. Table 1 therefore reports each baseline at its stronger recommended configuration. Under the same configuration, Mod 1 alone lifts GRPO from 56.38 to 74.97.

Mod 1 versus Mod 2. We agree that Mod 1 adds little on top of Mod 2 on RAGTruth QA (+0.92). But the point to note is that the two address different failures in different phases of training (Figure 1b). Mod 2 leaves the clipping rule unchanged, so Mod 1 is the component that controls NAHR updates and opens Q3.


Question 3: Entropy recovery

Section 5 is a mechanism argument that follows from the objective itself. Whenever the batch keeps the unclipped surrogate, Q3 samples stay gradient-active, which GRPO never allows. In uniform all-wrong batches under the Mod 2 fallback, Q3 stays active unless the NAHR ratios lie further above the clip range, in total, than the Q3 ratios lie below it. Figure 3 shows one seed; we have not measured entropy across seeds, but the mechanism does not depend on the seed. An entropy bonus is a complementary tool with its own coefficient. Our claim is that CARE recovers entropy without one, not that it outperforms one.


We once again thank you for your valuable comments! If you have any further questions, please feel free to let us know and we will do our utmost to respond. If you find our replies satisfactory, we kindly ask that you consider updating your score and confidence accordingly.

Comments on the paper

2 comments

Dear Reviewers,

A friendly reminder to check the authors’ responses and the other reviewers’ comments. It would be helpful to hear your thoughts on the differing assessments and whether they affect your evaluation.

Thank you for your time and effort!

AC

AuthorsOpenReview ↗

General Response to All Reviewers and Area Chairs

We thank all four reviewers for their time and careful feedback. We are encouraged that reviewers found:

  • the setting practically important and under-studied (7m8F, Yq4P), and the problem well motivated (Udhk);
  • the paper clear and easy to follow (cd6a, Udhk);
  • the analysis of the two failure modes insightful and useful (cd6a, Udhk);
  • the method simple, composable, and free of new hyperparameters (Udhk, 7m8F, Yq4P);
  • the experiments strong and broad: three seeds in-distribution, seven out-of-distribution sets, a second backbone, a multi-class task, and baselines beyond the GRPO family (cd6a, Udhk, 7m8F, Yq4P).

Shared points

  1. What batch-level Mod 1 guarantees (Udhk, 7m8F, Yq4P). Let s1 and s2 be the unclipped and clipped batch means. Eq. 6 keeps s1 only while it lies in CARE's gradient-active band, from −|s2| to +|s2|. Three consequences follow:

    • The applied batch objective is never larger in magnitude than the fully clipped one, in every batch. GRPO has no such bound (Proposition 1).
    • A Q2 (NAHR) sample keeps its corrective gradient only up to a threshold set by the rest of the batch. Batch-level Mod 1 is therefore a dynamic clip, not a fixed one as in sample-level Mod 1 or Dual-Clip PPO.
    • Opening Q3 is intended and stays inside this bound. It is the route to entropy recovery (Section 5).

    We agree that this is a guarantee on the batch objective rather than a cap on each sample's gradient, and the revision states this scope. The batch rule also outperforms its per-sample version (74.97 vs 73.35, Table 2). Section 5.2 (Figure 5) and Appendix D.4 give the full derivation and worked examples.

  2. Mod 1 on top of Mod 2 (Udhk, 7m8F). The +0.92 gain of Mod 1 on top of Mod 2 is small. Under the same configuration, however, Mod 1 alone lifts GRPO from 56.38 to 74.97 (Tables G.1 and 2). The two address different failures and are most active in different phases of training: Mod 1 early, Mod 2 after about step 250 (Figure 1b). Mod 2 leaves the clipping rule unchanged.

  3. Seeds (Udhk, 7m8F, Yq4P). In-distribution, three seeds give 81.90 ± 1.25 for CARE against 70.44 ± 0.66 for GRPO. Out-of-distribution entries are single-seed, as the paper states. The main gaps there, for example is more than 20 points over SFT on Internal QA and HaluEval Summarization, are far larger than the in-distribution seed spread.

  4. Additional comparisons (cd6a, Udhk, Yq4P). Our baselines hold the backbone, data, reward and budget fixed, so that the objective is the only variable. Each reply explains how the suggested method relates to CARE. For Dual-Clip PPO (Udhk, Yq4P), which Section 2 now discusses, sample-level Mod 1 equals Dual-Clip PPO at its tightest setting, c = 1+ε, on Q2 (NAHR) samples. Table 1 now includes Dual-Clip PPO, run at CARE's configuration with c = 3. At the same configuration and seed, it reaches 70.29, against 56.38 for GRPO, 73.35 for sample-level Mod 1 and 74.97 for batch-level Mod 1 (Tables 2 and G.1). Over three seeds it reaches 72.52 ± 1.59 in-distribution, 2.1 points above GRPO and 9.4 below CARE.

Revisions

  • Abstract, introduction, Section 3 and related work: Mod 1 closes the NAHR escape route through a dynamic, batch-dependent clip. Section 3 defines "escape route": the Q2 term is corrective in direction but has no ceiling. It also states that the guarantee applies to the batch objective.
  • New Section 5.2 with Figure 5: sample-level versus batch-level Mod 1, CARE's gradient-active band, and its consequences.
  • New Appendix D.4: derivation, two worked examples and Figure D.1. Appendix D.2 now defines when Mod 1 counts as active.
  • Figures 2 and 4: captions note that the panels show sample-level Mod 1.
  • Section 4.3 now states the LLaMA result precisely.
  • Related work: Dual-Clip PPO. Table 1 and Appendices B, F and G: Dual-Clip PPO as a baseline, with its loss, hyperparameters and per-seed results.

Points we invite reviewers and ACs to consider

  1. A practical tool. For anyone working on generative classification, CARE is a new algorithmic option to try alongside GRPO, TRULY-PPO, REINFORCE++ and others. It adds no new hyperparameters and needs only a small number of changes to a standard GRPO implementation.
  2. Practical problems, diagnosed and fixed. We identify two concrete failures of the default RLVR objective in a widely deployed setting, explain why they arise (Propositions 1 and 2), and fix both.
  3. A new direction. The quadrant analysis (Section 5) shows where the PPO/GRPO surrogate keeps or blocks the gradient. Changing which regions stay active changes training dynamics, for example enabling entropy recovery. This opens a direction beyond this paper: designing the surrogate region by region.

We hope these points help in the overall assessment. Thank you again for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.