acceptodds
Under review as a conference paper at ICLR 2027

Down-Weighting Uncertain Supervision: Frozen Checkpoint-Label Compatibility Weighting for GRPO

Abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as an effective paradigm for fine-tuning vision-language models on classification tasks, yet the standard objective does not explicitly distinguish the reliability of the observed supervision associated with different training samples. We study SCS-GRPO, which assigns each sample a frozen checkpoint-observed label-compatibility score and applies it as a post-normalization multiplier on the GRPO advantage. We analyze why applying a prompt-level scalar before within-group normalization can largely remove its intended effect, and give a conditional gradient signal-to-noise analysis under explicit assumptions about corrupted supervision and the compatibility ranking. Experiments on multiple public benchmarks show consistent improvements in the evaluated classification and noise settings. An anonymous implementation is available.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Large Language Models, Noise Label Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EoNvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies GRPO-style RLVR fine-tuning of VLMs for binary classification when some training supervision may be corrupted. It proposes SCS-GRPO. Before RL, a frozen cold-start (SFT) checkpoint scores each training example by the probability it gives the observed label, renormalized over the class verbalizers. The weight SCS_i = q_i(ỹ_i)^{1/τ} (τ = 5) then multiplies the group-normalized advantages of that prompt's rollouts. There are two points that motivate this design:

  • A prompt-constant scalar applied to rewards before group standardization cancels out (Prop. 1), so the weight is applied after normalization.
  • A gradient signal-to-noise decomposition states when non-uniform weights improve SNR, with an oracle corollary In the experiments, the papers finetuned Qwen3-VL-2B/4B-Thinkin on 3 datasets: Hateful Memes (HM), CHASM and Civil Comments (CC). The results show that SCS has the best score on all metrics across 3 datasets. There are also ablations over τ and group size
Critical strengths
  1. The method needs one offline scoring pass plus a scalar multiplication, and it applies to any GRPO variant. It is simple and cheap. A drop-in weighting with gains of the reported size is useful.
  2. The paper covers 3 datasets (2 multimodal, 1 text-only), 2 model scales, and 3 seeds with standard deviations. There is consistency of gains across these settings.
Critical weaknesses

I feel that the experiments in the paper do not isolate the proposed mechanism (1) The paper has no runs with shuffled SCS weights, a constant weight, or an oracle that down-weights the truly corrupted samples. It also never checks that SCS gives corrupted samples low scores, for example with an AUROC. SCS also scales only the policy-gradient term and not the KL term. So even a constant weight changes the effective regularization. (2) The \delta ablation in Table 3 does not fit the SNR explanation. HM Macro-F1 stays between 62.7 and 64.0 from \delta=1 which down-weights strongly, to \delta=10 where the weights are almost uniform. And fully uniform weights (vanilla GRPO) score about 10 points lower (53.3). If suppressing corrupted samples drives the gain, performance should fall gradually as \delta grows, not stay flat and then drop. (3) The noise sweep (Fig. 2a) does not show robustness. All methods improve as label noise rises from 0 to 40% (for example, vanilla GRPO goes from 56.5 to 59.6). SCS's outperformance over vanilla GRPO is largest with noise (+4.3) and smaller at high noise (+2.6 to +2.8). A class-prior effect might be a plausible explanation for this as the HM training set is about 36% hateful and the dev set about 50%. No threshold-free metric such as AUROC is reported.

Questions for authors
  1. which configuration produced Table 2, 3, 4 and Figure 2? can you specify the sampling temperature, G, lr, \beta_KL, LoRA vs full finetuning, reward terms and weight formula with \delta.
  2. please show the SCS weight distribution for each dataset, split by class and by clean vs. corrupted, together with the AUROC of SCS for flagging corrupted examples.
  3. In Fig. 2a, why does every method improve as label noise increases, and why is SCS's margin largest at η = 0? Also, at η = 20% the DAPO (59.9) and Dr.GRPO (58.9) values equal their Table 2 modality-swap values. Are these independent runs?
3 comments
AuthorsOpenReview ↗

Response to reviewer EoNv (part 1)

Thank you for pointing out where the configuration, weighting controls, and noise sweep need clarification.

Macro-F1 is mean ± SD over three RL seeds (42/123/456); values quoted from the submitted tables, Figure 2(a), or a diagnostic batch are reported as means with SDs omitted. AUROC and AUPRC are reported separately.

Q1. Configuration. We used full-parameter fine-tuning for cold-start SFT and RL, and a sampling temperature of 1.0 for RL rollouts. The default RL settings in Appendix A are G=8G=8, learning rate 3×10−63\times10^{-6}, βKL=0.01\beta_{\mathrm{KL}}=0.01, one epoch, batch size 16, and seeds 42/123/456. The verifier adds 11 for matching the observed label, 22 for valid answer format, and 0.5min⁡(1,L/512)0.5\min(1,L/512) for response length. The SCS weight is wi=qi(y~i)1/τw_i=q_i(\tilde y_i)^{1/\tau}, where qiq_i is the frozen checkpoint's next-token distribution renormalized over the valid class verbalizers. The default weight temperature is τ=5\tau=5. The temperature you denote δ\delta is τ\tau in the paper (Eq. (3), Table 3): δ=1\delta=1 is the most selective setting and large δ\delta approaches uniform weighting.

For HM, Table 2 and the HM columns of Tables 3–4 use 20% image swaps. Table 3 varies τ∈{1,3,5,10}\tau\in\{1,3,5,10\}, and Table 4 varies G∈{4,8,16}G\in\{4,8,16\}. CHASM and Civil Comments use their original annotations. Figure 2(a) varies the symmetric label-flip rate on HM from 0% to 40%, and Figure 2(b) uses the 20% image-swap setting. In the submitted HM experiments, both corruption protocols start from a checkpoint trained on the original pairs. The intervention is applied to the RL training set, with the evaluation split held fixed.

AuthorsOpenReview ↗

Response to reviewer EoNv (part 2)

Continuing our response (part 2).

Macro-F1 is mean ± SD over three RL seeds (42/123/456); values quoted from the submitted tables, Figure 2(a), or a diagnostic batch are reported as means with SDs omitted. AUROC and AUPRC are reported separately.

Q2. Weighting controls and corruption detection. On HM with 20% image swaps, SCS achieved 64.0±0.264.0\pm0.2 Macro-F1, compared with 53.7±0.953.7\pm0.9 for permuting the same weights within each observed class. This comparison preserves each class's weight distribution and tests whether assigning those weights to the original examples matters. The realized policy update and its interaction with KL can still change, and mean-matched constant weighting and mean-normalized SCS remain open controls. A two-level control using the known swap indicator achieved 63.3 Macro-F1 in the diagnostic batch; we do not treat this as a strict upper bound.

Using low SCS to identify swapped examples gave swap-detection AUROC/AUPRC of 0.842/0.818, compared with 0.501/0.202 after within-class shuffling; the injected-swap mask is used only as an evaluation label and does not enter the method. These are detection metrics for the image-swap protocol, not downstream classification metrics, and the AUPRC random baseline equals the positive-class prevalence (20% under this protocol). The requested class-stratified weight distributions, detection metrics for actual label flips, and held-out classification AUROC remain open; all three require only a single evaluation forward pass with no additional RL training, and we will add them in the camera-ready. For CHASM and Civil Comments, estimating detection performance would require independently verified corruption labels.

Table 3 reports 63.1, 63.8, 64.0, and 62.7 HM Macro-F1 for τ=1,3,5,10\tau=1,3,5,10, respectively. Finite τ=10\tau=10 can still produce different weights when observed-label probabilities vary substantially. Unit weights recover the Vanilla GRPO objective, and 53.3 is the Table 2 Vanilla result. Appendix B gives a conditional comparison of gradient SNR and makes no prediction of a monotonic Macro-F1 curve as τ\tau changes.

In a separate matched rebuttal batch we measured the endpoints outside that sweep: τ=0.5\tau=0.5 gives 61.8±0.861.8\pm0.8 and τ=20\tau=20 gives 59.7±0.859.7\pm0.8, while the uniform limit is the 53.3 Vanilla mean above. Together these form an inverted U rather than a flat plateau followed by a discontinuity. Since wi=qi(y~i)1/τw_i=q_i(\tilde y_i)^{1/\tau} is a monotone power transform, changing τ\tau rescales the weights without reordering them; insensitivity across τ∈[1,10]\tau\in[1,10] is therefore consistent with a mechanism that depends on the weight ordering, and the decline is graded once that ordering is flattened. The sweep alone does not isolate the mechanism — the isolation comes from the within-class shuffle above.

AuthorsOpenReview ↗

Response to reviewer EoNv (part 3)

Continuing our response (part 3).

Macro-F1 is mean ± SD over three RL seeds (42/123/456); values quoted from the submitted tables, Figure 2(a), or a diagnostic batch are reported as means with SDs omitted. AUROC and AUPRC are reported separately.

Q3. Label-flip sweep and repeated values. In Figure 2(a), SCS exceeds Vanilla GRPO at all five plotted rates, with the largest gap at 0% injected flips (60.8 vs. 56.5, +4.3 pp). Macro-F1 does not decrease monotonically with increasing flips. Symmetric flips move an imbalanced training label distribution toward 50/50. For example, if the original positive fraction is 36%, a 40% flip rate would move its expectation to 47.2%. The sweep alone leaves the effect of this prior shift and the mechanism behind the zero-flip gain unresolved.

Yes, the 20% label-flip results in Figure 2(a) and the 20% image-swap results in Table 2 came from independent training runs under their respective corruption protocols. The reported Macro-F1 values happen to coincide in both settings for DAPO (59.9) and Dr.GRPO (58.9). The per-seed records below show that this agreement holds only after rounding to one decimal.

Per-seed run records.

ProtocolMethodSeed 42Seed 123Seed 456MeanSD
20% label flip (Fig. 2a)DAPO59.4160.2460.1359.930.45
20% image swap (Table 2)DAPO60.3159.5559.7259.860.40
20% label flip (Fig. 2a)Dr.GRPO59.2558.4058.9958.880.44
20% image swap (Table 2)Dr.GRPO58.2359.3059.2458.920.60

Table D. Per-seed HM Macro-F1 from RL seeds 42, 123, and 456 under each corruption protocol. Means and sample SDs (denominator n−1) use full-precision scores; values are shown to two decimals here and to one decimal in the submitted Table 2. The two rows for each method come from separate training runs under the label-flip and image-swap protocols, respectively.

For completeness, the underlying per-seed (TP, TN) counts, in seed order, are:

ProtocolMethodSeed 42Seed 123Seed 456
20% label flip (Fig. 2a)DAPO(103, 202)(105, 204)(106, 202)
20% image swap (Table 2)DAPO(106, 203)(108, 196)(108, 197)
20% label flip (Fig. 2a)Dr.GRPO(92, 217)(94, 209)(95, 211)
20% image swap (Table 2)Dr.GRPO(83, 225)(87, 225)(96, 211)
Reviewer_TbwVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work studies how to handle uncertain or noisy supervision in reinforcement learning with verifiable rewards, specifically focusing on GRPO. The authors propose SCS-GRPO, which computes a Sample Credibility Score based on the frozen checkpoint's compatibility with the observed label, and applies it as a post-normalization multiplier on the GRPO advantage. The paper provides theoretical justification regarding why prompt-level scaling fails before group normalization, along with a conditional gradient signal-to-noise ratio analysis. Experiments across multiple benchmarks demonstrate consistent performance improvements over vanilla GRPO and other baselines under both noisy and clean conditions.

Critical strengths

SCS-GRPO is practically appealing because it requires only a single offline, cached scoring pass using a frozen pre-RL checkpoint, adding minimal overhead to standard GRPO without needing explicit noise-rate estimators or separately trained detectors. The evaluation spans multiple diverse benchmarks including multimodal and text-only datasets and examines robustness across various label-noise rates and model scales, complemented by clear advantage distribution analyses.

Critical weaknesses
  1. The core assumption relies on frozen checkpoint-label compatibility as a proxy for supervision reliability. However, as the authors acknowledge, low compatibility can easily stem from genuine sample difficulty, domain shift, or anchor bias rather than actual label corruption, making the weighting prone to penalizing hard-but-correct examples.

  2. Multiplying advantage-like policy update terms by prompt-level scalars or confidence/difficulty weights is conceptually well-explored in existing literature (such as curriculum learning and various reward-shaping strategies). The technical contribution is largely restricted to resolving the within-group normalization cancellation issue and applying it to GRPO.

  3. The method achieves significant performance gains even in clean, uncorrupted data settings (e.g., on clean benchmarks and η=0%\eta = 0\% noise rates). While the authors attribute this to an empirical regularization effect, the exact mechanism by which down-weighting low-compatibility samples benefits clean datasets lacks rigorous theoretical explanation.

Questions for authors

Since low checkpoint-label compatibility can confound label noise with genuine sample difficulty, how does SCS-GRPO ensure that hard-but-correct instances are not disproportionately suppressed during the early stages of policy optimization, and what empirical evidence distinguishes noise suppression from general regularization in clean-data experiments?

2 comments
AuthorsOpenReview ↗

Response to reviewer TbwV (part 1)

Thank you for asking how SCS affects difficult, correctly labeled examples and what explains its gains without injected label noise. Low checkpoint–label compatibility can reflect difficulty, domain shift, or checkpoint error as well as corrupted supervision. For the matched HM comparisons, we use the shared initialization within each experiment, the same verifier reward, eight rollouts per prompt, one epoch, and a KL coefficient of 0.01.

Macro-F1 is mean ± SD over three RL seeds (42/123/456); values quoted from the submitted tables, Figure 2(a), or a diagnostic batch are reported as means with SDs omitted. AUROC and AUPRC are reported separately.

Hard examples and early training. We stratified examples by class-balanced, five-fold out-of-fold negative log likelihood on the original labels, using checkpoints separate from the SCS anchor. With 20% modality swaps in Hateful Memes (HM) training, Macro-F1 on the hardest quartile of the unchanged evaluation split was 48.9 ± 0.7 for SCS, 44.3 ± 1.0 for Vanilla GRPO, and 46.8 ± 0.8 for DAPO (mean ± SD over three RL seeds). This gives a direct comparison of the final policies on difficult inputs. The question about early optimization remains open because we did not measure per-step contributions from difficult training examples. Freezing the scores keeps their values fixed while the policy changes, so persistent misranking can still attenuate useful updates.

In a separate diagnostic, we held each input and checkpoint fixed and switched only the attached binary label. Mean compatibility weight fell from 0.968 to 0.684 on inputs the anchor classified correctly and rose from 0.725 to 0.956 on inputs it classified incorrectly. In the hardest quartile of this diagnostic, the mean weight rose from 0.804 to 0.912 when switching from the original to the opposite label. These results show where the score can favor the flipped label. When we introduced the 20% modality swaps before teacher generation, filtering, and SFT, SCS reached 61.1 Macro-F1 against 52.4 for Vanilla GRPO using the same resulting checkpoint (+8.7 pp, compared with +10.7 pp under the original protocol). AUROC for identifying the injected swaps fell from 0.842 to 0.771. The smaller gain with the corrupted cold start supports treating anchor quality as a limitation.

Checkpoint probabilities / SCS weightsTask ECE / BrierSwap-detection AUROC / AUPRCHM Macro-F1
Raw checkpoint / raw SCS0.108 / 0.1460.842 / 0.81864.0 ± 0.2
Temperature-scaled checkpoint / mapped SCS0.039 / 0.1210.842 / 0.81863.9 ± 0.2
Raw checkpoint / within-class shuffled SCS0.108 / 0.1460.501 / 0.20253.7 ± 0.9

Table A. HM with 20% image swaps. Temperature scaling is fitted on clean validation for this diagnostic. Macro-F1 is the mean ± SD across three RL seeds.

Scoring scheduleFinal swap-detection AUROCHM Macro-F1
Frozen pre-RL checkpoint0.84264.0
Refresh every 100 updates0.80963.4
Refresh every 25 updates0.75462.4
Current-policy score every update0.69161.3

Table B. Separate matched batch. Macro-F1 is the batch mean, with between-seed SDs of 0.5–0.8. For refreshed variants, we reassign the original weights according to the updated ranking and evaluate each final scorer on the same holdout.

AuthorsOpenReview ↗

Response to reviewer TbwV (part 2)

Continuing our response (part 2).

Macro-F1 is mean ± SD over three RL seeds (42/123/456); values quoted from the submitted tables, Figure 2(a), or a diagnostic batch are reported as means with SDs omitted. AUROC and AUPRC are reported separately.

Clean-data gains. The HM setting with no injected label flips gives 60.8 vs. 56.5 Macro-F1 for SCS and Vanilla GRPO. This +4.3 pp gap is the largest in the five-rate sweep. It establishes a gain in that setting, with its cause unresolved. CHASM and Civil Comments similarly use their original annotations, which may still contain uncertainty.

In the separate 20% modality-swap setting, SCS reached 64.0 ± 0.2, compared with 53.7 ± 0.9 when SCS weights were shuffled within the observed class and 53.3 ± 0.7 for Vanilla GRPO. AUROC for detecting the injected swaps dropped from 0.842 to 0.501 with this shuffle. The shuffle preserves each class's weight distribution and changes which weights are assigned to which examples. This supports the value of example-specific assignment in this setting. The policy gradient and its balance with the KL penalty can still change when weights are reassigned. A constant multiplier and mean-normalized SCS would help separate these effects. We have yet to run the corresponding controls at zero injected label flips, so the swap experiment cannot distinguish mechanisms for the clean-setting gain. We will describe the gain as an empirical result. Appendix B gives conditions for an SNR improvement, and we have not measured the gradient terms needed to assess those conditions here.

Scope of the contribution. We agree that multiplying an advantage by a prompt-level scalar is established. SCS uses an offline weight conditioned on the observed label under a frozen checkpoint. SEED-GRPO derives its weight from the current rollout group's semantic dispersion independently of the attached label. On HM with 20% modality swaps, the SEED weighting rule adapted to the HM setup reached 58.6 ± 0.7 after validation tuning under the shared protocol, compared with 64.0 ± 0.2 for SCS. We will clarify the scope of the comparison and of Proposition 1. For a positive multiplier shared by a nondegenerate group, scaling rewards before group standardization cancels exactly with a zero stabilizer. With a finite stabilizer, the residual is given in Appendix B. We will present this identity as the rationale for where we apply the compatibility score and focus the contribution claim on the score itself and its empirical evaluation.

MethodHM Macro-F1
Vanilla GRPO (Table 2)53.3 ± 0.7
DAPO (Table 2)59.9 ± 0.4
SEED, published linear weighting58.0 ± 0.7
SEED, validation-tuned weighting58.6 ± 0.7
SEED, weight distribution matched to SCS58.3 ± 0.6
SCS, shuffled within observed class53.7 ± 0.9
SCS-GRPO (Table 2)64.0 ± 0.2

Table C. HM with 20% image swaps. Each method generates its own rollouts under the shared protocol. Macro-F1 is the mean ± SD across three RL seeds.

Reviewer_4fqXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies SCS-GRPO, a simple modification of GRPO for classification with potentially imperfect supervision. A frozen pre-RL checkpoint assigns each training example a score based on the probability of its observed label, renormalized over the valid answer classes. This score is cached and multiplies the example's advantage after within-group normalization. The paper explains why applying a prompt-constant multiplier before normalization can cancel its effect, and provides a conditional gradient signal-to-noise analysis. Experiments cover Hateful Memes, CHASM, and Civil Comments with Qwen3-VL-2B-Thinking and Qwen3-VL-4B-Thinking, including controlled corruption, unmodified annotations, and temperature/group-size studies. The results suggest that frozen compatibility weighting can improve the evaluated GRPO training setups, but the contribution of the particular scoring signal and its dependence on the initialization remain insufficiently isolated.

Critical strengths
  1. The method is simple, clearly specified, and appropriately qualified. Equations (3)–(6) make the scoring rule and its placement in the objective explicit, while Appendix A and Appendix D specify the verbalizers and scoring prefix. The normalization analysis correctly distinguishes exact cancellation from the finite-stabilizer case. I also appreciate the explicit distinction between checkpoint–label compatibility and the probability that an annotation is correct. Appendix B presents its SNR result conditionally and acknowledges that down-weighting difficult clean examples can remove useful signal. These qualifications make the technical claim considerably more credible than an unconditional assertion that low confidence identifies bad supervision. Caching the scores also makes the proposed modification practical to implement.

  2. The reported improvements are substantial and reasonably consistent within the evaluated setting. Table 2 covers two model sizes and three datasets, and reports means and standard deviations over three RL seeds. For the 2B model, SCS improves Macro-F1 over DAPO by 4.1, 4.8, and 3.7 percentage points on Hateful Memes, CHASM, and Civil Comments, respectively. The improvements also appear with the 4B model. The inclusion of both input corruption and a separate label-flip sweep, together with temperature and group-size ablations, provides useful evidence that the result is not confined to one hyperparameter choice. The per-class results in Appendix C further help interpret the aggregate gains.

Critical weaknesses

W1. The experiments do not sufficiently isolate the benefit of the proposed compatibility signal from simpler weighting and optimization effects.

  • The paper appropriately states that post-normalization advantage weighting is not itself new (Section 3.3). The main contribution therefore rests on the usefulness of the frozen, observed-label-conditioned score. However, the evaluated baselines do not establish why this particular signal is necessary. The closest advantage-modulation comparison, SEED-GRPO, appears only in the conceptual discussion; the missing empirical comparison is acknowledged in Section 5.
  • In Equation (6), all compatibility weights are at most one and scale the policy surrogate while leaving the KL penalty unchanged. Consequently, the intervention changes both the allocation of update weight across examples and the balance between the policy objective and regularization. The paper does not report a constant-weight control matched to the mean SCS, shuffled weights preserving the weight distribution, or mean-normalized SCS. These controls would help determine whether the gains depend on the example-to-weight association or on a broader change in update scale. The temperature ablation changes both factors simultaneously and does not resolve this issue.
  • Figure 2(b) shows contraction of the effective advantage distribution, but this follows partly from multiplying by values in (0,1](0,1]. The authors correctly acknowledge that this is an implementation check rather than a causal explanation. As a result, the substantial performance gains remain insufficiently connected to the specific mechanism that distinguishes SCS from simpler alternatives. This limits the strength of the methodological contribution.

W2. The evaluation does not establish when the frozen checkpoint's ranking remains useful under uncertain supervision.

  • In the controlled Hateful Memes experiments, the cold-start checkpoint is trained on the original training pairs, and corruption is introduced only afterward into the RL data (Section 4.1). This is a fair shared initialization for the compared methods, but it gives the compatibility scorer prior exposure to the original examples and annotations. It does not test the more difficult situation in which the checkpoint was itself trained on imperfect supervision or makes systematic errors on the examples it subsequently scores. This distinction matters because the frozen ranking is the central assumption behind the method.
  • The label-flip experiments provide known corruption indicators, yet the paper does not report how SCS separates corrupted examples from difficult but correctly labeled ones, or how much useful learning signal is suppressed in the latter group. Such diagnostics are especially important because Section 4.3 reports the largest improvement over Vanilla GRPO in the clean condition and no monotonic relationship between the gain and noise rate. These observations are compatible with several explanations and do not identify supervision-reliability discrimination as the source of the improvement.
  • Appendix B's SNR decomposition is mathematically conditional; it does not establish that the proposed weights satisfy the beneficial regime in the actual training runs. I do not regard this as a false theorem or an unsupported universal guarantee. The concern is the missing empirical link between the stated condition, the frozen compatibility ranking, and the observed improvement. For an otherwise modest modification, stronger evidence on this link would materially affect my assessment.
Questions for authors
  1. Contribution of the scoring signal: How do SCS, a constant multiplier matched to its mean, randomly reassigned SCS weights, and mean-normalized SCS compare under matched training and tuning budgets? Do the gains persist when the relative strength of the KL term is controlled?

  2. Closest comparison: Can you provide a matched comparison with SEED-GRPO, including the same checkpoint, reward, rollout budget, and hyperparameter-selection protocol? This would help establish the value of frozen observed-label compatibility relative to the alternative signal discussed in Table 1.

  3. Dependence on the anchor: How does performance change when corruption is present during cold-start SFT, or when the scorer has not trained on the examples it weights? Can you quantify the relationship between anchor quality and the benefit of SCS?

  4. Mechanistic evidence: In the controlled label-flip setting, what are the score distributions and resulting learning outcomes for corrupted examples versus difficult clean examples? Do these measurements support the beneficial SNR regime discussed in Appendix B, or suggest a different explanation for the gains?

2 comments
AuthorsOpenReview ↗

response to reviewer 4fqX (part 1)

Thank you for asking how much of the gain comes from matching compatibility weights to examples and whether that ranking remains useful when the anchor encounters corrupted supervision. We added controls in the Qwen3-VL-2B-Thinking Hateful Memes (HM) setup of Table 2, with 20% image swaps after cold-start SFT unless stated otherwise.

Macro-F1 is mean ± SD over three RL seeds (42/123/456); values quoted from the submitted tables, Figure 2(a), or a diagnostic batch are reported as means with SDs omitted. AUROC and AUPRC are reported separately.

Q1. Sample-specific weighting and KL. The comparisons below use the same cold-start checkpoint, verifier reward, optimizer, group size (G=8G=8), rollout budget, learning rate (3×10−63\times10^{-6}), one epoch, and βKL=0.01\beta_{\mathrm{KL}}=0.01. Each method generates its own rollouts.

MethodHM Macro-F1
Vanilla GRPO (Table 2)53.3 ± 0.7
DAPO (Table 2)59.9 ± 0.4
SEED, published linear weighting58.0 ± 0.7
SEED, validation-tuned weighting58.6 ± 0.7
SEED, weight distribution matched to SCS58.3 ± 0.6
SCS, shuffled within observed class53.7 ± 0.9
SCS-GRPO (Table 2)64.0 ± 0.2

Table A. HM with 20% image swaps. Macro-F1 is the mean ± SD across three RL seeds.

Shuffling within the observed class preserves the exact weight multiset and mean, both overall and by class. It breaks the original within-class association between weights and examples. The 10.3-point drop shows that the weight distribution alone cannot account for the gain in this setting. The permutation can still change the weighted advantages and realized updates. Since Eq. (6) weights the policy surrogate while keeping the KL coefficient fixed, a mean-matched constant would keep the average weight while removing differences between examples. Mean-normalized SCS would keep the relative weights while setting the average to one. These two controls and a matched KL sweep remain outstanding. Figure 2(b) shows advantage contraction as a descriptive check.

Q2. Comparison with SEED-GRPO. We evaluated SEED's online semantic-entropy weighting in the shared HM setup above. We used the same held-out split to select the tuned SEED variant and SCS's temperature. The tuned SEED variant reaches 58.6 and the variant matched to SCS's marginal weight distribution reaches 58.3, versus SCS at 64.0. Our contribution concerns the weight signal. SEED uses semantic dispersion across current-policy responses, and SCS uses a frozen checkpoint's compatibility with the observed label. On fixed inputs and rollouts, toggling the attached label changes mean SCS from 0.915 to 0.744, while the SEED weight remains 0.964.

AuthorsOpenReview ↗

response to reviewer 4fqX (part 2)

Continuing our response (part 2).

Macro-F1 is mean ± SD over three RL seeds (42/123/456); values quoted from the submitted tables, Figure 2(a), or a diagnostic batch are reported as means with SDs omitted. AUROC and AUPRC are reported separately.

Q3. Dependence on the anchor. We applied image swaps to 20% of HM training examples before teacher generation, filtering, and cold-start SFT. On this shared noisy anchor, SCS reaches 61.1 versus 52.4 for Vanilla GRPO (+8.7 pp). The AUROC for identifying swapped inputs is 0.771, compared with 0.842 when the anchor trains on the original pairs. Across three SFT seeds in the original setting, SCS ranks show median correlations of 0.93–0.96, with corruption AUROC of 0.832–0.847 and Macro-F1 of 63.6–64.2. A comparison that holds the RL initialization fixed and uses a scorer trained without the examples it weights remains open.

Scoring scheduleFinal swap-detection AUROCHM Macro-F1
Frozen pre-RL checkpoint0.84264.0
Refresh every 100 updates0.80963.4
Refresh every 25 updates0.75462.4
Current-policy score every update0.69161.3

Table B. Separate matched batch. Macro-F1 is the batch mean, with between-seed SDs of 0.5–0.8. For refreshed variants, we reassign the original weights according to the updated ranking and evaluate each final scorer on the same holdout.

Q4. Corruption, hard-clean examples, and SNR. In the image-swap setting, raw SCS separates swapped from original pairs with AUROC 0.842 and AUPRC 0.818, versus 0.501 and 0.202 for within-class shuffling. A control using the known corruption indicator reaches 63.3 Macro-F1, and a distribution-matched difficulty-only control reaches 57.4.

Checkpoint probabilities / SCS weightsTask ECE / BrierSwap-detection AUROC / AUPRCHM Macro-F1
Raw checkpoint / raw SCS0.108 / 0.1460.842 / 0.81864.0 ± 0.2
Temperature-scaled checkpoint / mapped SCS0.039 / 0.1210.842 / 0.81863.9 ± 0.2
Raw checkpoint / within-class shuffled SCS0.108 / 0.1460.501 / 0.20253.7 ± 0.9

Table C. Temperature scaling is fitted on clean validation for this diagnostic. Macro-F1 is the mean ± SD across three RL seeds.

In the separate counterfactual label-toggle diagnostic, we hold the input and checkpoint fixed and score the original and flipped labels. On the easier 75% of inputs by out-of-fold clean-label NLL, mean weights are 0.952 and 0.688 for the original and flipped labels. On the hardest 25%, they are 0.804 and 0.912, exposing a failure mode where the anchor can favor the wrong label. On the hard-clean evaluation quartile, Macro-F1 is 48.9 ± 0.7 for SCS, 44.3 ± 1.0 for Vanilla GRPO, and 46.8 ± 0.8 for DAPO. Score distributions and learning outcomes for these subgroups during the actual label-flip runs in Figure 2(a) remain open.

These diagnostics support a role for compatibility in detecting corruption in the image-swap setting and show where it can fail. Appendix B gives a conditional SNR result. Our scores and downstream metrics do not measure the gradient quantities in Eq. (15). The gains at zero injected flips in Figure 2(a) also leave room for effects beyond suppressing corrupted labels.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.