- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies GRPO-style RLVR fine-tuning of VLMs for binary classification when some training supervision may be corrupted. It proposes SCS-GRPO. Before RL, a frozen cold-start (SFT) checkpoint scores each training example by the probability it gives the observed label, renormalized over the class verbalizers. The weight SCS_i = q_i(ỹ_i)^{1/τ} (τ = 5) then multiplies the group-normalized advantages of that prompt's rollouts. There are two points that motivate this design:
- A prompt-constant scalar applied to rewards before group standardization cancels out (Prop. 1), so the weight is applied after normalization.
- A gradient signal-to-noise decomposition states when non-uniform weights improve SNR, with an oracle corollary In the experiments, the papers finetuned Qwen3-VL-2B/4B-Thinkin on 3 datasets: Hateful Memes (HM), CHASM and Civil Comments (CC). The results show that SCS has the best score on all metrics across 3 datasets. There are also ablations over τ and group size
Critical strengths
- The method needs one offline scoring pass plus a scalar multiplication, and it applies to any GRPO variant. It is simple and cheap. A drop-in weighting with gains of the reported size is useful.
- The paper covers 3 datasets (2 multimodal, 1 text-only), 2 model scales, and 3 seeds with standard deviations. There is consistency of gains across these settings.
Critical weaknesses
I feel that the experiments in the paper do not isolate the proposed mechanism (1) The paper has no runs with shuffled SCS weights, a constant weight, or an oracle that down-weights the truly corrupted samples. It also never checks that SCS gives corrupted samples low scores, for example with an AUROC. SCS also scales only the policy-gradient term and not the KL term. So even a constant weight changes the effective regularization. (2) The \delta ablation in Table 3 does not fit the SNR explanation. HM Macro-F1 stays between 62.7 and 64.0 from \delta=1 which down-weights strongly, to \delta=10 where the weights are almost uniform. And fully uniform weights (vanilla GRPO) score about 10 points lower (53.3). If suppressing corrupted samples drives the gain, performance should fall gradually as \delta grows, not stay flat and then drop. (3) The noise sweep (Fig. 2a) does not show robustness. All methods improve as label noise rises from 0 to 40% (for example, vanilla GRPO goes from 56.5 to 59.6). SCS's outperformance over vanilla GRPO is largest with noise (+4.3) and smaller at high noise (+2.6 to +2.8). A class-prior effect might be a plausible explanation for this as the HM training set is about 36% hateful and the dev set about 50%. No threshold-free metric such as AUROC is reported.
Questions for authors
- which configuration produced Table 2, 3, 4 and Figure 2? can you specify the sampling temperature, G, lr, \beta_KL, LoRA vs full finetuning, reward terms and weight formula with \delta.
- please show the SCS weight distribution for each dataset, split by class and by clean vs. corrupted, together with the AUROC of SCS for flagging corrupted examples.
- In Fig. 2a, why does every method improve as label noise increases, and why is SCS's margin largest at η = 0? Also, at η = 20% the DAPO (59.9) and Dr.GRPO (58.9) values equal their Table 2 modality-swap values. Are these independent runs?
3 comments
Response to reviewer EoNv (part 2)
Continuing our response (part 2).
Macro-F1 is mean ± SD over three RL seeds (42/123/456); values quoted from the submitted tables, Figure 2(a), or a diagnostic batch are reported as means with SDs omitted. AUROC and AUPRC are reported separately.
Q2. Weighting controls and corruption detection. On HM with 20% image swaps, SCS achieved Macro-F1, compared with for permuting the same weights within each observed class. This comparison preserves each class's weight distribution and tests whether assigning those weights to the original examples matters. The realized policy update and its interaction with KL can still change, and mean-matched constant weighting and mean-normalized SCS remain open controls. A two-level control using the known swap indicator achieved 63.3 Macro-F1 in the diagnostic batch; we do not treat this as a strict upper bound.
Using low SCS to identify swapped examples gave swap-detection AUROC/AUPRC of 0.842/0.818, compared with 0.501/0.202 after within-class shuffling; the injected-swap mask is used only as an evaluation label and does not enter the method. These are detection metrics for the image-swap protocol, not downstream classification metrics, and the AUPRC random baseline equals the positive-class prevalence (20% under this protocol). The requested class-stratified weight distributions, detection metrics for actual label flips, and held-out classification AUROC remain open; all three require only a single evaluation forward pass with no additional RL training, and we will add them in the camera-ready. For CHASM and Civil Comments, estimating detection performance would require independently verified corruption labels.
Table 3 reports 63.1, 63.8, 64.0, and 62.7 HM Macro-F1 for , respectively. Finite can still produce different weights when observed-label probabilities vary substantially. Unit weights recover the Vanilla GRPO objective, and 53.3 is the Table 2 Vanilla result. Appendix B gives a conditional comparison of gradient SNR and makes no prediction of a monotonic Macro-F1 curve as changes.
In a separate matched rebuttal batch we measured the endpoints outside that sweep: gives and gives , while the uniform limit is the 53.3 Vanilla mean above. Together these form an inverted U rather than a flat plateau followed by a discontinuity. Since is a monotone power transform, changing rescales the weights without reordering them; insensitivity across is therefore consistent with a mechanism that depends on the weight ordering, and the decline is graded once that ordering is flattened. The sweep alone does not isolate the mechanism — the isolation comes from the within-class shuffle above.
Response to reviewer EoNv (part 3)
Continuing our response (part 3).
Macro-F1 is mean ± SD over three RL seeds (42/123/456); values quoted from the submitted tables, Figure 2(a), or a diagnostic batch are reported as means with SDs omitted. AUROC and AUPRC are reported separately.
Q3. Label-flip sweep and repeated values. In Figure 2(a), SCS exceeds Vanilla GRPO at all five plotted rates, with the largest gap at 0% injected flips (60.8 vs. 56.5, +4.3 pp). Macro-F1 does not decrease monotonically with increasing flips. Symmetric flips move an imbalanced training label distribution toward 50/50. For example, if the original positive fraction is 36%, a 40% flip rate would move its expectation to 47.2%. The sweep alone leaves the effect of this prior shift and the mechanism behind the zero-flip gain unresolved.
Yes, the 20% label-flip results in Figure 2(a) and the 20% image-swap results in Table 2 came from independent training runs under their respective corruption protocols. The reported Macro-F1 values happen to coincide in both settings for DAPO (59.9) and Dr.GRPO (58.9). The per-seed records below show that this agreement holds only after rounding to one decimal.
Per-seed run records.
| Protocol | Method | Seed 42 | Seed 123 | Seed 456 | Mean | SD |
|---|---|---|---|---|---|---|
| 20% label flip (Fig. 2a) | DAPO | 59.41 | 60.24 | 60.13 | 59.93 | 0.45 |
| 20% image swap (Table 2) | DAPO | 60.31 | 59.55 | 59.72 | 59.86 | 0.40 |
| 20% label flip (Fig. 2a) | Dr.GRPO | 59.25 | 58.40 | 58.99 | 58.88 | 0.44 |
| 20% image swap (Table 2) | Dr.GRPO | 58.23 | 59.30 | 59.24 | 58.92 | 0.60 |
Table D. Per-seed HM Macro-F1 from RL seeds 42, 123, and 456 under each corruption protocol. Means and sample SDs (denominator n−1) use full-precision scores; values are shown to two decimals here and to one decimal in the submitted Table 2. The two rows for each method come from separate training runs under the label-flip and image-swap protocols, respectively.
For completeness, the underlying per-seed (TP, TN) counts, in seed order, are:
| Protocol | Method | Seed 42 | Seed 123 | Seed 456 |
|---|---|---|---|---|
| 20% label flip (Fig. 2a) | DAPO | (103, 202) | (105, 204) | (106, 202) |
| 20% image swap (Table 2) | DAPO | (106, 203) | (108, 196) | (108, 197) |
| 20% label flip (Fig. 2a) | Dr.GRPO | (92, 217) | (94, 209) | (95, 211) |
| 20% image swap (Table 2) | Dr.GRPO | (83, 225) | (87, 225) | (96, 211) |
Response to reviewer EoNv (part 1)
Thank you for pointing out where the configuration, weighting controls, and noise sweep need clarification.
Macro-F1 is mean ± SD over three RL seeds (42/123/456); values quoted from the submitted tables, Figure 2(a), or a diagnostic batch are reported as means with SDs omitted. AUROC and AUPRC are reported separately.
Q1. Configuration. We used full-parameter fine-tuning for cold-start SFT and RL, and a sampling temperature of 1.0 for RL rollouts. The default RL settings in Appendix A are G=8, learning rate 3×10−6, βKL=0.01, one epoch, batch size 16, and seeds 42/123/456. The verifier adds 1 for matching the observed label, 2 for valid answer format, and 0.5min(1,L/512) for response length. The SCS weight is wi=qi(y~i)1/τ, where qi is the frozen checkpoint's next-token distribution renormalized over the valid class verbalizers. The default weight temperature is τ=5. The temperature you denote δ is τ in the paper (Eq. (3), Table 3): δ=1 is the most selective setting and large δ approaches uniform weighting.
For HM, Table 2 and the HM columns of Tables 3–4 use 20% image swaps. Table 3 varies τ∈{1,3,5,10}, and Table 4 varies G∈{4,8,16}. CHASM and Civil Comments use their original annotations. Figure 2(a) varies the symmetric label-flip rate on HM from 0% to 40%, and Figure 2(b) uses the 20% image-swap setting. In the submitted HM experiments, both corruption protocols start from a checkpoint trained on the original pairs. The intervention is applied to the RL training set, with the evaluation split held fixed.