acceptodds
Withdrawn from ICLR 2027

GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The Group Relative Policy Optimization (GRPO), a reinforcement learning method used to fine-tune large language models (LLMs), has proved its effectiveness in practical applications such as DeepSeek-R1. It raises a question whether GRPO can be generalized to representation learning models. In this paper, we propose Group Relative Policy Optimization for Representation Model (GRPO-RM), and investigate the performance of GRPO-like policy in post-training representation models. Specifically, our method treats categories as a predefined output set to functionally replace token sequence sampling in LLMs, thereby generating an output group, which is essential for the probability-driven optimization of GRPO. In addition, a specialized reward function is designed to accommodate the properties of representation models. Extensive experiments are conducted on various real-world datasets to validate the effectiveness of our proposed method.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Learning, Fine-tuning, Reinforcement Learning, Group Relative Policy Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gJEFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adapts GRPO to post-train visual representation models via accuracy and uniformity rewards, evaluated on image classification and semantic segmentation.

Critical strengths

Clear motivation, extensive experiments across eight datasets, and a reasonable engineering adaptation of GRPO to deterministic representation models.

Critical weaknesses
  1. Missing non-RL baselines (critical). The paper only compares against standard fine-tuning. It lacks comparisons with label smoothing, entropy/uniform regularization, focal loss, and class-weighted CE—methods that are functionally close to the proposed uniformity reward. Without these, it is impossible to tell whether GRPO adds value beyond a well-designed loss regularizer.

  2. Limited necessity of the RL formulation. For fixed-class classification/segmentation, there is no genuine rollout diversity; the group-relative advantage largely reduces to a reweighted loss. The paper does not justify why this RL adaptation is preferable to a simple regularized objective.

  3. Statistical significance. Standard deviations are reported, but no significance tests are performed. Several gains (e.g., segmentation mIoU +0.2–0.7) may not be statistically meaningful.

  4. Overstated claims. The “first RL post-training for representation models” claim needs precise scoping, and the conclusion that GRPO is an effective paradigm is not fully supported without the above controls.

Note on fair comparison: The concern about asymmetric evaluation appears partly addressed: Table 1 reports both SR and kNN for fine-tuning, implying the same frozen re-probing protocol. However, this should be stated explicitly in the main text to avoid confusion.

Questions for authors

I encourage the authors to further clarify the above concerns and discuss how they plan to address these limitations in future revisions.

Reviewer_DbeCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper adapts a GRPO-style objective to supervised fine-tuning of visual representation models. For each input, the method uses all classes as a fixed group of candidate outputs. It computes rewards from the ground-truth label and the old model’s class probabilities. It then updates the model using group-normalized advantages, probability ratios, and a clipped objective. The authors evaluate the method on image classification and semantic segmentation. Compared with cross-entropy (CE) fine-tuning, the method improves classification accuracy and segmentation metrics in several settings.

Critical strengths

The classification results show useful gains across different ways of evaluating the learned features. On Tiny-ImageNet, GRPO-RM improves Softmax Regression (SR) accuracy from 58.67% to 65.96% and kNN accuracy from 58.59% to 66.36% (Table 1). After fine-tuning, the authors freeze the backbone and train a new classification head. They also report kNN results. These evaluations support an improvement in the features used for classification, beyond changes to the original classification head. The semantic segmentation experiments also broaden the evaluation.

Critical weaknesses

1. The motivation for the GRPO-style structure is not sufficiently clear.

In this setting, all candidate classes can be enumerated, and the accuracy rewards follow directly from the ground-truth labels. The paper needs to explain what specific problems the current-to-old probability ratio and clipping address in this supervised task. It also needs to explain why these choices are more suitable than simpler supervised objectives. Adopting the form of the GRPO objective does not, by itself, establish the value of these components.

Section 3.3 motivates the reward design through representation alignment and uniformity. However, the uniformity reward acts on output class probabilities. The paper does not directly analyze how this probability adjustment improves the structure of the feature space. The experiments show that the method can work, but the motivation for its main design choices remains largely based on intuition.

2. The experiments do not sufficiently identify the source of the gains or establish the need for the key components.

The paper includes reward and KL-term ablations. However, it lacks separate ablations that remove clipping or the probability ratio. It also lacks comparisons with relevant simple supervised objectives, such as CE with label smoothing. The current results do not separate the effects of reward design, gradient allocation across classes, and GRPO-style update control.

The reward ablation also gives mixed results. Adding the uniformity reward raises SR accuracy from 62.16% to 65.96% on Tiny-ImageNet, but lowers it from 77.04% to 76.21% on ImageNet (Table 6). This does not rule out the value of the reward. It does show that the conditions under which it helps need further explanation.

In addition, Figures 3 and 4 compare loss curves from different training objectives. An earlier plateau alone does not establish that classification performance converges faster. The shorter training runs in Table 9 provide more direct evidence of training efficiency. However, they still do not isolate the contribution of the probability ratio or clipping.

Questions for authors
  1. When all classes can be enumerated and accuracy rewards are determined by the labels, what specific problem does each of the probability ratio and clipping address? Please clarify how the objective relates to CE and supervised objectives with class-dependent weights. A gradient comparison would help clarify the contribution.

  2. With the rewards and other training settings fixed, how do performance and stability change when clipping and the probability ratio are removed separately? With an equal hyperparameter tuning budget for each method, does the full method still outperform CE and CE with label smoothing? These comparisons would directly affect my assessment of the need for the proposed components.

  3. Is there direct evidence that the uniformity reward improves the structure of the feature space, and that this improvement relates to downstream performance? How do the authors explain the negative effect of this reward on ImageNet?

  4. The paper defines the training objective and provides empirical results, but does not give a theoretical analysis of its advantages over CE. Under what assumptions could this objective offer an advantage in optimization or representation learning? An analysis in a simplified setting would help explain when such benefits should be expected.

Reviewer_35dvOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes GRPO-RM, a GRPO-inspired objective for supervised fine-tuning of visual representation models. It treats the complete set of task categories as a fixed output group, uses a task head to obtain class probabilities, and constructs rewards by combining a ground-truth correctness reward with a negative-probability penalty termed a uniformity reward. These rewards are normalized across categories to obtain advantages, which are used in a clipped probability-ratio objective. The method maintains an old model updated each epoch and removes the reference-model KL term.

The main evaluation uses DINOv2 ViT-S/14 on five classification and three semantic segmentation datasets. Compared with cross-entropy fine-tuning, GRPO-RM reports larger improvements in classification and smaller, consistent full-data mIoU improvements in segmentation. The appendices include reward and KL ablations, ResNet50 experiments, and comparisons showing that 50 epochs of GRPO-RM can outperform 100 epochs of standard fine-tuning on two classification datasets.

Critical strengths
  1. A concrete and implementable adaptation of GRPO-style optimization to non-generative visual models. Using the category set as the output group provides a straightforward way to apply relative advantages and clipped updates without requiring autoregressive generation or a learned reward model. The proposal is technically accessible and applicable to both image-level and pixel-level prediction. This makes it a useful empirical investigation of whether optimization techniques developed for language-model post-training can benefit supervised visual representation learning.

  2. Meaningful empirical gains, with additional experiments addressing important scope and efficiency concerns. On DINOv2, the classification improvements include 5.54 percentage points on CIFAR-100 and 7.29 points on Tiny-ImageNet. Evaluation with frozen post-training features and kNN provides evidence beyond the performance of the jointly trained task head. The current version also includes KL ablations, a non-transformer backbone, and shorter-training comparisons. In particular, the 50-versus-100-epoch results provide more convincing evidence of practical efficiency than comparisons between differently defined training losses alone. These results justify taking the proposed objective seriously, even though its benefits are not equally strong across settings.

Critical weaknesses
  1. The experiments do not isolate the contribution of GRPO-specific optimization from other changes to the supervised objective. The method uses ground-truth labels and enumerates all categories, so the central scientific question is whether group-relative normalization, the old-policy ratio, and clipping improve learning beyond a suitably designed supervised loss. The main comparison against standard cross-entropy does not resolve this. The reward ablation varies the uniformity component within GRPO-RM, but does not compare against competitive supervised regularization baselines or isolate the effects of clipping and advantage normalization. Consequently, the experiments establish that the complete training recipe can outperform the chosen baseline, but not that the GRPO mechanism is responsible. This attribution gap materially weakens the paper’s central contribution.

  2. The mechanistic and generality claims exceed the supporting evidence. The negative-probability reward operates on class predictions, but the paper does not establish that it promotes feature-space uniformity or explain its relationship to established supervised objectives. Its benefit is also conditional: on ImageNet, accuracy-only rewards achieve 77.04% SR accuracy versus 76.21% for the complete method. The added ResNet50 experiments are welcome, but SR improvements over standard fine-tuning are at most 0.32 percentage points, with a decrease on CIFAR-100. These findings support a promising method under the tested DINOv2 settings more strongly than a broadly effective representation-learning paradigm. A sharper analysis of the objective and more calibrated claims are needed to explain when and why it helps.

Questions for authors
  1. Assessment-critical experimental request — contribution of the optimization mechanism: Can you provide controlled comparisons that remove clipping and group-wise advantage normalization separately, together with a tuned cross-entropy baseline using label smoothing or a clearly defined entropy-based regularizer? These comparisons would materially affect my assessment of whether the gains arise from GRPO-specific optimization or from a different form of supervised regularization. Simply averaging the proposed −pi-p_i term over all classes would yield a constant, so any proposed “CE + uniformity” baseline needs an explicit, nonconstant formulation.

  2. Clarification — implementation and interpretation of the KL ablation: Appendix C.2 describes a KL constraint between representations rather than task-output distributions. How are the representations converted into distributions, what KL direction and coefficient are used, and how was that coefficient selected? Also, Table 7 shows that adding KL improves ImageNet kNN accuracy but reduces SR accuracy from 76.21% to 74.68%. Please clarify the metric-specific conclusion. These details determine how strongly the ablation supports removing KL.

  3. Clarification — evaluation protocol and baseline reliability: Can you clarify the classification input preprocessing, frozen-feature head architecture, and hyperparameter selection procedure for both methods? In particular, why does STL-10 exhibit such a large discrepancy between SR and kNN accuracy—for example, 46.42% versus 89.03% for GRPO-RM? Understanding this discrepancy is important for determining whether the reported SR gains reflect improved representations or limitations of the evaluation-head training protocol.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.