acceptodds
Under review as a conference paper at ICLR 2027

TARS: Extending Reinforcement Learning Gains to the Long Tail under Limited Data for Domain-Specific Visual Recognition

Abstract

Visual recognition in domain-specific applications, such as biodiversity research and agriculture, often suffers from limited and imbalanced training data. Large Vision-Language Models (LVLMs) offer a promising starting point with rich visual and semantic knowledge acquired through web-scale pretraining, but naive supervised fine-tuning (SFT) on such data can cause them to overfit to head classes. Although reinforcement learning (RL) is reported to generalize better than SFT, our analysis shows that its gains under Group Relative Policy Optimization (GRPO) do not consistently extend to tail classes. To address this limitation, we propose **Tailness-aware Advantage ReScaling (TARS)**, a novel RL-based method that amplifies GRPO advantages for classes that are under-represented and poorly recognized. This amplification is guided by *tailness*, defined as the product of class rarity in the training data and the per-class error rate of the SFT model. Experiments on four domain-specific classification and detection benchmarks show that TARS consistently improves tail-class performance over SFT and vanilla GRPO, while outperforming existing imbalance-aware GRPO variants in both tail-class and overall performance. TARS also provides further gains when applied on top of imbalance-aware SFT methods, demonstrating that the two approaches are complementary.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Domain-specific visual recognition, data imbalance, limited data, reinforcement learning, group relative policy optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_85EbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper defines tailness as the product of class rarity and the per-class error rate of the SFT model. And the proposed TARS designs a weighted GRPO objective based on class tailness.

Critical strengths
  1. The paper is well-written and easy to follow.
  2. The proposed tailness and TARS are simple yet effective. Across different benchmarks and tasks, TARS outperforms other RL based variants.
Critical weaknesses

1.Class-difficulty based reweighting is not fundamentally new, [1] and Difficulty-net also uses reweighting technology based on class-level difficulty. Difficulty-net is cited in this paper, however the discussion does not fully explain how its class-difficulty reweighting differs conceptually.

[1]Sinha, Saptarshi, Hiroki Ohashi, and Katsuyuki Nakamura. "Class-difficulty based methods for long-tailed visual recognition." International Journal of Computer Vision 130.10 (2022): 2517-2531.

2.The paper evaluates per-class error rate using training set performance using initial SFT model, which might be noisy for tail classes. A class with only a handful of examples could receive a very different tailness weight depending on one or two predictions.

Questions for authors

Please address the critical weaknesses.

Reviewer_7McjOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on the task of long tail image classification and detection. It hypothesises that GRPO training fails to account for data imbalance in training data and some classes classification being harder than others degrading/not improving rare classes accuracy. To overcome this, the paper proposes Tailness-aware Advantage ReScaling (TARS), which rescales advantage values during GRPO training based on tailness of the query class. Tailness is computed using as a product of class rarity (computed using number of samples for a class in training data) and error rate of the SFT model. TARS shows improvement across three dataset for image classification and one dataset for object detection.

Critical strengths
  1. The finding in fig1 that vanilla GRPO accuracy drops for “Few” frequency class is interesting and motivates the problem clearly
  2. The paper is clearly written and easy to follow.
Critical weaknesses
  1. Limited justification for the core assumption: One of the key assumptions of the method is that GRPO does not account for how well the model recognizes a class. However, this is only partially true. While GRPO does not explicitly account for class-wise accuracy, the base model's recognition capability is implicitly reflected in the rollout rewards and relative advantage computation.

  2. Insufficient theoretical motivation: Rescaling advantages using class-dependent weights appears to be a heuristic rather than a theoretically grounded approach. It would be helpful to provide a stronger theoretical motivation explaining how advantage scaling addresses the identified limitations. Moreover, if a rare class produces no useful GRPO advantage signal (e.g., all rollouts receive identical rewards), simply multiplying its advantages will not improve the learning signal.

  3. Missing baselines: Some simple but relevant baselines for GRPO training are missing, including (a) hard-negative data mining for GRPO training (e.g., DiVE-k) and (b) sampling strategies applied during GRPO training rather than only during SFT.

  4. Limited dataset diversity: Three classification datasets (two of which are subsets of iNaturalist) and one detection dataset are insufficient to establish the general effectiveness of the proposed method. The authors should consider evaluating on additional datasets, such as ImageNet-LT, CIFAR-100-LT, and ODinW, to strengthen their claims.

  5. Limited evaluation across model backbones and scales: All experiments use the relatively older Qwen2.5-VL-3B-Instruct backbone. Results on more recent models (e.g., the Qwen3-VL series) and larger models (e.g., 7B-scale) would help establish whether the improvements generalize to newer and more capable VLMs.

Questions for authors
  1. Why is no format reward used during GRPO training?

  2. In Table 1, what is the motivation for comparing different loss functions during the SFT stage when the main innovation of the paper lies in the GRPO training phase?

  3. Insufficient analysis of static vs. dynamic tailness: Why is the ablation comparing static and dynamic tailness moved to the appendix? This is an important ablation and should be included in the main paper. Additionally, while the results show that static tailness performs better than dynamic tailness, no explanation or insight is provided to understand this observation. Furthermore, why is a 0.98% improvement in training time considered relevant to present?

Reviewer_UhQYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TARS, a reinforcement learning-based method that improves model performance through effective data optimization under limited-data and long-tailed settings. Industrial datasets, in particular, often contain categories with limited or very few samples, and the paper demonstrate that focusing training on the categories that constitute performance bottlenecks can improve overall recognition performance. The experimental results also show that, while the baseline GRPO occasionally suffers from performance degradation depending on the task, the proposed method consistently achieves performance improvements across the evaluated settings.

Critical strengths
  • The paper proposes TARS, a GRPO-based method designed for visual recognition with limited datasets. The approach appears to work effectively by measuring tailness and applying class-wise normalization for optimization, resulting in meaningful improvements in visual recognition performance. The experimental results also demonstrate the effectiveness of the proposed method compared with the original GRPO.

  • Limited-data and long-tailed distributions are common challenges in industrial datasets. From an application perspective, the reviewer believes the proposed method offers a promising solution to these problems. The reviewer also expects that further methodological improvements could broaden its applicability to a wider range of tasks and practical scenarios.

Critical weaknesses
  • Isn't GRPO primarily designed for generative models, including language, image, and video generation? Although the reviewer understands that it can be applied as an optimization method, the reviewer would like to better understand why GRPO is particularly necessary for the combination of TARS, limited-data/long-tailed settings, and fine-grained visual recognition. The proposed method can be viewed as an extension of GRPO, but measuring tailness and applying normalization appear to be relatively general optimization strategies. If the authors intend to demonstrate that this approach is particularly effective for addressing the challenges of limited-data and long-tailed fine-grained visual recognition, the reviewer believes it is important to select appropriate comparison methods that are specifically designed for these problems.

  • The proposed method fine-tunes Qwen to address long-tailed distributions, but does this introduce any concerns regarding catastrophic forgetting after training? Since Qwen is a general-purpose model, the reviewer would expect that specializing it for fine-grained visual recognition might lead to some degradation in its recognition or explanation capabilities for other objects and domains. Have the authors investigated this issue or provided any qualitative examples or quantitative evaluations to demonstrate whether the model retains its general-purpose capabilities after fine-tuning?

Questions for authors
  • Is it possible to address the points raised in the 'Critical Weaknesses' section in the revised paper? Even without conducting extensive additional experiments, the reviewer believes the readability and overall quality of the paper could be improved by reframing the motivation in the introduction, clarifying the positioning of the proposed method in relation to related work, and providing stronger explanations of why the methodological choices are necessary. Regarding the potential forgetting issue in Qwen, it would also be helpful to provide some examples demonstrating that the model retains a certain degree of general-purpose capability after fine-tuning.

  • Even if TARS is considered an improvement over GRPO, the present study focuses specifically on fine-grained visual recognition, and its demonstrated applicability remains relatively limited. Could the paper explains more clearly how the proposed method makes a sufficiently meaningful contribution to the specific combination of TARS, limited-data/long-tailed settings, and fine-grained visual recognition? The reviewer believes a stronger explanation of this relationship would help clarify the significance and academic contribution of the proposed method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.