- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Compatibility-guided Group Reweighting (CoGR), a gradient-compatibility-based approach for multi-environment reinforcement learning with a shared language model. The method extends Group Relative Policy Optimization (GRPO) by considering gradient relationships among experience groups collected from different environments. Specifically, CoGR computes pairwise gradient similarities between groups and assigns bounded, positive weights according to their compatibility scores, while maintaining fixed sampling quotas and the total nominal policy-gradient weight for each environment. The objective is to reduce harmful gradient interference and improve joint learning across heterogeneous environments. The method is evaluated using a 4B-parameter language model with rank-8 LoRA on three environments: Search, TextCraft, and Code. The experimental results show that CoGR improves the environment-averaged success rate over standard GRPO by 2.39 percentage points across two training seeds after 64 policy updates, with the largest improvement of 7.64 percentage points on Code. In a single-seed comparison, CoGR also achieves competitive final performance against MT-GRPO and GradAlign-cosine while requiring fewer generated rollouts, although it incurs additional gradient-computation overhead.
Critical strengths
-
The paper investigates an important problem: training a single language model to handle heterogeneous tasks and environments through reinforcement learning. The proposed group-level coordination provides a potentially useful approach to improving joint optimization without modifying the predefined sampling allocation across environments.
-
The experimental results demonstrate improvements over standard GRPO, particularly on the challenging Code environment. These findings suggest that exploiting gradient relationships among experience groups can be beneficial for multi-environment training with a shared model.
Critical weaknesses
-
The novelty of the proposed method is not sufficiently clear. Existing conflict-aware multi-task optimization methods, such as MGDA, PCGrad, and CAGrad, also exploit gradient relationships to mitigate interference during joint optimization. CoGR appears to adopt a similar principle, with the main distinction being that gradient compatibility is evaluated at the experience-group level rather than the task level. However, it is unclear whether this change in optimization granularity constitutes a substantial methodological contribution. The paper would benefit from a clearer discussion of its fundamental differences from existing gradient-conflict-based approaches, ideally supported by comparisons with their group-level adaptations.
-
Although the proposed compatibility scores and bounded group weights are intuitively reasonable, their specific formulations appear heuristic. In particular, the additional conflict penalty in Eq. (5) and the score normalization and weight assignment in Eqs. (7)–(8) lack a sufficiently principled justification. It remains unclear why these particular designs should be preferred over alternative weighting strategies or optimization-based gradient aggregation methods. A more systematic explanation or additional comparisons would strengthen the methodological contribution.
-
The proposed method requires computing individual experience-group gradients and their pairwise similarities, potentially introducing substantial computational and memory overhead as the number of groups and trainable parameters increases. Although the experiments employ a 4B-parameter model, only rank-8 LoRA parameters are optimized. Moreover, the reported GPU-hour consumption of CoGR is approximately 1.75 times that of standard GRPO. Consequently, its practical scalability to larger trainable parameter spaces, higher LoRA ranks, or full-parameter fine-tuning remains insufficiently investigated.
Questions for authors
- Could existing gradient-conflict-aware methods, such as PCGrad or MGDA, be directly adapted to the group-level setting? What are the key advantages of CoGR over these methods?
- Could the authors further discuss the computational scalability of CoGR with increasing model size or LoRA rank, and possible ways to reduce its overhead?
- How does CoGR compare with independently trained task-specific models? Does joint training provide benefits over independent optimization?
- Why are the improvements mainly concentrated in Code, while the gains on Search and TextCraft are limited? Is this related to task characteristics or training randomness, or the selection of the seed?