acceptodds
Withdrawn from ICLR 2027

Beyond Task Balancing: Coordinating Experience Groups in Multi-Environment Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-environment reinforcement learning (RL) trains language agents to retrieve information, use tools, and solve problems within a shared policy. Besides deciding how much experience each environment contributes, joint training must decide how this experience combines into each update. Group Relative Policy Optimization (GRPO) compares trajectories within each problem’s group but averages groups without considering whether their policy gradients reinforce or oppose one another. We introduce Compatibility-guided Group Reweighting (CoGR), which coordinates experience groups while keeping each environment’s rollout quota fixed. Within-group relative rewards decide which trajectories to reinforce, and between-group gradient compatibility decides how strongly each group contributes. CoGR scores each complete group in the current on-policy batch by its environment-balanced gradient agreement with other groups and adds an extra penalty for strongly opposing gradients. Bounded positive weights with unit mean per environment rescale only the policy-gradient loss, preserving within-group comparisons and reusing the collected trajectories. We train a 4B-parameter agent on Search (Natural Questions and HotpotQA), TextCraft, and a restricted CodeContests subset. Across two training seeds, CoGR improves environment-averaged success on the development sets by 2.39 percentage points (pp) over GRPO after 64 policy updates, including a 7.64 pp gain in Code, the weakest environment. In a single-seed comparison at the same endpoint, CoGR exceeds agent-adapted GradAlign-cosine by 5.33 pp and achieves slightly higher environment-averaged success than agent-adapted MT-GRPO with 82.51% fewer generated rollouts. These results suggest that experience-group coordination is a useful way to improve joint learning within prescribed environment allocations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-task Reinforcement Learning, Group Relative Policy Optimization, Language Model Agents, Policy Gradient, Agentic Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_m8MsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Compatibility-guided Group Reweighting (CoGR), a gradient-compatibility-based approach for multi-environment reinforcement learning with a shared language model. The method extends Group Relative Policy Optimization (GRPO) by considering gradient relationships among experience groups collected from different environments. Specifically, CoGR computes pairwise gradient similarities between groups and assigns bounded, positive weights according to their compatibility scores, while maintaining fixed sampling quotas and the total nominal policy-gradient weight for each environment. The objective is to reduce harmful gradient interference and improve joint learning across heterogeneous environments. The method is evaluated using a 4B-parameter language model with rank-8 LoRA on three environments: Search, TextCraft, and Code. The experimental results show that CoGR improves the environment-averaged success rate over standard GRPO by 2.39 percentage points across two training seeds after 64 policy updates, with the largest improvement of 7.64 percentage points on Code. In a single-seed comparison, CoGR also achieves competitive final performance against MT-GRPO and GradAlign-cosine while requiring fewer generated rollouts, although it incurs additional gradient-computation overhead.

Critical strengths
  1. The paper investigates an important problem: training a single language model to handle heterogeneous tasks and environments through reinforcement learning. The proposed group-level coordination provides a potentially useful approach to improving joint optimization without modifying the predefined sampling allocation across environments.

  2. The experimental results demonstrate improvements over standard GRPO, particularly on the challenging Code environment. These findings suggest that exploiting gradient relationships among experience groups can be beneficial for multi-environment training with a shared model.

Critical weaknesses
  1. The novelty of the proposed method is not sufficiently clear. Existing conflict-aware multi-task optimization methods, such as MGDA, PCGrad, and CAGrad, also exploit gradient relationships to mitigate interference during joint optimization. CoGR appears to adopt a similar principle, with the main distinction being that gradient compatibility is evaluated at the experience-group level rather than the task level. However, it is unclear whether this change in optimization granularity constitutes a substantial methodological contribution. The paper would benefit from a clearer discussion of its fundamental differences from existing gradient-conflict-based approaches, ideally supported by comparisons with their group-level adaptations.

  2. Although the proposed compatibility scores and bounded group weights are intuitively reasonable, their specific formulations appear heuristic. In particular, the additional conflict penalty in Eq. (5) and the score normalization and weight assignment in Eqs. (7)–(8) lack a sufficiently principled justification. It remains unclear why these particular designs should be preferred over alternative weighting strategies or optimization-based gradient aggregation methods. A more systematic explanation or additional comparisons would strengthen the methodological contribution.

  3. The proposed method requires computing individual experience-group gradients and their pairwise similarities, potentially introducing substantial computational and memory overhead as the number of groups and trainable parameters increases. Although the experiments employ a 4B-parameter model, only rank-8 LoRA parameters are optimized. Moreover, the reported GPU-hour consumption of CoGR is approximately 1.75 times that of standard GRPO. Consequently, its practical scalability to larger trainable parameter spaces, higher LoRA ranks, or full-parameter fine-tuning remains insufficiently investigated.

Questions for authors
  1. Could existing gradient-conflict-aware methods, such as PCGrad or MGDA, be directly adapted to the group-level setting? What are the key advantages of CoGR over these methods?
  2. Could the authors further discuss the computational scalability of CoGR with increasing model size or LoRA rank, and possible ways to reduce its overhead?
  3. How does CoGR compare with independently trained task-specific models? Does joint training provide benefits over independent optimization?
  4. Why are the improvements mainly concentrated in Code, while the gains on Search and TextCraft are limited? Is this related to task characteristics or training randomness, or the selection of the seed?
Reviewer_Hoz4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose Compatibility guided Group Reweighting (CoGR), which measures compatibility using environment-balanced cosine similarities between the policy gradients of problem groups in the current batch and applies an additional penalty to strongly opposing gradients. The method reweights each group’s policy gradient loss using bounded positive weights with unit mean within each environment, while preserving GRPO’s within group relative comparisons. Its main contribution lies in distinguishing group level update coordination under fixed quotas from task-level allocation and providing a corresponding implementation.

Critical strengths

This paper distinguishes between "how much experience to sample from each environment" and "how the collected experience is combined to inform policy updates." Its contribution lies in the insight that even when the allocation of experience across environments is fixed, different subsets of tasks within the same environment may still require distinct update weights. This introduces a more granular control dimension to multi-environment reinforcement learning, demonstrating the value of such group-level coordination.

Critical weaknesses
  1. The primary performance gains in the three-environment experimental setup are concentrated in the "Code" domain, while "TextCraft" shows a decline relative to the baseline; the average gains across the two environment settings are relatively modest. Consequently these results reflect optimization benefits only for a specific combination of tasks, offering limited evidence for the general applicability of the method.
  2. Capturing gradients for the entire group and performing compatibility calculations incur significant overhead; under comparable conditions, GPU resource consumption is approximately 1.75 times that of GRPO.
Questions for authors

1.The paper does not show the distribution of conflicts during training or their relationship with update effectiveness; please provide gradient cosine similarity distributions and the proportion of strongly conflicting pairs for each environment pair, along with an analysis of actual policy update effectiveness with and without reweighting. 2.The existing Shuffled and Mean ablations can test weight assignment and the additional penalty, but do not establish whether the gains arise from cross-environment coordination; I would appreciate ablations using only within-environment or cross-environment references, or comparisons against simple reweighting methods based on success rates or learning potential. 3. Please provide matched-checkpoint comparisons of the full method, CoGR-Mean, and CoGR-Shuffled for the second three-environment training seed, and report the per seed performance differences. 4.The late-stage performance decline of the adapted GradAlign lacks diagnostic analysis. Please report the number of selected problems from each environment and the proportion of groups with nonzero policy gradients in each selection round, together with an analysis of the stability of the reference gradient or candidate rankings, and examine whether these factors are associated with the performance decline. 5.A comparison between CoGR-Mean and a representative task level gradient coordination method is missing; please include such a comparison under fixed environment quotas and the same training protocol, and analyze whether the performance differences arise from the change in coordination granularity. 6.The exact sample lists retrieval backend configuration, and some decoding records remain incomplete; please provide the data identifiers and filtering scripts, retrieval service and corpus versions, gradient capture and adapted baseline code, and raw evaluation records, and clarify the unrecorded top-k setting used in the two-environment evaluation.

Reviewer_wFrdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Compatibility-guided Group Reweighting (CoGR) for multi-environment GRPO. It scores complete rollout groups using environment-balanced pairwise gradient cosines, with an additional penalty for strongly opposing gradients. Bounded positive weights have unit mean within each environment and rescale only the policy-gradient loss, leaving rollout quotas and within-group comparisons unchanged.

Critical strengths
  • The fixed-quota formulation separates aggregation from data allocation. Reweighting complete groups preserves GRPO’s within-group comparisons, while the bounded, unit-mean construction keeps each environment’s nominal loss weight fixed. This provides an interpretable way to study group coordination without changing rollout allocation.

  • The matched comparisons provide evidence that group assignment matters. CoGR improves mean success over GRPO in both three-environment seeds.

Critical weaknesses
  • The experiments do not distinguish cross-environment coordination from within-environment reweighting. Equation (6) combines agreement with groups from the same environment and from other environments. The shuffled-weight control establishes that group assignment matters, but not that cross-environment information is useful. Likewise, the comparison with GradAlign changes the rollout allocation, scoring frequency, and trajectory reuse simultaneously (Appendix B.3).

  • The performance gain comes at a substantial, unmatched computational cost. CoGR uses 1.75 times the GPU-hours of GRPO under the same timing scope (Appendix B.9), while the main comparison fixes optimizer updates rather than compute.

Questions for authors
  • Evaluation independence. Were the Search and TextCraft problems in the main evaluation also used in the development sweep in Appendix B.10? Which configuration and checkpoint choices used these results, and is there an evaluation on problems excluded from those choices?

  • Reference construction. With the multiplier rule and rollout quotas unchanged, how does CoGR compare with scoring against only the candidate’s own environment, or with averaging all valid reference problems without environment balancing? These controls would isolate the proposed reference design.

  • Conflict penalty. How often do pairwise cosines fall below the margin of −0.05, and how much does the penalty change the resulting weights? Does its advantage over CoGR-Mean at 64 updates reproduce in the second main training seed?

  • Compute–performance trade-off. Appendix B.9 reports 1.75 times the GPU-hours of GRPO under the same timing scope. Does the performance advantage remain when GRPO receives the same total compute budget? The reported rollout savings are useful, but do not answer whether the additional gradient computation is preferable to further GRPO training.

Reviewer_SVQeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In multi-env GRPO training, the paper keeps the per-environment rollout quota fixed and reweights individual groups (one problem's G rollouts) by how well their policy gradient agrees with other groups. For each group it computes the PG gradient, takes pairwise cosines with groups from other problems (with extra penalty for strongly negative cosines), averages per problem, then per env, then centers and rescales into weights in [0.75, 1.25] that sum to the quota within each env. Only the PG loss is reweighted. Experiments use Qwen3-4B-Instruct with LoRA on Search (NQ/HotpotQA), TextCraft and a CodeContests subset. Over 2 seeds CoGR beats GRPO by 2.39 pp on mean dev success after 64 updates, mostly from Code (+7.64 pp), and in a single-seed run it beats GradAlign-cosine and roughly ties MT-GRPO with far fewer generated rollouts.

Critical strengths
  1. The question is reasonable and cleanly separated from task balancing: within a fixed environment allocation, can you combine groups better? The design choices (unit mean per env, bounded positive weights, reweight only the PG term) are careful and make CoGR easy to stack on top of an existing balancer.
  2. It needs no extra rollouts, which is a real advantage over MT-GRPO's oversampling, and the authors are upfront about the real cost (about 1.75x GPU-hours from computing per-group gradients).
Critical weaknesses
  1. The evidence is too small to support the claims. The dev sets have 48 / 85 / 18 problems, and the 7.64 pp Code gain (which drives most of the average) is on 18 problems with 4 attempts each, so roughly one or two problems' worth of difference. There are 2 seeds vs GRPO, and all external comparisons and ablations are single-seed. At 32 updates CoGR is actually behind MT-GRPO and GradAlign, and the ordering flips at 64. β was swept on the dev set that's also used for reporting. In the 2-env setting the gain is +0.58 pp with one seed negative. I honestly can't tell from this whether CoGR helps or whether this is noise plus one lucky endpoint.
  2. The comparison doesn't control for compute. CoGR costs ~1.75x the GPU-hours of GRPO, but the comparison is at equal updates. A fair question is what GRPO does with that extra compute (more rollouts per group, a bigger batch, more updates). There's also no comparison to the obvious multi-task gradient methods (PCGrad, CAGrad, etc.) applied at the group level, even though the related work section says the motivation is the same. The paper is also very hard to read. Almost every sentence is a percentage, and the main idea gets buried.
Questions for authors
  • Can you report results on a held-out test set, separate from the dev set used to pick β, with more seeds (ideally 3–5) and CIs?
  • How does GRPO do with compute matched to CoGR (same GPU-hours)?
  • What happens with a group-level PCGrad/CAGrad style projection instead of reweighting? That seems like the most direct competitor.
  • How many Code problems actually flip between CoGR and GRPO?
  • Does the Code gain persist past 64 updates, or do the methods converge?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.