acceptodds
Withdrawn from ICLR 2027

One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Although Reinforcement Learning with Verifiable Rewards (RLVR) provides a principled post-training paradigm for cultivating complex reasoning abilities of Large Language Models (LLMs), jointly training one foundation model across heterogeneous domains usually introduces multi-domain trade-offs and destabilizes the overall policy optimization. While existing approaches typically diagnose cross-domain interference via gradient alignment or curvature proxies, they usually overlook interference that accumulates across consecutive updates—where seemingly non-conflicting gradients can still overwrite prior policy gains. We show that these cross-step dynamics are directly tractable between adjacent checkpoints through token log-probability footprints, bypassing costly curvature approximations. Building on this observation, we propose OSOL, which organizes each mixed-domain update around a single focus domain and uses the preceding checkpoint footprint to identify rebound-prone tokens and apply a rank-based, adaptively scaled correction. Our analysis shows that this correction contracts the targeted negative-to-positive backtracking component, while controlled studies show that cross-step backtracking is more strongly associated with subsequent task damage than within-step gradient diagnostics and that checkpoint footprints identify future rebound more effectively than Hessian-based proxies. Across Qwen3-8B-Base and Qwen3-30B-A3B, OSOL achieves the best four-domain macro average on both backbones, while ranking first or second in most evaluations. On Qwen3-30B-A3B, it reaches a domain-macro average of 0.4822, outperforming the strongest baseline by 5.7%. Ablations support prioritizing one domain per step, using checkpoint drift to locate rebound-prone tokens, and correcting them at the token level. Code is available at https://anonymous.4open.science/r/anonymous-osol-83DF/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning, Multi-Domain Reasoning, Multi-Task Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rjx5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OSOL, which selects a focus domain during each mixed-domain update and adjusts GRPO coefficients based on the token log-probability drift between consecutive checkpoints. The experiments span four domains and two backbones, and the authors further investigate cross-step backtracking through theoretical analysis and controlled experiments.

Critical strengths
  1. The method constructs control signals using output changes from existing checkpoints, eliminating the need for explicit Hessian computation.
  2. Matched ablation experiments, which maintain a consistent focus structure, demonstrate that history residuals provide additional benefits beyond domain weighting.
  3. Diagnostic experiments employing disjoint probes, bootstrap intervals, and leave-one-out analysis enhance the credibility of the correlation results.
Critical weaknesses
  1. The main results lack multiple training seeds and uncertainty estimates; notably, the macro-average for the 8B model exceeds the strongest baseline by only 0.0035.
  2. Study I defines task damage as the increase in NLL on fixed responses, without verifying whether this change corresponds to an actual decline in accuracy or reward.
  3. The backtracking diagnostic utilizes information from after the second update, whereas gradient diagnostics use pre-update information; consequently, the higher correlation observed does not directly demonstrate superior predictive capability.
  4. The ablation regarding equal task weights does not directly compare single-focus versus multi-focus correction, and is thus insufficient to establish the necessity of selecting only one focus domain per step.
Questions for authors

NaN

Reviewer_o5ZqOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies cross-domain interference in multi-domain RLVR, arguing that interference can emerge across consecutive optimization steps even when within-step gradient diagnostics show little conflict. It proposes OSOL, which uses token-level log probability changes between adjacent checkpoints to identify rebound prone tokens and applies a history-conditioned correction within GRPO; experiments on two Qwen3 backbones report improvements in domain-macro performance across math, code, IF, and chat.

Critical strengths
  1. OSOL does not introduce a separate auxiliary objective; it modifies the token-level coefficient supplied to the existing clipped policy objective, which makes the method relatively simple to implement conceptually.
  2. The implementation and experimental setup are documented relatively thoroughly.
  3. The paper evaluated models of different scales.
Critical weaknesses
  1. The domain classification is overly coarse and insufficiently justified. Prior work has systematically investigated the limited cross-domain generalizability of existing RLVR methods, adopting a substantially more fine-grained and comprehensive domain taxonomy than this paper: for example, Cheng et al. [1] evaluate six reasoning domains, including mathematics, coding, science, logic, simulation, and tabular reasoning, where the lack of generalizability is particularly severe in domains that are not included in this paper such as scientific reasoning and tabular tasks.

  2. Following 1, because of the unreasonable domain selection the results in Tables 1 and 2 become not convincing at all. For Table 1, your method outperforms baseline on only math and code and not on chat and IF, and on Table 2 only on math. Given that prior studies have already identified some degree of transferability within structured reasoning domains such as mathematics and coding [1, 2], these results make it difficult to determine whether the observed gains arise from the proposed method or primarily reflect domain-specific characteristics.

  3. Even within the selected evaluation domains, the reported improvements are relatively small: less than 3 percentage points on average in Table 1 and less than 1 percentage point in Table 2. Given the limited consistency of these improvements across domains, I find the empirical evidence insufficient to demonstrate a substantial methodological contribution.

  4. The method is evaluated on the Qwen family only. Also the baseline RLVR methods are not sufficiently tested.

  5. The equation piling feels unmotivated.

[1] Cheng, Jorge Zhoujun, et al. "Revisiting reinforcement learning for llm reasoning from a cross-domain perspective." Advances in Neural Information Processing Systems 38 (2025).

[2] Hu, Chuxuan, et al. "Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

Can you explain how you developed the 4 domains you selected? IF and chat seem to be substantially easier than math and coding tasks.

Reviewer_cf2oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies interference between consecutive updates during multi-domain reinforcement learning for LLMs. It proposes OSOL, which selects a focus domain at each step and uses token log-probability changes from the previous checkpoint to apply a rank-based correction within GRPO. Experiments on Qwen3-8B-Base and Qwen3-30B-A3B show the best four-domain macro average among the compared methods. Controlled studies suggest that cross-step backtracking is more informative of later task damage than the tested gradient diagnostics.

Critical strengths
  1. The paper identifies a plausible limitation of within-step gradient-conflict measures and proposes a direct signal based on changes between adjacent checkpoints.
  2. OSOL improves the domain-macro average on both tested backbones. On Qwen3-30B-A3B, the reported gain over the strongest baseline is 5.7%, with a notable improvement in code generation.
Critical weaknesses
  1. The main results are reported for only two backbones without uncertainty across independent full training runs. On Qwen3-8B-Base, OSOL scores 0.3735 versus 0.3700 for the strongest baseline, a small difference that may not be robust. On the 30B model, the aggregate gain is not uniform across domains.
  2. The mechanism evidence is limited. The checkpoint-footprint ranking reaches ROC-AUC 0.567 in an eight-seed study, only modestly above chance. In addition, the theoretical contraction guarantee relies on a local sign-transfer condition; the paper does not show how often this condition holds during the main training runs.
Questions for authors
  1. Were the main results reproduced with multiple independent training seeds? Please report mean and variance, especially for the small gain on Qwen3-8B-Base.
  2. How often does the sign-transfer condition required by the theoretical guarantee hold during actual OSOL training? Can the authors report this directly?
  3. Given the ROC-AUC of 0.567, how much of the final performance gain can be attributed to the checkpoint-history correction rather than the focus-domain weighting? A matched ablation on both backbones would help clarify this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.