acceptodds
Under review as a conference paper at ICLR 2027

Continuous Expert Representation Geometry for Multi-Task Reinforcement Learning

Abstract

Multi-task reinforcement learning (MTRL) requires representations that are shared enough to support transfer yet specialized enough to limit interference. However, expert-based policies typically fix the degree of representational separation through architectural or optimization choices, leaving a basic question unresolved: how much separation should a multi-task learner impose? We study expert specialization as a continuous representation-geometry problem and introduce Continuous Orthogonality Control (COC), which exposes a controllable sharing–specialization spectrum while holding the expert networks, task conditioning, aggregation mechanism, optimization objective, and model capacity fixed. COC interpolates between an overlap-permitting expert frame and an orthogonal boundary, while realized geometry is measured independently through expert coherence. Across the evaluated settings, the favorable geometry is not universal. On Meta-World, both MT10 and MT50 achieve their highest observed performance within a broad intermediate-separation region across five seeds, whereas the maximally separated boundary achieves the highest observed return among the evaluated MiniGrid settings. Increasing the control coordinate systematically reduces expert coherence, confirming that COC produces the intended geometric intervention, while auxiliary response-structure and expert-usage diagnostics reveal setting-dependent internal responses. On Meta-World, however, performance varies non-monotonically with realized coherence, indicating that coherence is a geometry diagnostic rather than a universal monotonic criterion for selecting the favorable regime. Finally, on MT10, neither the tested open-loop schedules nor end-to-end optimization of learnable separation strengths matches the best fixed regime; the learned active coefficients instead move toward the maximally separated boundary. These results separate two problems that are often conflated: controlling representation geometry and identifying which geometry is favorable for a given multi-task learning setting.

Primary area:
reinforcement learning
Keywords:
Multi-Task Reinforcement Learning, Mixture of Experts, Representation Geometry, Expert Specialization, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_n1JZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the role of expert representation geometry in MTRL and proposes Continuous Orthogonality Control (COC), a framework that continuously interpolates between overlapping and orthogonal expert representations. Experiments on Meta-World and MiniGrid show that the preferred degree of expert separation varies across settings, suggesting that maximal orthogonality is not always optimal. The authors further analyze expert coherence, representation structure, and expert usage, and explore scheduling and learnable separation strengths.

Critical strengths
  1. MOORE employs orthogonalization to reduce redundancy and interference among experts. However, complete orthogonality may not necessarily facilitate knowledge sharing across tasks. This paper raises an important question: Should expert representations be fully orthogonal, or should some degree of overlap be preserved?

  2. The paper introduces a controllable and interpretable framework for adjusting expert representation geometry.

  3. The paper provides a comprehensive experimental analysis that helps explain how geometry control affects the model's internal representations and behavior, rather than merely demonstrating performance improvements. The results suggest that maximizing expert orthogonality does not necessarily yield the best multi-task learning performance, providing a valuable complement to existing orthogonal-expert methods.

Critical weaknesses
  1. Insufficient explanation of why different task environments favor different degrees of expert separation. This is one of my major concerns. The paper finds that Meta-World favors intermediate separation, whereas MiniGrid favors complete orthogonality. However, these two environments differ in multiple aspects. I particularly appreciate the experimental design in Appendix A.1, but unfortunately, the results still fail to establish a clear mechanism or underlying principle that explains this phenomenon.

  2. The relationship between expert orthogonality and cross-task knowledge sharing has not been directly validated. Even when all experts are completely orthogonal, different tasks can still utilize the same experts and thereby share knowledge. Conversely, highly overlapping expert representations do not necessarily imply effective knowledge transfer between tasks. The authors do not directly measure positive or negative transfer across tasks, nor do they analyze how geometry adjustment affects gradient conflicts between tasks.

  3. Lack of an effective adaptive geometry selection mechanism. COC requires manually evaluating different values of α\alpha to identify a favorable degree of expert separation. Although the authors investigate linear scheduling, cosine scheduling, and learnable α\alpha, none of these approaches achieves performance comparable to the best fixed α\alpha. This limitation reduces the practical applicability of COC.

  4. Insufficient control of potential confounding factors introduced by geometry adjustment. The authors aim to control the directional relationships among different expert outputs, but the effects of other factors introduced by this adjustment have not been sufficiently ruled out. Although COC normalizes each expert output, the norm of the aggregated representation obtained through weighted expert combination may still vary with α\alpha. The authors should conduct additional experiments with normalization applied to the aggregated features to verify that the observed performance differences are indeed attributable to directional changes. Alternatively, they could directly measure and report feature magnitudes to demonstrate that the aggregated feature scales remain comparable across different values of α\alpha.

  5. Limited methodological novelty. The proposed method is primarily an incremental extension of MOORE, introducing a relatively straightforward interpolation mechanism on top of the existing orthogonal-expert framework.

These weaknesses are listed in descending order of importance. I would be willing to raise my score if the authors can satisfactorily address these concerns.

Questions for authors

See Weakness

Reviewer_jPsPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies expert representation separation in multi-task reinforcement learning, it proposes continuous orthogonality control (COC), which adjusts the separation by interpolating between normalized expert outputs and their orthogonalized counterparts. Intermediate separation performs best on Meta-World MT10 and MT50, whereas full orthogonality performs best among the tested MiniGrid settings. The paper also examines representation geometry and tests scheduled and learnable separation strengths.

Critical strengths
  1. The method is simple and addresses a useful question. Varying separation within the same expert architecture makes it easier to study its effect without changing model capacity or the training objective.
  2. Experiments cover MT10, MT50, and MiniGrid, with five seeds for the main comparisons. On MT10, an intermediate setting reaches 0.898 success, compared with 0.860 and 0.848 at the two endpoints. The different trends across benchmarks provide useful evidence that full orthogonality is not always the best choice.
  3. The analysis goes beyond average performance. Task-level results, coherence measurements, and scheduled and learnable variants provide a fuller picture of the method. The limitations section also acknowledges that the proposed explanations for the observed trends require further validation.
Critical weaknesses
  1. The findings are interesting, but my main concern is that the paper does not provide a clear explanation for the different effects of expert separation across settings. Intermediate separation works better on Meta-World, while full orthogonality performs best among the tested MiniGrid settings. The paper discusses possible explanations, but the analyses do not show which task or representation properties account for this difference. When and why a particular separation level helps are partly answered.
  2. The role of aggregate feature scale remains unclear. Section 3.3 acknowledges that changing α may also change the norm of the aggregated representation. The performance differences may therefore reflect changes in both expert directions and feature scale. A comparison that controls the aggregate feature norm would help clarify whether the advantage of intermediate separation persists.
Questions for authors
  1. What task or representation properties explain the different separation preferences on Meta-World and MiniGrid? Further analysis linking these properties to task-level performance would help support the proposed explanations.
  2. Does intermediate separation still perform better when the aggregate feature norm is controlled? This comparison would help distinguish the effects of directional separation from changes in feature scale.
  3. Why do the learnable separation strengths approach full orthogonality on MT10, despite the better performance of intermediate fixed settings?
  4. The paper could be better organized. Some points about what α and the diagnostic measures mean, and what they cannot tell us, are repeated. Consolidating these points where appropriate and giving more attention to the underlying mechanisms may help make the discussion clearer and easier to follow.
1 comment

Dear Reviewer,

As today is the final day of the rebuttal period, we are eager not to miss this opportunity. May we kindly ask whether our responses have addressed your concerns? If so, we would be sincerely grateful if you would consider raising your score. Thank you again for your time and consideration.

Reviewer_t2buOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks how separated the experts in a multi-task RL policy should be. MOORE makes the expert outputs exactly perpendicular at every state, and the authors add a knob that blends each normalized expert output with its orthogonalized version, so one end keeps the learned directions and the other end is MOORE. They find middle settings work best on Meta-World with SAC and full orthogonalization works best on MiniGrid with PPO. On MT10, neither ramping the knob during training nor learning it matches the best fixed setting.

Critical strengths
  • It's a fair question whether full orthogonalization is really needed, and I like that only the knob changes while everything else stays fixed.
  • The authors are upfront about negative or inconclusive results, like the task-structure probe, the variance penalty and the learned knob.
  • The per-task breakdown is useful, since it shows the overall curve is driven by a few sensitive tasks.
Critical weaknesses
  • The method is a small change to MOORE. Every knob setting spans the same per-state subspace, and the appendix shows the learned experts are usually already close to their orthogonalized versions, so the knob mostly turns them a little. A soft orthogonality penalty with a tunable weight is the obvious alternative and should be compared, even though it changes the loss.
  • I'm not convinced the sweep shows that the best geometry differs across benchmarks. MT50 has a clean plateau above both ends, but on MT10 one task is never learned and most others are at or near full success everywhere, so the gap comes from partial failures on a few tasks in some seeds, with overlapping error bars. On MiniGrid only three values are tested, and the boundary's lead over the middle is within the spread.
  • The baseline table overstates the gain. It shows only the best knob value, picked on the same final success it reports, against external numbers listed at five times the steps. The authors' own boundary runs are left out, even though they already beat every listed method on MT50 and match or beat all but one on MT10. The MOORE row also looks lower than what the MOORE paper reports, by enough to change the picture on MT50.
  • The schedule and learned-knob runs say little about whether learning can find a middle setting. Both schedules end at full orthogonalization, so the policy is evaluated at the boundary after being shifted in steps during off-policy training. The learned knob is trained on the policy's own loss, so there's no clear reason it would find the setting with the best final success, and its learning rate and the regularizer's form and weight aren't given.
  • Some diagnostics don't measure what they're used for. Gate-response agreement and expert switching pick the dominant expert by raw output norm, which the layer divides out before gating, so it doesn't directly affect the policy. Coherence is presented as an independent measure of the realized geometry, but it's computed after the blend and is zero at full orthogonalization by construction, so its drop mostly reflects the layer itself.
  • The effective-rank analysis is shaky. The main-text rank uses random Gaussian inputs, and the one real-state check in the appendix gives quite different values. In the synthetic test, squeezing the experts into a low-dimensional subspace gives a rank above the subspace dimension after orthogonalization, which can't happen if the outputs stay in it. I suspect Gram-Schmidt is amplifying noise from near-zero residuals. And on MT50, the boundary's rank after the layer stays far above its raw rank, so the raw drop mostly doesn't reach the policy.

Related Works that are relevant and missing The related work misses several close papers that should be in the background/literature review:

  • D'Eramo et al., Sharing Knowledge in Multi-Task Deep Reinforcement Learning, ICLR 2020. They study when sharing a representation across tasks helps in multi-task RL, which is the sharing side of the trade-off studied here.
  • Arnob et al., Efficient Reinforcement Learning by Discovering Neural Pathways, NeurIPS 2024. They find sparse task-specific pathways inside one shared network, including on multi-task Meta-World, which is a different way of deciding how much to share.
  • Bansal et al., Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?, NeurIPS 2018. They impose orthogonality as a soft penalty with a tunable weight rather than a hard constraint (on network weights), which is the natural alternative to the hard blend here.
  • Roy and Vetterli, The effective rank: A measure of effective dimensionality, EUSIPCO 2007, and Bardes et al., VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning, ICLR 2022. The first introduces the effective rank this paper adapts and shows it can't exceed the true rank, which matters for the synthetic test. The second pairs its variance term with a decorrelation term because variance alone doesn't stop features from becoming redundant copies of each other, which matches the authors' finding that their variance penalty was met by rescaling without adding new directions.
Questions for authors
  • What's the overlap between the raw expert outputs, before the blend, for models trained at different knob values?
  • How is the synthetic rank test set up, and why does the rank exceed the injected rank?
  • Does door-open really fail at the boundary? The heatmap and the boundary's mean success suggest it fails only in some seeds.
  • Where do the baseline numbers come from, especially the MOORE row?

Minor

  • Several axis labels are garbled in the task-sensitivity and basis-comparison figures.
  • The rank values in the text don't match the plotted points.
  • The MT50 sweep numbers are only in a plot, and the per-task MiniGrid win/loss counts mentioned in the setup are missing.
  • The variance-penalty test is run on MT10, where the boundary's rank wasn't low to begin with, and its formula isn't given.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.