- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the role of expert representation geometry in MTRL and proposes Continuous Orthogonality Control (COC), a framework that continuously interpolates between overlapping and orthogonal expert representations. Experiments on Meta-World and MiniGrid show that the preferred degree of expert separation varies across settings, suggesting that maximal orthogonality is not always optimal. The authors further analyze expert coherence, representation structure, and expert usage, and explore scheduling and learnable separation strengths.
Critical strengths
-
MOORE employs orthogonalization to reduce redundancy and interference among experts. However, complete orthogonality may not necessarily facilitate knowledge sharing across tasks. This paper raises an important question: Should expert representations be fully orthogonal, or should some degree of overlap be preserved?
-
The paper introduces a controllable and interpretable framework for adjusting expert representation geometry.
-
The paper provides a comprehensive experimental analysis that helps explain how geometry control affects the model's internal representations and behavior, rather than merely demonstrating performance improvements. The results suggest that maximizing expert orthogonality does not necessarily yield the best multi-task learning performance, providing a valuable complement to existing orthogonal-expert methods.
Critical weaknesses
-
Insufficient explanation of why different task environments favor different degrees of expert separation. This is one of my major concerns. The paper finds that Meta-World favors intermediate separation, whereas MiniGrid favors complete orthogonality. However, these two environments differ in multiple aspects. I particularly appreciate the experimental design in Appendix A.1, but unfortunately, the results still fail to establish a clear mechanism or underlying principle that explains this phenomenon.
-
The relationship between expert orthogonality and cross-task knowledge sharing has not been directly validated. Even when all experts are completely orthogonal, different tasks can still utilize the same experts and thereby share knowledge. Conversely, highly overlapping expert representations do not necessarily imply effective knowledge transfer between tasks. The authors do not directly measure positive or negative transfer across tasks, nor do they analyze how geometry adjustment affects gradient conflicts between tasks.
-
Lack of an effective adaptive geometry selection mechanism. COC requires manually evaluating different values of to identify a favorable degree of expert separation. Although the authors investigate linear scheduling, cosine scheduling, and learnable , none of these approaches achieves performance comparable to the best fixed . This limitation reduces the practical applicability of COC.
-
Insufficient control of potential confounding factors introduced by geometry adjustment. The authors aim to control the directional relationships among different expert outputs, but the effects of other factors introduced by this adjustment have not been sufficiently ruled out. Although COC normalizes each expert output, the norm of the aggregated representation obtained through weighted expert combination may still vary with . The authors should conduct additional experiments with normalization applied to the aggregated features to verify that the observed performance differences are indeed attributable to directional changes. Alternatively, they could directly measure and report feature magnitudes to demonstrate that the aggregated feature scales remain comparable across different values of .
-
Limited methodological novelty. The proposed method is primarily an incremental extension of MOORE, introducing a relatively straightforward interpolation mechanism on top of the existing orthogonal-expert framework.
These weaknesses are listed in descending order of importance. I would be willing to raise my score if the authors can satisfactorily address these concerns.
Questions for authors
See Weakness
Dear Reviewer,
As today is the final day of the rebuttal period, we are eager not to miss this opportunity. May we kindly ask whether our responses have addressed your concerns? If so, we would be sincerely grateful if you would consider raising your score. Thank you again for your time and consideration.