- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies routing in mixture-of-LoRA models under non-stationary opponents. In hard routing, offline trajectories are clustered and each forward pass uses a single adapter, making adapter selection sensitive to errors in inferred context. The proposed method, CARP, represents context with a continuous Gaussian latent variable and combines LoRA adapters using distance-based top-K softmax weights over learned prototypes. The latent representation is trained with teacher-student supervision. The paper analyzes the Lipschitz properties and regret of soft and hard routing, and evaluates CARP in a synthetic game, recommendation tasks, and small language models under different context conditions.
Critical strengths
-
The analysis connects the instability of hard routing to its lack of Lipschitz continuity, providing a mathematical explanation for its sensitivity to context prediction errors. The theory examines both the stability of soft routing and the limitations of hard routing.
-
The ablations include a linear-gate baseline and an oracle-context experiment. These comparisons help distinguish the effects of the routing mechanism, distance kernel, and context prediction errors. The paper also reports settings where CARP does not outperform hard routing.
Critical weaknesses
-
The theoretical guarantees do not apply directly to the implemented method. Lemma 1 and Theorem 2 assume a full softmax router, whereas all experiments use top-K routing with K=2. Top-K truncation introduces discontinuities when prototype rankings change, violating the Lipschitz assumption used in the analysis. Consequently, the reported regret bound does not provide guarantees for the method evaluated in the experiments.
-
The LG-c ablation weakens the case for the prototype-based distance kernel. A simple linear gate operating on the same supervised latent representation achieves nearly identical performance to CARP across most settings. The results indicate that the main improvement comes from continuous routing over a supervised latent representation rather than the specific prototype and distance-based design.
-
The language model experiments do not demonstrate consistent improvements at the 3B scale. The reported gains are small and vary across seeds, with several runs failing. The current results provide limited evidence that CARP offers reliable benefits for language models at this scale.
Questions for authors
-
Do the theoretical guarantees extend to top-K routing? A comparison between K=2 and full softmax routing (K=M) would also be informative.
-
LG-c performs similarly to CARP in most settings. What does the prototype-based distance kernel contribute beyond the linear gate, and how convincing is the tau-sweep result with five seeds and R²=0.38?
-
How is HDR's cluster predictor trained in the synthetic experiments? In particular, are the supervision signals comparable across methods?