acceptodds
Under review as a conference paper at ICLR 2027

CARP: Continuous Latent Routing for Mixture-of-LoRA under Non-Stationary Opponents

Abstract

Specialising LoRA adapters to non-stationary opponents is commonly done by hard-routing each forward pass through the adapter selected by a predicted cluster label. We show this interface is structurally fragile: the predicted-cluster map is non-Lipschitz, so a small error in the inferred context can flip the cluster and silently swap adapters. We propose CARP, which instead routes through a soft mixture of LoRA prototypes indexed by a continuous latent c, with weights softmax(−‖c−μ_k‖²/τ) trained by an EMA teacher–student on the realised context. The map is L_w-Lipschitz, so context error β enters the regret as O(L_w β) rather than as a 0–1 flip. We prove a two-sided guarantee: a Lipschitz upper bound for CARP and an Ω(1) per-round lower bound for any hard router under non-zero predictor noise. Empirically the separation appears exactly where the theory places it. On a controlled strategic game (30 seeds, 6 baselines) CARP beats hard routing by +2.6 LTR out-of-distribution and is 5–9× less sensitive to predictor noise; a c-conditioned linear gate recovers most of the gain, so the interface, not the kernel, carries it. On MovieLens-1M, Last.fm-1K and Amazon-Books the predicted cross-over materialises: CARP wins on drifting or noisy-boundary splits, hard routing wins on cleanly partitioned ones, and all routers tie where the context geometry is near-linear. At LM scale (Qwen2.5 0.5B–3B) hard routing on the inferred context is worse than no routing, and an oracle-context control confirms the mechanism: given the true context the hard router's penalty vanishes. The Lipschitz gain is therefore robustness to context-inference error—which also locates the LM-scale bottleneck in context inference, not routing.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
mixture-of-LoRA, continuous latent routing, non-stationary opponents, parameter-efficient fine-tuning, contextual regret, Lipschitz routing, mixture-of-experts, opponent modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bjuTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies routing in mixture-of-LoRA models under non-stationary opponents. In hard routing, offline trajectories are clustered and each forward pass uses a single adapter, making adapter selection sensitive to errors in inferred context. The proposed method, CARP, represents context with a continuous Gaussian latent variable and combines LoRA adapters using distance-based top-K softmax weights over learned prototypes. The latent representation is trained with teacher-student supervision. The paper analyzes the Lipschitz properties and regret of soft and hard routing, and evaluates CARP in a synthetic game, recommendation tasks, and small language models under different context conditions.

Critical strengths
  1. The analysis connects the instability of hard routing to its lack of Lipschitz continuity, providing a mathematical explanation for its sensitivity to context prediction errors. The theory examines both the stability of soft routing and the limitations of hard routing.

  2. The ablations include a linear-gate baseline and an oracle-context experiment. These comparisons help distinguish the effects of the routing mechanism, distance kernel, and context prediction errors. The paper also reports settings where CARP does not outperform hard routing.

Critical weaknesses
  1. The theoretical guarantees do not apply directly to the implemented method. Lemma 1 and Theorem 2 assume a full softmax router, whereas all experiments use top-K routing with K=2. Top-K truncation introduces discontinuities when prototype rankings change, violating the Lipschitz assumption used in the analysis. Consequently, the reported regret bound does not provide guarantees for the method evaluated in the experiments.

  2. The LG-c ablation weakens the case for the prototype-based distance kernel. A simple linear gate operating on the same supervised latent representation achieves nearly identical performance to CARP across most settings. The results indicate that the main improvement comes from continuous routing over a supervised latent representation rather than the specific prototype and distance-based design.

  3. The language model experiments do not demonstrate consistent improvements at the 3B scale. The reported gains are small and vary across seeds, with several runs failing. The current results provide limited evidence that CARP offers reliable benefits for language models at this scale.

Questions for authors
  1. Do the theoretical guarantees extend to top-K routing? A comparison between K=2 and full softmax routing (K=M) would also be informative.

  2. LG-c performs similarly to CARP in most settings. What does the prototype-based distance kernel contribute beyond the linear gate, and how convincing is the tau-sweep result with five seeds and R²=0.38?

  3. How is HDR's cluster predictor trained in the synthetic experiments? In particular, are the supervision signals comparable across methods?

Reviewer_ZNYXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Mixture-of-LoRA routing through a continuous latent context against non-stationary opponents, replacing hard top-k prototype routing with a Lipschitz softmax-of-distance map trained against a supervised context estimate (Eq. 3). Two-sided theory (Thm. 2 upper bound, Thm. 3 hard-routing lower bound) plus an oracle-context control. Validated on opponent-adaptation benchmarks at 0.5B-3B against HDR, a linear gate, and oracle variants.

Critical strengths
  1. The oracle-context control (Tab. 10, App. C) is the most informative experiment in the paper. With backbone, recipe and seeds held fixed, HDR routed on the true c⋆c^\star gains +2.5 to +3.3 LTR over HDR on c^\hat c at 0.5B and matches or beats CARP$^\star$ (better at p<.05 on 2/5 splits, tied on 3/5), while on c^\hat c HDR falls below the plain head on 13/15 cells of Tab. 8. The LM-scale CARP-HDR gap is therefore chiefly about sensitivity to context-inference error, and others building user- or opponent-conditioned adapters could reuse this control.

  2. The paper also reports controls that cut into its own headline. LG-c (App. I) recovers +3.01 of the +3.42 CARP-LG OOD gap on the synthetic game (Tab. 22) and ties CARP on every MovieLens split (Tab. 23). Tab. 3 keeps the ties and losses too, such as CraigslistBargain and the flat Amazon-Books split. This made it possible to audit what remains, and I read the work as an honest diagnostic study even where I disagree with the framing.

Critical weaknesses
  1. The theory is billed as a main contribution, with l. 262-263 calling it "a sharp asymptotic separation, not a constant-factor tuning question", yet the proofs do not support that. In App. M.2 the per-round routing term is LwβtL_w\beta_t, and the combined display on p. 31 rewrites it as (Lw/η)∑tβt(L_w/\eta)\sum_t\beta_t with no justification; this is the step that places Lw∑tβtL_w\sum_t\beta_t beneath the square root in Thm. 2. Carried out correctly, a constant βt=β\beta_t=\beta leaves a term LwβTL_w\beta T, linear in TT. CARP's own guarantee then scales linearly in TT, as does the hard router's lower bound in Thm. 3, so the stated results do not separate the routers by their rate in TT. CARP's upper bound drops below HDR's lower bound only when LwβL_w\beta, which grows as τ\tau shrinks (Lemma 1: Lw≤4R/τL_w\le 4R/\tau), falls under the hard router's per-round penalty. That is exactly the condition at l. 309-310 ("the right choice whenever Lwβ<1L_w\beta<1"), a constant-factor comparison. I would want Thm. 2 restated with Lw∑tβtL_w\sum_t\beta_t kept outside the square root, and the separation claim of l. 259-263 replaced by that condition. Smaller issues in M.3 and in the deployed router are in Q4.

  2. The central claim concerns the interface (hard vs. Lipschitz map on the same context estimate), but I could not find a CPU-scale comparison documented to hold the predictor and its supervision fixed (App. L says its Last.fm HDR uses the "same c", without stating whether that is CARP's c⋆c^\star-supervised c^\hat c). Tab. 13 labels its HDR row "τ→0\tau\to0", yet App. E says those rows re-tabulate the Tab. 1 baselines. Beyond the generic nearest-centroid rule of Sec 3, synthetic HDR is given only as "m=5 k-means regimes with routed GRPO" (Sec 6), with nothing on what z^\hat z is predicted from or whether it sees c⋆c^\star (App. A.3 says HDR is given the true style id at training time in Mini-NLHE). CARP is trained against c⋆c^\star via Eq. 3, and LG-c, which retains that supervision, likewise beats HDR by +2.19 OOD (Tab. 22), so the +2.60 in Tab. 1 cannot separate the hard-vs-Lipschitz interface from supervised continuous cc. The 5-9x noise result (Sec 6.2) replaces HDR's label with a uniform one but adds N(0,pσc)N(0,p\sigma_c) to CARP's c^\hat c, which are different perturbations. The main same-c^\hat c comparison (Tab. 8) pairs m=5 offline k-means heads against M=8 learned prototypes: CARP wins 5/5 splits at 0.5B (4 at p<.05) but only 2/5 at 3B at the best-in-dist checkpoint used for Tab. 8 (none at p<.05; 5/5, 3 at p<.05, at the final checkpoint, l. 920-921), and the MovieLens-LM probe (App. C) ties on all 8 splits.

Questions for authors
  1. (Clarification) In Tab. 1 and Tab. 2, does synthetic HDR route on CARP's c^\hat c or on its own z^\hat z at train and test time, and is that predictor trained against c⋆c^\star?

  2. (Suggestion) Is Tab. 13's "τ→0\tau\to0, HDR" row CARP with argmax over the same M=8 prototypes, predictor and Eq. 3 supervision, or the Tab. 1 HDR baseline? If the latter, could you add such a CARP-top1 arm together with a noise sweep applying the same Gaussian perturbation to c^\hat c for both routers? A CI excluding 0 in CARP's favour there would raise my score.

  3. (Clarification) Alg. 1 line 4 forms the action log-prob with μS\mu_S.detach(), whereas Sec 4.3 says the update routes on the teacher's ct⋆c^\star_t. Which one was used for the reported runs?

  4. (Clarification) Thm. 3 as stated gives only β/(4δ)\beta/(4\delta) per round, with Ω(1)\Omega(1) reached only at β=δ\beta=\delta, while the M.3 construction with ϵ→0\epsilon\to0 yields a β\beta-independent constant. The construction also assumes fully specialised prototype policies (l. 1661-1663), whereas Thm. 3 and l. 308-309 assert the penalty for any hard router and any per-prototype policy; if adjacent prototypes share a policy, a crossing costs nothing. Which statement and policy assumption are intended? Relatedly, U$[-\beta,\beta]$ in M.3 has E∣ζ∣=β/2E|\zeta|=\beta/2, not β\beta, and the deployed top-2 router (Sec 4.2) is not Lipschitz, since its weights jump where the 2nd and 3rd prototypes swap rank while Lemma 1 covers only the full softmax. Which guarantee applies to the reported runs?

Reviewer_uUNwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes CARP, a method that combines multiple LoRA adapters using a continuous estimate of an opponent's behavior.

Critical strengths

None

Critical weaknesses

The main weakness of this paper is its presentation. Both abstract and introduction jump directly to technical details and results without giving enough context for the problem, motivation or clearly explaining the proposed approach. Many sentences are packed with notation, numerical results, references and multiple claims, while failing to explain what they mean or why they matter. Notation is not introduces clearly enough and mathematical details are used in the prose instead of explanations and analysis. The sentence structure is often formulaic and very compressed, making it difficult to follow the narrative and understand the meaning. The general flow of the paper is broken: many paragraphs read as stand-alone pieces and do not connect with the surrounding text. It makes it difficult to understand the motivation for this paper and its contribution. Unfortunately, the current presentation does not allow me to properly evaluate this submission.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.