acceptodds
Under review as a conference paper at ICLR 2027

Mixture of Latent Recursions for Low-Rank Adaptation

Abstract

Low-Rank Adaptation (LoRA) performs task-specific adaptation using a fixed low-rank architecture, whose expressiveness is typically increased by enlarging or adding learnable modules at a proportional parameter overhead. We introduce Mixture of Latent Recursions (MoLR), an adaptive recursion framework that enhances adaptation expressiveness while preserving parameter efficiency. MoLR recursively refines the down-projected latent representation of each input using a shared transformation. A lightweight threshold-based router independently determines whether each token should undergo further refinement, while dynamically adjusted thresholds encourage balanced utilization across recursion depths. Extensive experiments across diverse language and vision tasks demonstrate that MoLR consistently outperforms representative LoRA variants with negligible parameter overhead over LoRA. Further analyses show that MoLR learns input-dependent recursion-depth allocation and enables a controllable inference-time compute–performance trade-off without requiring retraining.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
PEFT, LoRA

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_D6yVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Mixture of Latent Recursions (MoLR), which adds adaptive recursion to the latent space of LoRA. A shared matrix Wlatent∈Rr×rW_{\text{latent}} \in \mathbb{R}^{r \times r} and a fixed sinusoidal encoding refine the down-projected latent state over multiple steps. A threshold router decides the recursion depth per token, and an auxiliary-loss-free update keeps the depth usage balanced. Because the transformation is shared, the parameter count does not grow with the maximum depth TT. Experiments on language understanding, language reasoning, and image classification show that MoLR outperforms representative LoRA variants at a similar parameter budget.

Critical strengths

Placing adaptive recursion inside the LoRA bottleneck is a novel idea. The threshold update separates load balancing from task learning, which is a clean design. The evaluation covers 21 datasets on six backbones across three task types, with seed variance reported for all results. Eq. 6 gives a compact polynomial view of the recursion, making the method easy to follow. The practical gains are also notable: MoLR at r=8r=8 beats LoRA at r=32r=32, and adding recursion improves three other LoRA variants.

Critical weaknesses

W1. The drop at large TT is not explained. In Fig. 2b, accuracy at r=8r=8 falls to about 57% at T=8T=8, below LoRA at 59.21%. The paper attributes this to "interference" but gives no evidence. Eq. 6 contains WlatentdW_{\text{latent}}^d, and no norm constraint is applied, so the latent state may explode or vanish. This weakens the claim that deeper recursion can scale LoRA.

W2. The ablation lacks a depth-matched baseline. In Tab. 6a, "Fixed" runs one step, while MoLR runs up to two. Without a fixed 2-step baseline, the gain cannot be attributed to routing. The gain also comes mainly from one task: HumanEval improves by 2.44 points with a std of 1.83, while the other six tasks improve by only 0.47 points on average.

W3. Parameter sharing shows no clear benefit. In Tab. 6c, the step-specific variant scores 61.39%, slightly above MoLR at 61.21%. At T=2T=2, it adds only 128 parameters per module.

W4. Inference cost is high. In Tab. 5, per-token latency is 0.69 ms, about 4× that of LoRA at 0.18 ms. This overhead is large relative to the accuracy gain.

W5. The motivation is weak. LoRA assumes that weight updates have low intrinsic rank, and most follow-up work shares this view. The paper does not explain why richer latent computation is needed under this premise. Its own results also show limited headroom: in App. Tab. 11, raising the LoRA rank from 8 to 32 improves the NLR average by only 1.1 points.

Questions for authors

Q1. Could the authors report the spectral norm of WlatentW_{\text{latent}} and the norm of ztz_t at each step for r=8r=8 and T∈2,4,6,8T \in {2,4,6,8}? Does spectral normalization or an orthogonality constraint remove the drop at large TT?

Q2. Could the authors report a fixed 2-step baseline without routing on all seven NLR tasks? Does MoLR still outperform it, with and without HumanEval?

Q3. Could the authors run the step-specific variant at T=4T=4 and T=6T=6? Does parameter sharing help at larger TT?

5 comments

We thank the reviewer for the thoughtful feedback.


W1: Interference. By interference, we refer to potential competition between recursion steps for a shared latent transformation, as each step operates on a different latent-state distribution. The improvement from step-specific transforms over the parameter-sharing variant in Tab. 6c is consistent with this interpretation.

W1 & Q1: Spectral Norm of Wlatent\boldsymbol{W}_{\text{latent}}. The table below shows the [min, max] of spectral norms of Wlatent\boldsymbol{W}_{\text{latent}} across adapted modules in each selected layer. We observe no systematic change as TT increases from 2 to 8. This suggests that the performance drop is unlikely to be driven by growing weight spectral norms.

TTLayer 1Layer 8Layer 15Layer 21Layer 28
2[0.86, 1.11][0.89, 1.11][0.90, 1.08][0.95, 1.09][0.88, 1.12]
4[0.86, 1.11][0.88, 1.10][0.89, 1.10][0.96, 1.09][0.88, 1.11]
6[0.84, 1.16][0.92, 1.11][0.81, 1.00][0.89, 1.08][0.89, 1.14]
8[0.86, 1.10][0.88, 1.10][0.88, 1.06][0.94, 1.08][0.85, 1.06]

W1 & Q1: Norm of zt\boldsymbol{z}_t. The table below shows the [min, max] of the norm ratio ∥zt∥2∥z0∥2\frac{\lVert z_t \rVert_2}{\lVert z_0 \rVert_2} across adapted modules in each selected layer. We observe norm expansion of zt\boldsymbol{z}_t as TT increases.

TTLayer 1Layer 8Layer 15Layer 21Layer 28
2[0.14, 11.55][0.43, 3.28][0.35, 3.76][0.23, 3.59][0.71, 2.88]
4[0.24, 14.51][0.16, 8.94][0.27, 5.79][0.39, 7.25][0.50, 10.41]
6[0.32, 33.68][0.13, 8.77][0.18, 17.68][0.26, 18.76][0.32, 25.18]
8[0.23, 56.46][0.19, 30.54][0.19, 26.38][0.25, 32.33][0.36, 103.34]

Following the ablation study setup in Sec. 4.5, we investigate whether the performance drop at T=8T=8 can be addressed by mitigating this norm expansion via:

  • spectral normalization of Wlatent\boldsymbol{W}_{\text{latent}}
  • scaled recursion: zt+1=zt+stTWlatent(zt+ϕ(t))\boldsymbol{z}_{t+1} = \boldsymbol{z}_t + \frac{s_t}{T}\boldsymbol{W}_{\text{latent}}\left(\boldsymbol{z}_t + \phi(t)\right)

Scaled recursion mitigates the performance degradation at large TT, although it yields no clear performance gain compared with T=2T=2. We leave determining when deeper recursion is necessary and how to exploit it effectively as valuable directions for future work.

MethodARC-CBoolQGSM8KHellaSwagHumanEvalMMLUWinoGrandeAvg.
none (current)51.3466.3238.1179.3424.3955.1083.6456.89
spectral normalization51.3163.8437.4579.2822.5654.8883.1656.07
scaled recursion54.3289.0840.8478.9724.5955.7384.1361.14

W2 & Q2. Tab. 6a compares different latent recursion designs when all inputs receive the same expected number of recursion steps. Following the ablation study setup in Sec. 4.5, we compare MoLR with the required fixed 2-step baseline. The results below show that MoLR outperforms this baseline with and without HumanEval.

MethodARC-CBoolQGSM8KHellaSwagHumanEvalMMLUWinoGrandeAvg.Avg. (w/o HumanEval)
Fixed 2-step50.1188.3134.2279.0724.5955.3883.7459.3565.14
Input-dependent (current)54.6989.5240.1879.1125.6155.4883.9061.2167.15

W3 & Q3. Parameter sharing allows:

  • the trainable parameter overhead is independent of the maximum recursion depth TT
  • the shared latent transformation Wlatent\boldsymbol{W}_{\text{latent}} is supervised across all executed recursions.

In contrast, for the step-specific variant, each step-specific residual ΔWt\Delta \boldsymbol{W}_{t} can only receive supervision from tokens reaching that step. Later step-specific residuals thus receive fewer training signals, potentially hindering convergence, particularly when fine-tuning data and updates are limited. We extend the ablation of step conditioning to T=4T=4 and T=6T=6 following the experimental setup in Sec. 4.5. These results favor parameter sharing at larger TT, consistent with the potential benefit of training the shared transformation using supervision from all recursion steps.

TTConditioningARC-CBoolQGSM8KHellaSwagHumanEvalMMLUWinoGrandeAvg.
4step-specific51.0288.1137.8879.3424.5955.8484.0060.11
sinusoidal (current)54.4989.3840.5179.2525.2055.7183.7461.18
6step-specific52.1987.9238.9279.2423.9855.7084.0060.28
sinusoidal (current)53.6788.4238.5279.2424.8055.3683.9860.57

W4. In Tab. 5, we profile methods without runtime optimization to enable controlled comparisons and minimize optimization-induced confounding effects. To better reflect practical deployment, we additionally measure per-token latency (ms) with commonly used inference optimizations, including KV caching, compilation, and CUDA Graphs. With these general-purpose optimizations alone, MoLR already approaches the latency of merged LoRA, supporting its practical deployment. Specialized kernels could further reduce the remaining overhead. We leave their development to future work.

MethodEagerOptimizedAvg.
LoRA (merged)0.180.0759.21
MoLR0.690.0961.21

W5. The low-rank premise motivates a compact adaptation space but does not imply that LoRA’s direct mapping from latent representations to output fully exploits its potential. Consequently, improving adaptation performance and narrowing the gap with full-parameter fine-tuning under a limited parameter budget remains an active and central research direction (L86). MoLR pursues this objective by implementing adaptive latent recursion, enriching latent computation without sacrificing parameter efficiency. The performance gains brought by increasing LoRA rank are task- and backbone-dependent.

Dear Reviewer D6yV,

As the discussion phase draws to a close, we would like to sincerely thank you for your time and thoughtful feedback. We hope that our rebuttal and additional experimental results have adequately addressed your concerns and provided further clarity on our contributions.

If you find that your previous concerns have been sufficiently resolved, we would be grateful if you could kindly consider reflecting these clarifications in your final evaluation. Of course, we would also be happy to address any remaining questions.

Thank you again for your valuable time and consideration.

Best regards, The Authors

Reviewer_N8WbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MoLR increases computation within a low-rank adapter by repeatedly applying a shared latent transformation. Starting from z0=Axz_0=Ax, a shared router scales each executed update and decides whether recursion continues. Fixed step encodings distinguish stages, and adjusted thresholds encourage balanced stopping depths. The additional trainable parameters are independent of the maximum depth. Experiments cover six language and vision backbones, higher-rank alternatives, depth policies, compatibility with other adapters, and execution costs.

Critical strengths

The complete adapter has substantial positive comparative evidence. Tables 2–4 report the highest average on all six backbones, exceeding the strongest competing averages by 0.77–1.97 points. Llama achieves 61.21 with 12.17M trainable parameters versus TopLoRA's 60.44 with 18.12M. Table 11 places rank-8 MoLR at the quality level of rank-32 TopLoRA and MoSLoRA. DTD gains over TopLoRA reach 8.71 and 5.18 points on the two vision backbones. These results give the method a credible practical advantage when adapter parameter count is a priority.

The paper includes a relevant control for adaptive recursion. In Table 6(a), input-dependent depth scores 61.21, fixed one-step gated processing 60.46, and random depth 59.53. The fixed control already includes state-dependent gating, and input-dependent depth improves all seven task scores. Table 8 also shows improvements when the module is added to VeRA, LoRA-FA, and AuroRA, supporting its compatibility with different adapter constructions.

Critical weaknesses

The architectural development beyond existing conditional recurrence is limited. MoSLoRA [1] already learns a latent mixer, and TopLoRA [2] establishes input-conditioned adaptation without increasing rank. Mixture-of-Recursions [3] already supplies sequential score re-evaluation, gated shared updates, and permanent exit in its expert-choice variant; its token-choice variant makes a causal one-shot decision. MoLR combines causal per-token decisions with sequential re-evaluation by replacing sequence-relative top-k selection with local thresholds and sharing a small latent router. This usefully gives a frozen backbone conditional adapter depth without recurrent-backbone pretraining. However, the change preserves the same state-score-update-exit mechanism, with balancing transferred to local threshold feedback. The parameter saving follows from the smaller shared state, and the polynomial interpretation follows by expansion. The contribution is a compact realization of conditional computation in PEFT; the demonstrated development of that mechanism is modest.

The experiments establish an effective configuration more strongly than a substantive advance in how latent computation should be allocated. The direct adaptive increment is about 0.75 points on Llama; sequential thresholds improve over the tested routing alternatives by 0.26–0.36 points. These are positive results, while the token visualization does not establish which inputs benefit from refinement. Increasing the training horizon hurts low-rank performance in Figure 2(b); Figure 3(c) shows useful depth-cap adjustment within a trained horizon. These observations delimit the configuration but provide little explanation of the underlying capacity–depth relationship. The practical gain also has a material cost: Table 5 reports 0.69 ms/token versus TopLoRA's 0.44 and merged LoRA's 0.18, with similar inference memory. MoLR is faster than AuroRA and has modest training overhead, but its advantage is primarily in task parameters and quality.

References

[1] Taiqiang Wu, Jiahao Wang, Zhe Zhao, and Ngai Wong, "Mixture-of-Subspaces in Low-Rank Adaptation," Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024. Paper

[2] Shiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang, Ziqiang Cui, Dugang Liu, Yuhua Li, Xiuqiang He, and Ruixuan Li, "Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation," Advances in Neural Information Processing Systems 38 (NeurIPS), 2025. Paper

[3] Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron C. Courville, and Se-Young Yun, "Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation," Advances in Neural Information Processing Systems 38 (NeurIPS), 2025. Paper

Questions for authors
  1. Clarification — recursion budget. Table 6(a) reports 61.21 for input-dependent depth versus 60.46 for the gated one-step control. How closely did the adaptive runs match the control's mean of one executed update per token during training and evaluation? Equation (7) specifies a target, and Figure 2(c) provides balancing evidence, but its correspondence to these runs is unclear. Clarifying the realized budget would strengthen the interpretation of this gain as better allocation of computation rather than additional computation.

  2. Clarification — routing comparison. For the one-shot token-choice variants in Table 6(b), what update rule and router-training path were used after selecting a depth? In particular, did routing scores scale the executed updates and receive task-loss gradients, as the threshold router does in Appendix B? Section 4.5 specifies the routing inputs and parameter counts but leaves this implementation unclear. The answer would clarify how much the 61.21 versus 60.93/60.95 comparison supports sequential state re-evaluation, as opposed to differences in continuous gating or router supervision.

4 comments

We thank the reviewer for the insightful feedback.


W1. Our objective is to enhance adaptation expressiveness under the constrained parameter budget in PEFT, rather than to advance existing conditional recurrence as a general-purpose computational framework. To this end, MoLR employs adaptive recursion to enrich latent-space computation without introducing trainable parameter overhead that scales with the maximum recursion depth. A detailed methodological comparison with the mentioned methods is provided below.

MethodPEFTInput-dependent ComputationAdaptive Recursion
MoSLoRA✓✗✗
TopLoRA✓✓✗
MoR✗✓✓
MoLR✓✓✓
  • vs. MoSLoRA. MoSLoRA applies a learnable, input-independent mixer between LoRA’s down- and up-projections, lacking the capability to capture the semantic variation between inputs. MoLR instead recursively refines the latent state with a shared transformation, enabling input-dependent computation without proportional parameter overhead.
  • vs. TopLoRA. TopLoRA adjusts latent features through an input-conditioned diagonal transformation at a fixed computational depth. MoLR explores adaptive latent depth as a complementary way to enrich adaptation, with empirical benefits demonstrated in Tabs. 2–4.
  • vs. MoR. MoR investigates conditional recurrence at the Transformer-block level, exploring layer sharing and specialized KV-cache strategies for efficient backbone computation. Building on this principle, MoLR introduces conditional recurrence within low-rank adapters to enrich adaptation while limiting parameter overhead. Its shared router uses local thresholds to re-evaluate continuation from evolving latent states. This enables balanced token-wise routing without causality violation or auxiliary routing supervision. The thresholding also provides an additional degree of freedom for controlling recursion-depth utilization (see discussion below).

W2: Latent Computation Allocation. MoLR learns to allocate input-dependent recursion depths from the primary task objective, as demonstrated in Fig. 2a. To further investigate the relationship between adaptation capacity and recursion depth, we analyze the distribution of recursion depths for tokens sampled uniformly from the vocabulary based on their recursion utilization in Fig. 4. Each box summarizes the average recursion depths over token occurrences within all modules in the model. We observe that semantically complete words (e.g., "Tools", "jobs", "sleep") generally receive greater recursion depths than subword fragments (e.g., "ic", "-s", "ud"). These findings support that MoLR learns heterogeneous recursion allocations implicitly from task loss.

W2: Explicit Capacity-Demand Alignment. As a preliminary investigation of explicit capacity-demand alignment, we explore estimating input-specific recursion requirements from the marginal reduction in task loss and adjusting the target recursion-depth distribution accordingly. Specifically, the marginal loss reduction is approximated to first order as:

$\displaystyle \Delta\mathcal{L}_t(x) \approx -s_t\left\langle \frac{\partial\mathcal{L}}{\partial z_{t+1}}, W_{\text{latent}}(z_t+\phi(t)) \right\rangle. $

The preferred depth D^(x)\widehat{D}(x) is then selected by minimizing the marginal loss over the current depth and its feasible neighboring depths:

$\displaystyle \widehat{D}(x) = \arg\min_{d\in\{D(x)-1,D(x),D(x)+1\}\cap\{0,\ldots,T\}} \begin{cases} \Delta\mathcal{L}_{D(x)-1}(x), & d=D(x)-1, \\ 0, & d=D(x), \\ -\Delta\mathcal{L}_{D(x)}(x), & d=D(x)+1. \end{cases} $

We construct a maximum-entropy target distribution whose expected depth matches the batch-averaged preferred depth, promoting balanced depth utilization under the estimated recursion budget. The routing thresholds are then updated according to Eq. (9).

$\displaystyle P_{\text{target}}(D(x)=d) = \frac{e^{\theta d}}{\sum_{j=0}^{T}e^{\theta j}}, \qquad d=0,\ldots,T. $

where θ\theta is chosen to satisfy

$\displaystyle \frac{\sum_{d=0}^{T}d e^{\theta d}}{\sum_{d=0}^{T}e^{\theta d}} = \mathbb{E}_x[\widehat{D}(x)]. $

This explicit alignment yields no consistent performance improvement over the current design, potentially due to the optimization instability introduced by dynamically varying target distributions. These findings highlight the challenges of explicit capacity-demand alignment, and we leave developing more effective mechanisms for difficulty-aware or task-dependent depth allocation as a meaningful direction for future research.

MethodARC-CBoolQGSM8KHellaSwagHumanEvalMMLUWinoGrandeAvg.
w/ budget estimation54.4789.1740.5479.1224.3955.3483.6460.94
w/o budget estimation (current)54.6989.5240.1879.1125.6155.4883.9061.21

W2: Inference Latency. In Tab. 5, we profile methods without runtime optimization to enable controlled comparisons and minimize optimization-induced confounding effects. To better reflect practical deployment, we additionally measure per-token latency (ms) with commonly used inference optimizations, including KV caching, compilation, and CUDA Graphs. With these general-purpose optimizations alone, MoLR already approaches the latency of merged LoRA, supporting its practical deployment. Specialized kernels could further reduce the remaining overhead. We leave their development to future work.

MethodEagerOptimizedNLR Avg.
LoRA (merged)0.180.0759.21
MoLR0.690.0961.21

Q1. Eq. (7) defines the target depth for the input-dependent variant, and Fig. 2c provides evidence of depth balancing for this variant. To clarify the realized computational budget for the input-dependent variant in Tab. 6a, we measured the average recursion depth per input, averaged across all modules over the full training and evaluation sets on each task. The results below show that most measurements are matched to the control’s fixed budget of one recursion per token.

SplitARC-CBoolQGSM8KHellaSwagHumanEvalMMLUWinoGrande
Training0.9401.0441.2890.9460.9911.0080.995
Test0.9401.0431.2880.9430.9910.8410.995

Q2. For the token-choice routing from inputs variant, we define the router Wrouter∈R(T+1)×din\boldsymbol{W}_{\text{router}} \in \mathbb{R}^{(T+1) \times d_{\text{in}}}. The token-choice routing from latent states variant instead uses z0=Ax\boldsymbol{z}_{0} = \boldsymbol{A}\boldsymbol{x} as input, with router dimensions (T+1)×r(T+1) \times r. The depth probabilities are computed once before recursion as:

$\displaystyle \boldsymbol{p} = \operatorname{softmax}(\boldsymbol{W}_{\text{router}}\boldsymbol{x}), \qquad D=\operatorname*{arg\,max}_{d\in\{0,\ldots,T\}}(p_d+b_d). $

At each recursion step t∈{0,…,T−1}t \in \{0,\dots,T-1\}, the latent state is updated as follows:

$\displaystyle \boldsymbol{z}_{t+1} = \begin{cases} \boldsymbol{z}_t+p_{t+1}\boldsymbol{W}_{\text{latent}}(\boldsymbol{z}_t+\phi(t)), & t < D, \\ \boldsymbol{z}_t, & \text{otherwise}. \end{cases} $

We retain the same step-encoding function ϕ(⋅)\phi(\cdot). The non-trainable balancing bias b\boldsymbol{b} only affects depth selection and is updated based on the observed depth frequencies following Eq. (9). All compared variants in Tab. 6b incorporate continuous gating and receive task-loss gradients for router supervision.

Thank you for the rebuttal and for clarifying the one-shot routing comparison. The results support a modest benefit from re-evaluating the updated latent state. I am keeping my rating of 3 because causal, threshold-based stopping within a low-rank adapter is a useful architectural contribution of limited scope. MoR already provides the underlying mechanism of sequential re-evaluation and early exit.

We thank the reviewer for the suggestions and comments during both the review and rebuttal stages. We are glad that our response addresses your concerns and that you find MoLR to have a useful architectural contribution. We will include these additional results and discussion in the revision.

Reviewer_eJFfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Mixture of Latent Recursions (MoLR), a parameter-efficient adaptation method that introduces recursive computation within the low-rank latent space of LoRA. A shared transformation iteratively refines the latent representations without increasing the number of trainable parameters proportionally to recursion depth. The method further introduces a threshold-based router to adaptively determine the recursion depth for each token, along with a load-balancing mechanism to encourage balanced utilization of different depths. Extensive experiments across language understanding, reasoning, and image classification tasks demonstrate consistent improvements over representative LoRA variants. Additional ablation studies and analyses investigate the effects of recursive computation, routing strategies, and computational overhead.

Critical strengths

1. Reasonable and parameter-efficient design.

Introducing shared recursive transformations within LoRA's compact latent space is a reasonable approach to enhancing adaptation expressiveness while maintaining a small parameter budget. The framework provides a straightforward way to increase computational depth without introducing separate trainable transformations at each step. The overall design is technically sound and offers a practical approach to enriching latent-space computation.

2. Strong empirical performance and comprehensive evaluation.

The experimental results are encouraging, showing consistent improvements over representative LoRA variants across multiple backbone architectures and both language and vision tasks. The paper also provides extensive ablation studies covering recursion strategies, routing mechanisms, step conditioning, and hyperparameter sensitivity. These experiments offer convincing empirical support for the effectiveness of the proposed framework.

Critical weaknesses

Insufficient motivation and analysis of adaptive recursion.

While introducing shared recursive transformations is a reasonable approach to enhancing adaptation expressiveness, the motivation for further incorporating input-dependent recursion depths is not sufficiently developed. The paper attributes this design to semantic and contextual variations across tokens, but provides limited explanation of why such variations would lead to different requirements for recursive refinement, or what characteristics of the inputs might determine the appropriate recursion depth.

Although Table 6(a) demonstrates the empirical benefits of adaptive recursion over fixed and random depth strategies, the underlying behavior of the learned routing mechanism remains insufficiently explored. Fig. 2(a) provides an illustrative example of token-wise depth allocation, but it is unclear whether the observed patterns generalize across different inputs and tasks. A more systematic analysis would help connect the motivation for adaptive computation with the actual patterns learned by the model.

Questions for authors

1. Can the authors further clarify the motivation for input-dependent adaptive recursion? In particular, what characteristics of tokens or their latent representations are expected to determine the need for different numbers of recursive refinement steps?

2. Can the authors provide a more systematic analysis of the learned depth allocation patterns beyond the qualitative example in Fig. 2(a)? For instance, examining how recursion depth relates to token characteristics, prediction uncertainty, or task difficulty across different samples would help explain what the routing mechanism has learned and provide stronger support for the motivation of adaptive recursion.

1 comment

We thank the reviewer for the helpful feedback.


W1 & Q1: Motivation. We observe that the same word can have different meanings and functional roles across contexts, resulting in different contextualized token representations. We hypothesize that these representations may benefit from different degrees of task-specific latent refinement, making a uniform recursion depth unnecessarily restrictive. This motivates conditioning continuation decisions on the evolving latent state, allowing MoLR to learn when further transformation is beneficial. Input conditioning also has a structural motivation: when both recursion depth and update coefficients are input-independent, repeated linear updates collapse to a single fixed mixer, up to an offset from step encodings. MoLR instead induces input-dependent transformation coefficients and orders and learns to allocate input-dependent recursion depths from the primary task objective, as shown in Fig. 2a. Its empirical advantages over fixed and random recursion allocation are demonstrated in Tab. 6a.


W1 & Q2: Analysis. To further investigate the relationship between adaptation capacity and recursion depth, we analyze the distribution of recursion depths for tokens sampled uniformly from the vocabulary based on their recursion utilization in Fig. 4. Each box summarizes the average recursion depths over token occurrences within all modules in the model, with the box indicating the interquartile range, the central line denoting the median, and the whiskers representing the minimum and maximum. We observe that semantically complete words (e.g., "Tools", "jobs", "sleep") generally receive greater recursion depths than subword fragments (e.g., "ic", "-s", "ud"). These findings support that MoLR learns heterogeneous recursion allocations implicitly from task loss.

As a preliminary investigation of explicit capacity-demand alignment, we explore estimating input-specific recursion requirements from the marginal reduction in task loss and adjusting the target recursion-depth distribution accordingly. Specifically, the marginal loss reduction is approximated to first order as:

$\displaystyle \Delta\mathcal{L}_t(x) \approx -s_t\left\langle \frac{\partial\mathcal{L}}{\partial z_{t+1}}, W_{\text{latent}}(z_t+\phi(t)) \right\rangle. $

The preferred depth D^(x)\widehat{D}(x) is then selected by minimizing the marginal loss over the current depth and its feasible neighboring depths:

$\displaystyle \widehat{D}(x) = \arg\min_{d\in\{D(x)-1,D(x),D(x)+1\}\cap\{0,\ldots,T\}} \begin{cases} \Delta\mathcal{L}_{D(x)-1}(x), & d=D(x)-1, \\ 0, & d=D(x), \\ -\Delta\mathcal{L}_{D(x)}(x), & d=D(x)+1. \end{cases} $

We construct a maximum-entropy target distribution whose expected depth matches the batch-averaged preferred depth, promoting balanced depth utilization under the estimated recursion budget. The routing thresholds are then updated according to Eq. (9).

$\displaystyle P_{\text{target}}(D(x)=d) = \frac{e^{\theta d}}{\sum_{j=0}^{T}e^{\theta j}}, \qquad d=0,\ldots,T. $

where θ\theta is chosen to satisfy

$\displaystyle \frac{\sum_{d=0}^{T}d e^{\theta d}}{\sum_{d=0}^{T}e^{\theta d}} = \mathbb{E}_x[\widehat{D}(x)]. $

This explicit alignment yields no consistent performance improvement over the current design, potentially due to the optimization instability introduced by dynamically varying target distributions. These findings highlight the challenges of explicit capacity-demand alignment, and we leave developing more effective mechanisms for explicit difficulty-aware or task-dependent depth allocation as a meaningful direction for future research.

MethodARC-CBoolQGSM8KHellaSwagHumanEvalMMLUWinoGrandeAvg.
w/ budget estimation54.4789.1740.5479.1224.3955.3483.6460.94
w/o budget estimation (current)54.6989.5240.1879.1125.6155.4883.9061.21

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.