- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a test-time routing method called Block-Level Recursion (BLR) that supports running one contiguous block of layers a second time before the forward pass continues. The paper also introduces two policies for BLR, static policy repeats fixed blocks, while adaptive version adds a lightweight global router that selects the block per input. On Qwen2.5 base models, BLR generally improves average accuracy over both the unrouted model and Dr.LLM, though it underperforms both on DART math. On instruction-tuned checkpoints the gains largely disappear. Under the optimal settings with gold-guided search, Cola’s larger route space also finds more correct answers than BLR’s.
Critical strengths
The paper provides a trasparent and comprehensive evaluation and analysis on proposed methods. Restricting the route family to single contiguous-block repetition makes exhaustive per-instance evaluation feasible, so the oracle can be measured exactly rather than approximated by search.
Critical weaknesses
W1. The paper has limited novelty as the core mechanism has been established prior work, repeating a block of layers in a frozen, pretrained LLM for test-time adaptation has been explored. Relative to the cited routing methods, BLR is a restricted design rather than extension. Dr.LLM already learns per-input skip/execute/repeat decisions, and Cola composes skip and repeat edits over contiguous segments. BLR narrows these to a single extra pass over one contiguous block.
W2. The practical gains from the proposed methods are limited and can be largely explained by a weak baseline, as the Qwen 2.5 base models has close-to-random score under the paper's protocol. However when compared to instruction-tuned models, as shown in Table 15 and 16, the gain is minimal and sometimes the performance is worse than default setting under pooled sBLR. Since instruction-tuned models are those deployed in reality, this sharply limits practical relevance of the proposed method. Cross-referencing Table 9 with Table 1, LoRA outperforms aBLR at every scale. The authors argue LoRA is not directly comparable because it modifies weights, but it is the obvious alternative for a practitioner who already has labeled training data, which BLR also requires.
Questions for authors
-
How does BLR differ from the published test-time routing methods, beyond restricting to a single repeat? Can the authors clarify which contributions are new relative to the prior work?
-
Since LoRA and BLR are framed as addressing different problems, can they be combined? Applying sBLR and aBLR on top of the LoRA-tuned backbone would test whether BLR provides gains beyond what parametric adaptation already achieves. Because LoRA likely removes the output-format failures that account for much of the base-model gain, this would also show whether BLR’s improvements reflect better computation rather than format repair.
-
aBLR has marginal improvement over sBLR overall (Table 1), yet requires roughly 2x the FLOPs of the default model (Table 20). On instruction-tuned Qwen2.5, both policies are flat or negative at 1.5B and 7B (Table 15). Under what conditions do the authors expect this accuracy–cost trade-off to be favorable? Can the authors compare aBLR against simple alternatives at matched FLOPs, such as running the larger model or self-consistency with a few samples?
7 comments
Response (2/2)
W2b. Comparison with LoRA
We agree with the reviewer that LoRA is the natural choice for a practitioner who has labeled data and can update the model weights. We study a related but narrower question: how much can test-time routing improve an LLM whose weights must stay frozen? We include LoRA as a reference for what efficient post-training achieves on the same data, and we made this framing clearer in Section 4.1 and Appendix C.2.6.
LoRA outperforms not only BLR but also the other frozen-weight methods we evaluate, Dr.LLM and PoLaR. It reaches 43.2, 58.0 and 68.6% overall at 0.5B, 1.5B and 7B (Table 11), compared with 36.9, 44.5 and 61.9% for aBLR, 27.8, 25.9 and 37.9% for Dr.LLM (Table 1), and 27.5, 25.6 and 25.9% for PoLaR (revised Table 1 and our response to Reviewer VSFu, W3). We see this as the expected advantage of updating the weights over keeping them fixed. Despite this advantage, DART is an exception: LoRA falls below the unrouted model at every scale (7.4, 12.1 and 21.6% against 9.0, 16.4 and 29.6%) and also below aBLR and Dr.LLM.
Q2. Combining LoRA and BLR
We agree that combining LoRA and BLR is the most direct way to test whether BLR adds to parametric adaptation. Due to the short rebuttal period, we do not train sBLR and aBLR on the LoRA-tuned models. Instead, we merge the LoRA adapters from Table 11 into the three Qwen2.5 base models and evaluate every BLR route on 128 questions per test set, the same subsample as in Appendix D.3. This shows whether repeating a block can still improve a model after LoRA fine-tuning. We average the ten test sets equally, while Tables 1 and 11 weight them by size.
| Qwen2.5-0.5B | Qwen2.5-1.5B | Qwen2.5-7B | |
|---|---|---|---|
| Base model | 22.3 | 24.1 | 38.5 |
| LoRA | 31.2 (+8.9) | 40.1 (+16.0) | 51.5 (+13.0) |
| + best single block | 31.1 (−0.1) | 41.3 (+1.2) | 52.0 (+0.5) |
| + best block per test set | 34.7 (+3.5) | 44.3 (+4.2) | 54.8 (+3.3) |
| + Oracle BLR | 69.5 (+38.3) | 74.6 (+34.5) | 80.5 (+29.0) |
Accuracy (%). Changes are relative to the base model for LoRA and to LoRA for all rows below it.
Even after LoRA fine-tuning, some blocks still improve accuracy. A single block for all test sets adds 1.2 and 0.5 points at 1.5B and 7B. Choosing one block per test set adds 3.3 to 4.2 points over LoRA at all three scales. Oracle BLR also remains far above the LoRA baseline, similar to what we observe before LoRA fine-tuning. We added this experiment to Appendix C.2.7 (Table 12).
Q3. Accuracy–cost trade-off of aBLR
We agree that the gains of aBLR have to be weighed against its cost. We discuss instruction-tuned checkpoints in W2a above.
We expect the cost of aBLR to pay off in two settings. aBLR costs about twice the default FLOPs (Table 29). The first setting is when memory rules out a larger model. BLR reuses the same weights, so aBLR on Qwen2.5-0.5B needs about 14 times less peak GPU memory than the unrouted Qwen2.5-7B (1,027 vs 14,615 MiB). The second is on larger backbones. From 0.5B to 7B, the gain of aBLR over the default grows from 9.4 to 26.3 points and its margin over sBLR from 0.2 to 3.6 points (Table 1), while its cost stays at about twice the default FLOPs. At 0.5B, where this margin is small, sBLR is the cheaper choice at 1.4 times the default FLOPs.
aBLR outperforms larger models and self-consistency at the same or lower cost. aBLR on Qwen2.5-0.5B (0.24 TFLOPs per example, 36.9%) outperforms the unrouted Qwen2.5-1.5B (0.36 TFLOPs, 25.6%) and Qwen2.5-7B (1.79 TFLOPs, 35.6%), and aBLR on Qwen2.5-1.5B (0.74 TFLOPs, 44.5%) outperforms Qwen2.5-7B at less than half its FLOPs (Tables 1 and 29). For self-consistency, we take a majority vote over answers sampled from the default route of Qwen2.5-0.5B (temperature 0.7, top- 0.95, three seeds) on 128 test questions per dataset, and evaluate the default, sBLR and aBLR on the same questions. At the same cost, aBLR is 5.6 points more accurate than self-consistency, which only comes close with 64 or 128 samples, at 31 to 61 times the FLOPs. At 0.7 times the FLOPs of aBLR, sBLR also outperforms self-consistency with up to 16 samples. We added this comparison to Appendix G.2 (Table 30).
| Method (Qwen2.5-0.5B) | FLOPs relative to aBLR | Accuracy, all datasets |
|---|---|---|
| Default | 22.3 | |
| sBLR | 25.9 | |
| aBLR | 27.0 | |
| Self-consistency, | 21.4 | |
| Self-consistency, | 22.9 | |
| Self-consistency, | 24.2 | |
| Self-consistency, | 25.2 | |
| Self-consistency, | 26.1 | |
| Self-consistency, | 26.5 | |
| Self-consistency, | 26.4 |
Accuracy (%) on 128 test questions per dataset, with each dataset weighted equally.
References
[1] Li et al. Skip a Layer or Loop It? Learning Program-of-Layers in LLMs. ICML 2026.
Dear Reviewer Bfd8,
We would like to thank you once again for your detailed review and for recognizing the value of our comprehensive evaluation and the ability to measure the oracle exactly.
In response to your review, we provided the following clarifications and additional experiments:
- Clarified the contributions of BLR relative to existing routing methods, particularly Dr.LLM and PoLaR, and added an experimental comparison with the learned PoLaR predictor.
- Expanded our discussion of the limitations on instruction-tuned models and clarified how BLR differs from LoRA-based adaptation.
- Evaluated BLR routes on LoRA-fine-tuned models to investigate whether block repetition can provide additional improvements beyond parameter-efficient fine-tuning.
- Provided additional accuracy–FLOPs comparisons against larger unrouted models and self-consistency, alongside a discussion of memory requirements and when adaptive routing is beneficial.
As we approach the end of the discussion period, we would appreciate hearing whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining questions or provide further clarifications.
Response (1/2)
We thank Reviewer Bfd8 for the detailed review and for recognizing that restricting the route family lets us measure the oracle exactly instead of approximating it by search. Below, we address the remaining questions and points of criticism.
W1/Q1. Novelty relative to prior routing methods
Since CoLa is the preliminary version of PoLaR [1], which uses the same MCTS search to find its training routes, the revised paper refers to this search as PoLaR (MCTS) and to the learned predictor as PoLaR. We agree that BLR is a restricted design, since it allows only one repetition of one contiguous block, and this restriction is our main contribution. It changes our approach to routing in four ways.
The BLR family is small enough to evaluate exhaustively. With L(L−1)/2+1 routes (277 to 379 for our backbones), we run every route on every example and measure the coverage of the family directly, which we report as Oracle BLR. The route spaces of Dr.LLM (3L) and PoLaR (MCTS) grow exponentially with depth and cannot be enumerated, so a finite search over them only gives a lower bound on what they contain. On ARC and DART, Oracle BLR reaches 89 to 93% of the gain of PoLaR (MCTS) over the unrouted model at the same budget of 277 to 379 routes (Table 3).
Exhaustive evaluation gives a dense reward for every route. Because we evaluate every route on every training example, our global router learns from a reward for each of the 277 to 379 routes per example, including the routes that answer incorrectly. Dr.LLM and PoLaR [1] instead search for routes with MCTS and train only on part of the routes they evaluate. In our comparison, aBLR outperforms Dr.LLM by 9.1, 18.6 and 24.0 points overall at 4 to 12% more FLOPs (Tables 1 and 29).
A small route family allows a router over whole routes. Because the BLR family has only 277 to 379 routes, aBLR can choose one whole route for each input. Dr.LLM and PoLaR instead predict their routes layer by layer, and their training routes change different layers for different examples. As a result, at almost every layer the most common label keeps the layer as it is, and together these labels form the default route. For Dr.LLM, for example, at least 82% of the training labels at each layer keep the layer (Appendix C.2.2). In our experiments, both methods stay close to the unrouted model at 0.5B and 1.5B (Table 1 and our response to Reviewer VSFu, W3), and PoLaR selects the default route for 99.4% and 100% of the test questions. A router over whole routes avoids this averaging, and aBLR selects a non-default route for every one of 1,000 MMLU test questions at all three scales (Appendix C.2.2).
BLR repeats blocks of any length. Dr.LLM repeats a single layer, and one PoLaR (MCTS) action repeats at most four consecutive layers. We instead test whether a much smaller family, one repetition of a contiguous block of arbitrary length, improves the model in a similar way. Two of our three sBLR blocks are longer than four layers: layers 13 to 21 (nine layers) on Qwen2.5-0.5B and layers 2 to 6 (five layers) on Qwen2.5-1.5B. The best block for Qwen2.5-0.5B on DART-5 spans layers 6 to 21.
We made these differences clearer in the related work (Section 2).
W2a. Gains on instruction-tuned models
Several instruction-tuned checkpoints do benefit from BLR. On Qwen2.5-0.5B-Instruct, sBLR raises overall accuracy from 37.3 to 39.5% (Table 19). In other model families, a single static block selected on the training data raises in-domain accuracy by 3.9 points on Llama-3.2-1B-Instruct, 1.9 on OLMo-2-1B-Instruct and 1.4 on OLMo-3-7B-Instruct (Table 22). On OLMo-3-7B-Instruct, the same block improves all five DART levels, by 1.9 to 9.3 points (Section 4.3).
BLR also improves models whose default is not weak. In our answer to Q2, LoRA fine-tuning raises the default accuracy of the three base models by 8.9 to 16.0 points, and choosing one block per test set still adds 3.3 to 4.2 points on top of it. In our responses to Reviewers VSFu (Q1) and RsD7 (W3), we also evaluate chain-of-thought prompting on 128 GSM8K test questions, where the 7B default model reaches 78.9% with a 0-shot prompt and 86.7% with an 8-shot prompt. The best single block on these questions raises its accuracy to 87.5% and 89.8%.