acceptodds
Under review as a conference paper at ICLR 2027

Block-Level Recursion: Adaptive Test-Time Routing in Large Language Models

Abstract

Test-time routing improves frozen large language models (LLMs) by taking non-linear paths through their layers, without modifying weights or generating extra tokens. Existing approaches define route spaces that grow exponentially with depth, making them costly to search and hard to learn from. We therefore introduce (), a restricted route family that repeats a single contiguous block of transformer layers once. This reduces the number of routes from exponential to quadratic in the number of layers, making exhaustive per-instance evaluation tractable and the oracle upper bound directly measurable. Despite this restriction, yields large accuracy gains while keeping route selection practical. Across six model families and ten reasoning benchmarks, the optimal block varies across models, tasks, and individual inputs, with per-instance oracle gains of overall and up to on individual tasks. also supports two practical policies: a single train-selected block () that requires no router or per-input overhead, and a learned global router () trained from dense per-instance rewards over all routes. already improves accuracy substantially, while improves further by selecting routes per input. With a frozen Qwen2.5-0.5B backbone, achieves higher accuracy than the unrouted Qwen2.5-7B model at lower FLOPs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llms, test-time routing, adaptive computation, inference-time scaling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Bfd8OpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a test-time routing method called Block-Level Recursion (BLR) that supports running one contiguous block of layers a second time before the forward pass continues. The paper also introduces two policies for BLR, static policy repeats fixed blocks, while adaptive version adds a lightweight global router that selects the block per input. On Qwen2.5 base models, BLR generally improves average accuracy over both the unrouted model and Dr.LLM, though it underperforms both on DART math. On instruction-tuned checkpoints the gains largely disappear. Under the optimal settings with gold-guided search, Cola’s larger route space also finds more correct answers than BLR’s.

Critical strengths

The paper provides a trasparent and comprehensive evaluation and analysis on proposed methods. Restricting the route family to single contiguous-block repetition makes exhaustive per-instance evaluation feasible, so the oracle can be measured exactly rather than approximated by search.

Critical weaknesses

W1. The paper has limited novelty as the core mechanism has been established prior work, repeating a block of layers in a frozen, pretrained LLM for test-time adaptation has been explored. Relative to the cited routing methods, BLR is a restricted design rather than extension. Dr.LLM already learns per-input skip/execute/repeat decisions, and Cola composes skip and repeat edits over contiguous segments. BLR narrows these to a single extra pass over one contiguous block.

W2. The practical gains from the proposed methods are limited and can be largely explained by a weak baseline, as the Qwen 2.5 base models has close-to-random score under the paper's protocol. However when compared to instruction-tuned models, as shown in Table 15 and 16, the gain is minimal and sometimes the performance is worse than default setting under pooled sBLR. Since instruction-tuned models are those deployed in reality, this sharply limits practical relevance of the proposed method. Cross-referencing Table 9 with Table 1, LoRA outperforms aBLR at every scale. The authors argue LoRA is not directly comparable because it modifies weights, but it is the obvious alternative for a practitioner who already has labeled training data, which BLR also requires.

Questions for authors
  1. How does BLR differ from the published test-time routing methods, beyond restricting to a single repeat? Can the authors clarify which contributions are new relative to the prior work?

  2. Since LoRA and BLR are framed as addressing different problems, can they be combined? Applying sBLR and aBLR on top of the LoRA-tuned backbone would test whether BLR provides gains beyond what parametric adaptation already achieves. Because LoRA likely removes the output-format failures that account for much of the base-model gain, this would also show whether BLR’s improvements reflect better computation rather than format repair.

  3. aBLR has marginal improvement over sBLR overall (Table 1), yet requires roughly 2x the FLOPs of the default model (Table 20). On instruction-tuned Qwen2.5, both policies are flat or negative at 1.5B and 7B (Table 15). Under what conditions do the authors expect this accuracy–cost trade-off to be favorable? Can the authors compare aBLR against simple alternatives at matched FLOPs, such as running the larger model or self-consistency with a few samples?

7 comments
AuthorsOpenReview ↗

Response (1/2)

We thank Reviewer Bfd8 for the detailed review and for recognizing that restricting the route family lets us measure the oracle exactly instead of approximating it by search. Below, we address the remaining questions and points of criticism.

W1/Q1. Novelty relative to prior routing methods

Since CoLa is the preliminary version of PoLaR [1], which uses the same MCTS search to find its training routes, the revised paper refers to this search as PoLaR (MCTS) and to the learned predictor as PoLaR. We agree that BLR is a restricted design, since it allows only one repetition of one contiguous block, and this restriction is our main contribution. It changes our approach to routing in four ways.

The BLR family is small enough to evaluate exhaustively. With L(L−1)/2+1L(L-1)/2+1 routes (277 to 379 for our backbones), we run every route on every example and measure the coverage of the family directly, which we report as Oracle BLR. The route spaces of Dr.LLM (3L3^L) and PoLaR (MCTS) grow exponentially with depth and cannot be enumerated, so a finite search over them only gives a lower bound on what they contain. On ARC and DART, Oracle BLR reaches 89 to 93% of the gain of PoLaR (MCTS) over the unrouted model at the same budget of 277 to 379 routes (Table 3).

Exhaustive evaluation gives a dense reward for every route. Because we evaluate every route on every training example, our global router learns from a reward for each of the 277 to 379 routes per example, including the routes that answer incorrectly. Dr.LLM and PoLaR [1] instead search for routes with MCTS and train only on part of the routes they evaluate. In our comparison, aBLR outperforms Dr.LLM by 9.1, 18.6 and 24.0 points overall at 4 to 12% more FLOPs (Tables 1 and 29).

A small route family allows a router over whole routes. Because the BLR family has only 277 to 379 routes, aBLR can choose one whole route for each input. Dr.LLM and PoLaR instead predict their routes layer by layer, and their training routes change different layers for different examples. As a result, at almost every layer the most common label keeps the layer as it is, and together these labels form the default route. For Dr.LLM, for example, at least 82% of the training labels at each layer keep the layer (Appendix C.2.2). In our experiments, both methods stay close to the unrouted model at 0.5B and 1.5B (Table 1 and our response to Reviewer VSFu, W3), and PoLaR selects the default route for 99.4% and 100% of the test questions. A router over whole routes avoids this averaging, and aBLR selects a non-default route for every one of 1,000 MMLU test questions at all three scales (Appendix C.2.2).

BLR repeats blocks of any length. Dr.LLM repeats a single layer, and one PoLaR (MCTS) action repeats at most four consecutive layers. We instead test whether a much smaller family, one repetition of a contiguous block of arbitrary length, improves the model in a similar way. Two of our three sBLR blocks are longer than four layers: layers 13 to 21 (nine layers) on Qwen2.5-0.5B and layers 2 to 6 (five layers) on Qwen2.5-1.5B. The best block for Qwen2.5-0.5B on DART-5 spans layers 6 to 21.

We made these differences clearer in the related work (Section 2).

W2a. Gains on instruction-tuned models

Several instruction-tuned checkpoints do benefit from BLR. On Qwen2.5-0.5B-Instruct, sBLR raises overall accuracy from 37.3 to 39.5% (Table 19). In other model families, a single static block selected on the training data raises in-domain accuracy by 3.9 points on Llama-3.2-1B-Instruct, 1.9 on OLMo-2-1B-Instruct and 1.4 on OLMo-3-7B-Instruct (Table 22). On OLMo-3-7B-Instruct, the same block improves all five DART levels, by 1.9 to 9.3 points (Section 4.3).

BLR also improves models whose default is not weak. In our answer to Q2, LoRA fine-tuning raises the default accuracy of the three base models by 8.9 to 16.0 points, and choosing one block per test set still adds 3.3 to 4.2 points on top of it. In our responses to Reviewers VSFu (Q1) and RsD7 (W3), we also evaluate chain-of-thought prompting on 128 GSM8K test questions, where the 7B default model reaches 78.9% with a 0-shot prompt and 86.7% with an 8-shot prompt. The best single block on these questions raises its accuracy to 87.5% and 89.8%.

AuthorsOpenReview ↗

Response (2/2)

W2b. Comparison with LoRA

We agree with the reviewer that LoRA is the natural choice for a practitioner who has labeled data and can update the model weights. We study a related but narrower question: how much can test-time routing improve an LLM whose weights must stay frozen? We include LoRA as a reference for what efficient post-training achieves on the same data, and we made this framing clearer in Section 4.1 and Appendix C.2.6.

LoRA outperforms not only BLR but also the other frozen-weight methods we evaluate, Dr.LLM and PoLaR. It reaches 43.2, 58.0 and 68.6% overall at 0.5B, 1.5B and 7B (Table 11), compared with 36.9, 44.5 and 61.9% for aBLR, 27.8, 25.9 and 37.9% for Dr.LLM (Table 1), and 27.5, 25.6 and 25.9% for PoLaR (revised Table 1 and our response to Reviewer VSFu, W3). We see this as the expected advantage of updating the weights over keeping them fixed. Despite this advantage, DART is an exception: LoRA falls below the unrouted model at every scale (7.4, 12.1 and 21.6% against 9.0, 16.4 and 29.6%) and also below aBLR and Dr.LLM.

Q2. Combining LoRA and BLR

We agree that combining LoRA and BLR is the most direct way to test whether BLR adds to parametric adaptation. Due to the short rebuttal period, we do not train sBLR and aBLR on the LoRA-tuned models. Instead, we merge the LoRA adapters from Table 11 into the three Qwen2.5 base models and evaluate every BLR route on 128 questions per test set, the same subsample as in Appendix D.3. This shows whether repeating a block can still improve a model after LoRA fine-tuning. We average the ten test sets equally, while Tables 1 and 11 weight them by size.

Qwen2.5-0.5BQwen2.5-1.5BQwen2.5-7B
Base model22.324.138.5
LoRA31.2 (+8.9)40.1 (+16.0)51.5 (+13.0)
+ best single block31.1 (−0.1)41.3 (+1.2)52.0 (+0.5)
+ best block per test set34.7 (+3.5)44.3 (+4.2)54.8 (+3.3)
+ Oracle BLR69.5 (+38.3)74.6 (+34.5)80.5 (+29.0)

Accuracy (%). Changes are relative to the base model for LoRA and to LoRA for all rows below it.

Even after LoRA fine-tuning, some blocks still improve accuracy. A single block for all test sets adds 1.2 and 0.5 points at 1.5B and 7B. Choosing one block per test set adds 3.3 to 4.2 points over LoRA at all three scales. Oracle BLR also remains far above the LoRA baseline, similar to what we observe before LoRA fine-tuning. We added this experiment to Appendix C.2.7 (Table 12).

Q3. Accuracy–cost trade-off of aBLR

We agree that the gains of aBLR have to be weighed against its cost. We discuss instruction-tuned checkpoints in W2a above.

We expect the cost of aBLR to pay off in two settings. aBLR costs about twice the default FLOPs (Table 29). The first setting is when memory rules out a larger model. BLR reuses the same weights, so aBLR on Qwen2.5-0.5B needs about 14 times less peak GPU memory than the unrouted Qwen2.5-7B (1,027 vs 14,615 MiB). The second is on larger backbones. From 0.5B to 7B, the gain of aBLR over the default grows from 9.4 to 26.3 points and its margin over sBLR from 0.2 to 3.6 points (Table 1), while its cost stays at about twice the default FLOPs. At 0.5B, where this margin is small, sBLR is the cheaper choice at 1.4 times the default FLOPs.

aBLR outperforms larger models and self-consistency at the same or lower cost. aBLR on Qwen2.5-0.5B (0.24 TFLOPs per example, 36.9%) outperforms the unrouted Qwen2.5-1.5B (0.36 TFLOPs, 25.6%) and Qwen2.5-7B (1.79 TFLOPs, 35.6%), and aBLR on Qwen2.5-1.5B (0.74 TFLOPs, 44.5%) outperforms Qwen2.5-7B at less than half its FLOPs (Tables 1 and 29). For self-consistency, we take a majority vote over KK answers sampled from the default route of Qwen2.5-0.5B (temperature 0.7, top-pp 0.95, three seeds) on 128 test questions per dataset, and evaluate the default, sBLR and aBLR on the same questions. At the same cost, aBLR is 5.6 points more accurate than self-consistency, which only comes close with 64 or 128 samples, at 31 to 61 times the FLOPs. At 0.7 times the FLOPs of aBLR, sBLR also outperforms self-consistency with up to 16 samples. We added this comparison to Appendix G.2 (Table 30).

Method (Qwen2.5-0.5B)FLOPs relative to aBLRAccuracy, all datasets
Default0.5×0.5\times22.3
sBLR0.7×0.7\times25.9
aBLR1.0×1.0\times27.0
Self-consistency, K=2K = 21.0×1.0\times21.4
Self-consistency, K=4K = 41.9×1.9\times22.9
Self-consistency, K=8K = 83.8×3.8\times24.2
Self-consistency, K=16K = 167.7×7.7\times25.2
Self-consistency, K=32K = 3215×15\times26.1
Self-consistency, K=64K = 6431×31\times26.5
Self-consistency, K=128K = 12861×61\times26.4

Accuracy (%) on 128 test questions per dataset, with each dataset weighted equally.

References

[1] Li et al. Skip a Layer or Loop It? Learning Program-of-Layers in LLMs. ICML 2026.

Dear Reviewer Bfd8,

We would like to thank you once again for your detailed review and for recognizing the value of our comprehensive evaluation and the ability to measure the oracle exactly.

In response to your review, we provided the following clarifications and additional experiments:

  • Clarified the contributions of BLR relative to existing routing methods, particularly Dr.LLM and PoLaR, and added an experimental comparison with the learned PoLaR predictor.
  • Expanded our discussion of the limitations on instruction-tuned models and clarified how BLR differs from LoRA-based adaptation.
  • Evaluated BLR routes on LoRA-fine-tuned models to investigate whether block repetition can provide additional improvements beyond parameter-efficient fine-tuning.
  • Provided additional accuracy–FLOPs comparisons against larger unrouted models and self-consistency, alongside a discussion of memory requirements and when adaptive routing is beneficial.

As we approach the end of the discussion period, we would appreciate hearing whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining questions or provide further clarifications.

Reviewer_RsD7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether frozen LLMs can benefit from repeating a contiguous block of transformer layers at inference time. The proposed Block-Level Recursion (BLR) repeats one block once, making the route space small enough to evaluate exhaustively. The authors consider two selection methods: sBLR chooses one fixed block using training data, while aBLR trains a router to predict per-route rewards and select a block for each input. Experiments examine how useful blocks vary across models and tasks, and evaluate routing accuracy, generalization, and computational cost.

Critical strengths

I find the direction interesting, particularly the observation that different models and tasks benefit from repeating different parts of the network. Restricting the route space makes it possible to study these differences systematically and construct dense supervision for route selection. The distinction between oracle coverage and practical route selection is also useful, and I appreciate the additional controls for sampling diversity and answer formatting.

Critical weaknesses
  • My main concern is that the practical gains are concentrated on base models. On Qwen2.5-1.5B-Instruct and 7B-Instruct, neither sBLR nor aBLR improves overall accuracy. Since instruction-tuned models are more representative of practical use, I would appreciate a more prominent discussion of this limitation and why the gains become smaller after instruction tuning.

  • Constructing the route-supervision dataset requires evaluating every candidate route for every training example. The reported offline costs are helpful, but it remains unclear how this approach scales to deeper models, larger datasets, or longer responses. Could evaluating only a subset of routes or training examples achieve similar performance?

  • The evaluation covers several benchmarks, but router training uses only ARC and DART, and all experiments use short-answer generation. The OOD evaluations are useful, though it remains unclear whether the findings extend to longer reasoning or open-ended generation.

  • Adaptive routing provides relatively modest improvements over the fixed-block policy at some scales: 0.2 and 1.2 points on the 0.5B and 1.5B base models. Given the additional inference cost, it would help to discuss when input-dependent routing is preferable to sBLR. Results across router training seeds would also help interpret these smaller differences.

Questions for authors

1/ Does the preferred block depend on problem difficulty? Relating block location and length to difficulty could provide further insight beyond the observed differences across benchmarks.

2/ How does aBLR compare with selecting a separate static block for each task in the main base-model setting? This would help distinguish task-level adaptation from finer input-level adaptation.

3/ Figure 2 appears before its main explanation and combines different settings. In particular, its lower panel includes instruction-tuned Qwen2.5 models, while Table 1 reports base models. Clearer labeling and an earlier explanation would make the results easier to follow.

4 comments
AuthorsOpenReview ↗

Response (1/2)

We thank Reviewer RsD7 for the careful review and are glad that the reviewer finds it interesting that different models and tasks benefit from repeating different blocks. Below, we address the remaining questions and points of criticism.

W1. Gains on instruction-tuned models

We agree with the reviewer that instruction-tuned models are closer to practical use. Several instruction-tuned checkpoints benefit from block-level recursion. sBLR raises the overall accuracy of Qwen2.5-0.5B-Instruct from 37.3 to 39.5% (Table 19), and a single static block raises in-domain accuracy by 3.9 points on Llama-3.2-1B-Instruct, 1.9 on OLMo-2-1B-Instruct and 1.4 on OLMo-3-7B-Instruct (Table 22).

To understand when BLR helps after instruction tuning, we compare each Qwen2.5 base model with its instruction-tuned version. The table below shows the accuracy of the default route, the number of routes that outperform it, and the gain of Oracle BLR over it.

0.5B1.5B7B
BaseInstructBaseInstructBaseInstruct
Default accuracy27.537.325.656.135.667.8
Routes above the default34913901322
Oracle BLR gain over the default+59.1+55.5+58.1+38.3+56.6+27.9

Overall accuracy (%) on the ten test sets (Tables 1 and 19). Routes above the default out of 276 (0.5B) and 378 (1.5B, 7B) non-default routes.

Two factors decide how much room an instruction-tuned model leaves for BLR. First, part of the base-model gain comes from repairing the output format, while the instruction-tuned checkpoints always give a parsable answer in our control set (Appendix A.3). However, format repair explains at most 45, 41 and 27% of the aBLR gain on the base models (Table 4). Second, the stronger the default route, the harder it is to find a route that improves on it. An instruction-tuned model already answers more questions correctly, so changing its computation more often breaks a correct answer than fixes a wrong one. Instruction tuning raises the default accuracy by 9.8 points at 0.5B and by 30.5 and 32.2 points at 1.5B and 7B. At the same time, the number of routes that outperform the default drops from 34 to 9 at 0.5B, and from 139 and 132 to almost none at 1.5B and 7B (0 and 2). With almost no better route available, neither policy improves Qwen2.5-1.5B-Instruct and 7B-Instruct (Table 19). At 0.5B, the nine remaining routes leave room for the gain of sBLR. The gain of Oracle BLR over the default shrinks in the same way (see also the heatmaps in Appendix D.2).

We added this discussion to Appendix E.2 (Table 20).

W2. Cost of the route sweep

The number of routes grows only quadratically with depth (277 at 24 layers, 379 at 28 and 2,017 at 64), compared with the exponential route spaces of Dr.LLM and PoLaR [1], and the full sweep over our 6,370 training examples takes 4.2 to 8.4 hours on one node with eight GPUs (Table 28). Prior routing methods also generate many outputs per training example. Dr.LLM and PoLaR run NN MCTS simulations per example, and Dr.LLM keeps a single path for training. Our sweep instead generates one output per route and keeps a reward for every route. Longer responses raise the cost of all these methods alike.

One way to reduce this cost is successive halving. We evaluate all routes on 64 examples per training dataset, keep the better half and double the examples in each round, which generates 19% of the outputs of the full sweep. sBLR uses the last remaining route, and aBLR is trained on the last three routes.

ModelsBLR, FSsBLR, SHaBLR, FSaBLR, SH
0.5B36.736.736.935.7
1.5B43.343.344.543.4
7B58.358.361.962.1

Overall accuracy (%). FS: full sweep; SH: successive halving.

sBLR is not affected, since successive halving finds the route with the highest training accuracy in each of 100 random draws of the examples. aBLR, which can only choose among the three remaining routes, matches the full sweep at 7B but loses 1.2 and 1.1 points at 0.5B and 1.5B. A cheaper sweep therefore suffices for sBLR, while aBLR benefits from evaluating more routes on the smaller models. We added this experiment to Appendix G.3 (Table 31).

W3. Longer reasoning

We agree with the reviewer that all our experiments use short answers. We therefore test BLR on 128 GSM8K questions, where the models write chains of thought of 96 to 151 tokens on average before answering, compared with at most 16 tokens in the paper. At 7B, the best block raises accuracy from 78.9% to 87.5% with a 0-shot chain-of-thought prompt and from 86.7% to 89.8% with an 8-shot prompt. sBLR, selected only on short ARC and DART answers, also improves the 0-shot accuracy from 78.9% to 81.2%. We show the full setup and results in our response to Reviewer VSFu (Q1), and we added them to the paper in Appendix F.2 (Table 27).

AuthorsOpenReview ↗

Response (2/2)

W4. aBLR versus sBLR and router seeds

The advantage of aBLR over sBLR grows with scale (0.2, 1.2 and 3.6 points overall at 0.5B, 1.5B and 7B) and is largest out of domain at 7B (61.9 vs 57.0%, Table 1). aBLR costs about twice the default FLOPs because of its preliminary pass (Table 29), and we expect this cost to pay off in two settings. The first is when memory rules out a larger model. BLR reuses the same weights, so aBLR on Qwen2.5-0.5B needs about 14 times less peak GPU memory than the unrouted Qwen2.5-7B (1,027 vs 14,615 MiB, Table 29). The second is on larger backbones, since the advantage of aBLR grows with scale while its cost stays at about twice the default FLOPs. In our response to Reviewer Bfd8 (Q3), we further show that aBLR is more accurate than self-consistency at matched FLOPs and outperforms larger unrouted models at lower cost (Appendix G.2).

We also repeat the full training pipeline with two additional seeds per backbone. The standard deviation of aBLR across the three seeds is at most 0.6 points (36.4±0.636.4 \pm 0.6, 44.1±0.544.1 \pm 0.5 and 61.6±0.361.6 \pm 0.3 at 0.5B, 1.5B and 7B, against 36.7, 43.3 and 58.3 for sBLR). With every seed, aBLR improves on sBLR at 1.5B, and at 7B it does so by a consistent 3.1 to 3.6 points. At 0.5B, the two policies differ by less than the variation across seeds. We show the per-seed results in our response to Reviewer VSFu (Q2), and we added them to the paper in Appendix C.2.5 (Table 10).

Q1. Preferred blocks and problem difficulty

We agree with the reviewer that relating block location and length to difficulty gives further insight beyond the differences across benchmarks. We use two difficulty axes for 14 models. The first is the five DART levels, which share the task and answer format and differ only in difficulty. The second compares ARC-Easy with ARC-Challenge. We ask two questions: whether the same blocks score best at different difficulties, and how many default errors some route can fix at each difficulty.

The task changes which blocks score best much more than difficulty does. For each model, we correlate the accuracy of every route on one dataset with its accuracy on another and take the median over the 14 models. Route accuracies on ARC-Easy and ARC-Challenge correlate at 0.98, and those on DART-1 to DART-4 at 0.86 to 0.95. Between ARC and these DART levels, the correlation is only 0.62 to 0.67, so different tasks prefer different blocks, as in Finding 2. Only the hardest level, DART-5, where the default model answers a median of 5.2% of the questions correctly, deviates from the other DART levels (0.60 to 0.72). For example, the best block of Qwen2.5-0.5B is layers 20 to 21 on DART-1 but layers 6 to 21 on DART-5.

Harder problems leave fewer errors to fix. Some route corrects a median of 80% of the default errors on DART-1 but only 26% on DART-5. This drop is expected, since BLR can only recover answers that the model can reach with its own layers.

We added this analysis to Appendix D.4 (Figure 25 and Table 16).

Q2. A static block per task

Table 22 reports this comparison for the instruction-tuned checkpoints, and the table below shows it for the three base models.

ModelTest setsDefaultsBLRsBLR per taskaBLR
Qwen2.5-0.5BIn-domain27.933.634.334.1
Out-of-domain27.438.1–38.2
Qwen2.5-1.5BIn-domain27.344.044.146.2
Out-of-domain24.843.0–43.8
Qwen2.5-7BIn-domain42.361.162.462.0
Out-of-domain32.657.0–61.9

Accuracy (%). In-domain: ARC and DART. Out-of-domain: OpenBookQA, GPQA Diamond and MMLU.

On the in-domain tasks, selecting a block per task improves slightly on sBLR, and aBLR performs similarly. Choosing a block per task, however, requires knowing which task an input belongs to. On out-of-domain tasks, this would need a separately trained classifier that assigns each input to one of the training tasks. aBLR instead selects the route from the input alone and improves on sBLR at all three scales, most clearly at 7B (61.9 vs 57.0%). We added this comparison to Appendix E.3 (Table 21).

Q3. Figure 2

We moved Figure 2 into Section 4.2, next to Findings 1 and 2, and split its lower panel into base and instruction-tuned checkpoints.

References

[1] Li et al. Skip a Layer or Loop It? Learning Program-of-Layers in LLMs. ICML 2026.

Dear Reviewer RsD7,

We would like to thank you once again for your thoughtful review and for highlighting the value of systematically studying how different models and tasks benefit from block repetition.

In response to your review, we provided the following clarifications and additional experiments:

  • Expanded our analysis of instruction-tuned models to better explain when BLR is beneficial and when its improvements diminish.
  • Introduced a successive-halving experiment that reduces the cost of route evaluation while largely preserving performance.
  • Evaluated BLR on longer chain-of-thought reasoning.
  • Repeated aBLR training across multiple seeds to assess the stability of its improvements.
  • Analyzed how preferred blocks vary with problem difficulty and compared aBLR with task-specific static routing.
  • Improved the presentation of Figure 2 and clarified the distinction between base and instruction-tuned models.

As the discussion period is coming to an end, we would greatly appreciate hearing whether these additions have addressed your main concerns. We remain happy to answer any further questions or clarify the new experiments.

The additional experiments address several of my concerns, particularly the cost of route supervision and the comparison with task-specific static blocks. I still have reservations about generalizability to larger, deeper instruction-tuned models and more demanding reasoning tasks. However, the new evidence strengthens the empirical contribution sufficiently for me to raise my rating.

Reviewer_VSFuOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Block-Level Recursion (BLR), a test-time method that repeats one contiguous block of layers of a frozen LLM once. This criteria limits the possible routes to only L(L−1)/2 + 1, and hence we can run every route on every training example. This also gives an oracle upper bound and dense rewards for training a router. Based on this, they propose sBLR, which applies one block chosen on the training set to all inputs, and aBLR, a light global router that picks the block for each input after a preliminary pass. On ARC, DART, OpenBookQA, MMLU and GPQA Diamond, both policies improve the Qwen2.5 base models by 9 to 26% overall and clearly outperform Dr.LLM, and the analysis is repeated on five other model families.

Critical strengths

(1) I really like this paper, and the gains are impressive. The idea is very simple, and I was surprised by how much a single static block gives. Without any router, sBLR improves the overall accuracy of the three Qwen2.5 base models by 9.2, 17.7 and 22.7 points (Table 1), with no training of the backbone and no extra tokens.

(2) I like the fact that the route family is small enough to evaluate exhaustively. Table 2 shows that it still keeps 89 to 93% of the oracle gain of the much larger CoLa search space, which is impressive. The fact that a single global router called once per input, rather than a router at every layer as in Dr.LLM, and it still beats Dr.LLM at all three sizes is also impressive to me

Critical weaknesses

(1) My main concern is that the baselines look too weak. In Table 1, the unrouted base models are close to random guessing (e.g., 24.6% on MMLU for 1.5B), while the Qwen2.5 report gives 60.9% for the same model with few-shot prompts, which is higher than aBLR here (43.4%). Also, sBLR barely helps the instruction-tuned models in Fig. 2. I would like to see BLR with few-shot prompts.

(2) I find the oracle gains in the abstract (+59.1 and +75.8) misleading. The oracle is basically pass@277 over routes, so on questions with four options some route is almost always right just by luck, which is why even the 0.5B model reaches 86.9% on GPQA Diamond. I would like to see it compared with pass@277 of the normal model with sampling, in the main text.

(3) I think the closest published work is missing. The ICML 2026 follow-up of CoLa by the same authors (“Skip a Layer or Loop It? Learning Program-of-Layers in LLMs”) also trains a light network that predicts a skip and repeat program for each input of a frozen LLM, which makes it the natural baseline for aBLR. However, the paper only compares with the MCTS search of the earlier arXiv version. Also, Transformer Layers as Painters (AAAI 2025) found that repeating middle layers of a frozen Llama2 did not beat the normal forward pass. I would like to see both discussed, and a comparison with the learned predictor.

Questions for authors

(1) Does BLR still help when the model can generate a chain of thought before answering? I ask because the paper presents BLR as complementary to CoT, but I could not find this tested.

(2) How stable is aBLR across router training seeds? Since its gain over sBLR is only 0.2 and 1.2 points at 0.5B and 1.5B, so I wonder if it is above noise.

4 comments
AuthorsOpenReview ↗

Response (1/2)

We thank Reviewer VSFu for the thoughtful and encouraging review. We appreciate the positive assessment and the insightful suggestions. Below, we address the remaining questions and points of criticism.

W1. Weak baselines and few-shot prompting

We agree with the reviewer that our zero-shot protocol leaves the base models close to chance on MMLU. Part of this is output format: 26% of the default base-model MCQ outputs contain no valid option letter (Appendix A.3). Following the reviewer's suggestion, we run all BLR routes with 5-shot prompts for the three Qwen2.5 base models. Due to the short rebuttal period, we focus on MMLU and use 1,000 randomly sampled test questions. The default model then reaches 48.9%, 60.7% and 74.1% accuracy, in line with the Qwen2.5 report.

Few-shot prompting adds annotated examples to the prompt, while BLR keeps the zero-shot prompt and adds computation inside the model. Without any examples, sBLR recovers 51% to 60% of the 5-shot gain over the zero-shot default and aBLR recovers 51% to 72%, even though both are selected and trained on ARC and DART rather than on MMLU. To see how the two approaches relate, we compare which errors of the zero-shot default each one fixes.

Qwen2.5-0.5BQwen2.5-1.5BQwen2.5-7B
Errors of the 0-shot default723757682
Fixed by 5-shot prompting309424467
Fixed by sBLR (0-shot)201263297
 also fixed by 5-shot prompting148225271
 fixed only by sBLR533826
Fixed by aBLR (0-shot)197244372
 also fixed by 5-shot prompting145210339
 fixed only by aBLR523433

Counts on the 1,000 MMLU test questions.

BLR fixes 26 to 53 questions that 5-shot prompting does not fix, so the two approaches are partly complementary. At the same time, 74% to 91% of the errors that sBLR and aBLR fix are also fixed by 5-shot prompting. This suggests that much of the knowledge that few-shot prompting elicits is already present in the zero-shot model, and that repeating a block can reach it without annotated examples. Once the examples are in the prompt, the best block chosen on these 1,000 questions adds 0.2 to 0.4 points. We added this comparison to Appendix F.1 (Tables 25 and 26).

We discuss several instruction-tuned models that still benefit from BLR in our response to Reviewer RsD7 (W1).

W2. Oracle BLR versus pass@277 with sampling

Since the BLR family consists of 277 different executions of the same backbone, pass@277 under sampling is an important comparison. We therefore included this comparison in Appendix D.3 and moved it to the main text (Section 4.2, Table 2). On Qwen2.5-0.5B with 128 test examples per dataset, we compare the 277 greedy BLR outputs per example with 277 samples from the default route (temperature 0.7, top-pp 0.95, three seeds).

SuiteDefault (pass@1)Oracle BLR (pass@277)Oracle sampling (pass@277)
MCQ suite32.094.795.4
DART12.740.654.8

On multiple-choice questions, 277 noisy candidates often include the correct option by chance, so both oracles reach about 95%. With four options, even 277 random guesses would almost surely include the correct one.

However, DART shows that the gain of Oracle BLR is not due to chance. Its answers are free-form numbers, so a route cannot reach the correct answer by picking among a few options. Still, Oracle BLR raises DART accuracy from 12.7% to 40.6% on these questions, and on the full DART test sets it adds 23.9, 31.3 and 34.0 points at 0.5B, 1.5B and 7B (Table 1). These correct answers therefore come from the computation of different routes, not from guessing. Sampling from the default route reaches 54.8% on DART but needs a verifier to choose among the samples, whereas a route is a deterministic execution of the model that a policy can select from the input alone.

AuthorsOpenReview ↗

Response (2/2)

W3. PoLaR and Transformer Layers as Painters

We added PoLaR [3] to Table 1 and now discuss both papers in Section 2.

We train PoLaR on the ARC and DART training questions of the three Qwen2.5 base models, using the released code and settings (Appendix G.1). At test time, we run the top-ranked program for each question, with the same generation budget and answer extraction we use for BLR.

ModelMethodIn-domainOODOverall
0.5BUnrouted27.927.427.5
PoLaR27.927.327.5
sBLR33.638.136.7
aBLR34.138.236.9
1.5BUnrouted27.324.825.6
PoLaR27.324.825.6
sBLR44.043.043.3
aBLR46.243.844.5
7BUnrouted42.332.635.6
PoLaR33.122.725.9
sBLR61.157.058.3
aBLR62.061.961.9

PoLaR stays close to the unrouted accuracy at 0.5B and 1.5B, while sBLR and aBLR improve on it at every scale. At these two scales, PoLaR selects the default route for 99.4% and 100% of the test questions. At 7B, it skips a block of four middle layers on 76% of the questions and falls below the unrouted model.

Transformer Layers as Painters [4] tests a different operation than BLR. Its repetition variants replace the middle layers with copies of a single layer, or run them in parallel and feed the averaged output back through them several times, whereas BLR runs one contiguous block of layers twice and selects this block for each model and input. We added this distinction to Section 2.

Q1. BLR with chain-of-thought reasoning

We agree with the reviewer that testing BLR with chain-of-thought reasoning is an important experiment. Due to the short rebuttal period, we evaluate this on 128 randomly sampled GSM8K test questions. We use two standard chain-of-thought prompts: the 0-shot prompt "Let's think step by step" and the 8-shot prompt with worked examples of Wei et al. [1]. The model reasons for up to 256 tokens, and we take the last number it writes as its answer. With the 8-shot prompt, the default models reach 39.1%, 65.6% and 86.7%, close to the 41.6%, 68.5% and 85.4% reported on the full test set in the Qwen2.5 technical report [2]. We evaluate every BLR route as well as our two policies, sBLR and aBLR, even though both are selected and trained on short answers, without few-shot examples or chain of thought.

0.5B1.5B7B
0-shot8-shot0-shot8-shot0-shot8-shot
Default35.239.164.865.678.986.7
sBLR21.120.340.641.481.281.2
aBLR18.827.352.364.171.986.7
Best block36.739.864.865.687.589.8

Accuracy (%) on 128 GSM8K test questions with 0-shot and 8-shot chain-of-thought prompts. Results above the default model are in bold.

At 7B, BLR and chain of thought work well together. The single block with the highest accuracy on these questions (best block) raises accuracy from 78.9% to 87.5% with the 0-shot prompt. With the 8-shot prompt it raises accuracy from 86.7% to 89.8%. This gain is larger than with 5-shot prompts on MMLU, where the best block adds 0.2 to 0.4 points (W1). sBLR, which we select on ARC and DART, also improves the 0-shot accuracy of the 7B model, from 78.9% to 81.2%. At 0.5B, the best block also improves on the default model, from 35.2% to 36.7% with the 0-shot prompt and from 39.1% to 39.8% with the 8-shot prompt. We added these results to Appendix F.2 (Table 27).

Q2. Stability across router training seeds

We repeat the full training pipeline with two additional seeds per backbone. sBLR does not depend on a seed, since it selects its block by exhaustive evaluation on the training set.

ModelDefaultsBLRaBLR, seeds 42 / 43 / 44aBLR, mean ±\pm stdaBLR − sBLR
Qwen2.5-0.5B27.536.736.9 / 36.5 / 35.736.4±0.636.4 \pm 0.6−0.3±0.6-0.3 \pm 0.6
Qwen2.5-1.5B25.643.344.5 / 44.3 / 43.544.1±0.544.1 \pm 0.5+0.8±0.5+0.8 \pm 0.5
Qwen2.5-7B35.658.361.9 / 61.4 / 61.561.6±0.361.6 \pm 0.3+3.3±0.3+3.3 \pm 0.3

aBLR is stable across seeds, with a standard deviation of at most 0.6 points, and every seed stays well above the default model. At 1.5B, aBLR outperforms sBLR on all three seeds, and at 7B its advantage is consistent across seeds (+3.1 to +3.6 points). At 0.5B, aBLR and sBLR are within seed variation of each other. We added these results to Appendix C.2.5 (Table 10).

References

[1] Wei et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022.

[2] Qwen Team. Qwen2.5 Technical Report. 2025.

[3] Li et al. Skip a Layer or Loop It? Learning Program-of-Layers in LLMs. ICML 2026.

[4] Sun et al. Transformer Layers as Painters. AAAI 2025.

Dear Reviewer VSFu,

We would like to thank you once again for your encouraging review, your positive assessment of BLR, and your constructive suggestions for strengthening our experiments.

In response to your review, we provided the following clarifications and additional experiments:

  • Added few-shot MMLU experiments to compare BLR with standard prompting and examine their complementarity.
  • Compared Oracle BLR with pass@277 sampling and added the learned PoLaR predictor as a baseline.
  • Evaluated BLR with chain-of-thought prompting on GSM8K.
  • Repeated aBLR training across multiple seeds to assess the stability of its improvements.
  • Expanded the discussion of related work, including Transformer Layers as Painters.

As we approach the end of the discussion period, we would appreciate hearing whether our responses have addressed your questions. We would be happy to provide any further clarifications.

Thank you for addressing my questions with such detailed experiments. Great work! I'm maintaining my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.