acceptodds
Under review as a conference paper at ICLR 2027

Scaling Where It Matters: Predictive Contraction for Efficient Internal Scaling in LMs

Abstract

Scaling language models with larger networks or more inference trajectories improves performance but significantly increases computational load. This raises an alternative: can extra computation be allocated selectively inside the Transformer, rather than globally, to improve performance with modest overhead? Our core idea is that prediction-relevant refinement is highly non-uniform across layers. Using a fixed logit lens, we observe sharp predictive entropy contraction at a few layers and treat it as an observable signal of strong prediction-relevant refinement. We call such layers predictive concentration layers. Based on this signal, EntroBranch performs efficient localized scaling by selecting the layer and branch width from frozen-backbone calibration, adding prefix-conditioned branches there, and aggregating their outputs with a token-wise gate before resuming single-path computation. Across Qwen3 and Gemma4 on code, mathematics, and multiple-choice tasks, EntroBranch improves overall scores by 33.70% on average over single-path prefix tuning with only 4.58% additional per-token inference FLOPs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language Models, Internal Scaling, Adaptive Computation, Predictive Entropy, Prefix Tuning, Efficient Inference, Conditional Computation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YVXaOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2(+1)
Read review
Paper summary

In their work, the authors present EntroBranch, a layer-selection method based on logit-lens entropy for prefix tuning, combined with an ensemble method that uses a learnable gate within the LLM. The method first calibrates and identifies the layer (on a portion of the training set) that shows the largest drop in predictive entropy, and then attaches multiple prefix tuning adapters to this layer. These adapters are then merged into a single adapter using linear combination. The authors further evaluate EntroBranch on various LLM-based tasks, which is sufficient. However, the baseline selection is quite weak and feels like a mix of token sampling and fine-tuning methods.

The paper is hard to read in general. The authors are not using standardized terms from related works (e.g., referring to adding an adapter as scaling, training trajectories as branches, or scaling methods as sampling methods), without defining them beforehand. The motivation for EntroBranch is unclear in the introduction, and its contribution relative to related work is indistinguishable. The general idea of EntroBranch is interesting. However, the execution of the idea lacks the fundamentals to be considered acceptable.

Critical strengths

S1: Introduction of the EntroBranch method. The general idea of using the entropy of login lens logits seems quite straightforward and interesting.

Critical weaknesses

W1: Weak or underfitted baselines. The authors compare EntroBranch only with single-path prefix tuning, ParScale, and decoding strategies such as temperature sampling and beam search. However, many PEFT methods explicitly address where to add trainable parameters and how to allocate them, and the authors neither discuss nor compare to these. This includes:

  • methods that choose the layers where prompts or prefixes are inserted, such as Late Prompt Tuning (Liu et al., 2022), SPT (Zhu & Tan, 2023) and Adaptive Prefix Tuning (Zhang et al., 2023b);

  • methods that allocate the trainable-parameter budget across layers or modules based on importance, such as AdaLoRA (Zhang et al., 2023a), AutoPEFT (Zhou et al., 2024), IST (Yao et al., 2024) and LISA (Pan et al., 2024);

  • methods that give each layer its own number of parallel, gated modules, which is very close to EntroBranch’s contraction-guided branch width, such as MoLA (Gao et al., 2024) and AlphaLoRA (Qing et al., 2024).

Additionally, comparing sampling and beam search with EntroBranch is not an equal comparison. EntroBranch adds and trains new parameters inside the network, while these methods only change how the single-path model is decoded. The two are orthogonal and could be combined. This also applies to the efficiency comparison. The comparison with LoRA and SMoP (Table 3e) is a step in the right direction, but it covers only Qwen3-0.6B-Base.

I suggest reviewing related work or PEFT benchmarks with similar math and code setups (e.g., Hu et al., 2023; Meng et al., 2024) and comparing EntroBranch in a similar setup. At minimum, it should be compared against the layer-selective prompt and prefix methods above on all evaluated models, which would significantly strengthen the paper.

W2: Lack of novelty. As I mentioned in my previous paragraphs, it is unclear from the paper how the authors distinguish EntroBranch from related work that proposes similar ideas using different heuristics. In addition, the paper lacks a related work section on state-of-the-art ensemble methods.

W3: Paper presentation and readability. The paper is generally hard to read. The introduction is confusing, often introducing terms that were not defined earlier or are difficult to understand from context. It is hard to understand the motivation of EntroBranch from the introduction and related work. Moreover, the tables presenting the results are hard to orient within (multiple subtables with vague subsections is not a good pattern), and the same goes for Figure 2. I suggest adding more detailed explanations to the captions, as this would help readers better understand what is going on in the Figures and Tables. Also, I would consider consolidating Figure 3, as it would free up space for other, more important results.

W4: Formal inconsistencies and missing definitions. Mostly in Section 3 and its relevant figures, I have noticed some formal inconsistencies and missing definitions. On Line 150, the authors define the logit lens logits (zl,tz_{l,t}), but never use them in further equations. In Figure 2, I have noticed that RTR_T or rl,tr_l,t is never mentioned or defined in the text. On Line 215, the authors never define U(j)U^{(j)}. Is it the hidden representation from the previous layer?

I would also like to point out that the trajectories may technically not start from the same point in space, since the authors are adding new, randomly initialized parameters for each trajectory. This should also be addressed in the paper.

Questions for authors

Q1: The authors are using an ensemble of different prefix tuning checkpoints during training. However, it is unclear how the checkpoints differ at the beginning. Are the authors using different seeds for each trajectory, or different prefix initializations?

Q2: In the paragraph starting on line 294, the authors mention scaling methods as beam search or temperature sampling. What do the authors mean by scaling methods? Token sampling methods? Or are the authors using these methods to find the best layer to add the prefix to?

5 comments

W1: Weak or underfitted baselines

EntroBranch is an internal scaling method, rather than a method for redistributing a fixed parameter or computation budget. It introduces additional computation within a selected Transformer layer, using predictive entropy contraction to determine where and how much to scale. This motivates our comparison with inference-time scaling methods such as beam search and temperature sampling, which also improve performance through additional computation, but at the decoding stage rather than within the model. Notably, these baselines require approximately 2× the single-path FLOPs, whereas EntroBranch achieves superior performance with only 4.58% additional FLOPs on average (Table 2).

From the perspective of parameter scaling, Qwen3-0.6B with EntroBranch approaches Qwen3-1.7B single-path performance on shared benchmarks, including HellaSwag (36.07 vs. 36.35) and Math500 (36.40 vs. 37.00), while surpassing it on GSM8K (60.12 vs. 57.24). This further illustrates the effectiveness of localized internal scaling.

We appreciate the suggestion to include additional PEFT baselines, although allocating trainable parameters differs fundamentally from scaling internal computation. We therefore evaluated LPT and MoLA using their official configurations:

MethodParamsHumanEval (+)MBPP (+)
LPT0.792M20.1 (18.9)51.1 (43.7)
MoLA26.66M32.3 (28.1)53.7 (45.8)
EntroBranch2.15M37.2 (32.3)56.3 (48.4)

EntroBranch consistently outperforms both methods, even though MoLA uses 12.4× more trainable parameters. Together with the existing LoRA, SMoP, and parameter-matched prefix-tuning comparisons (Table 3e), these results further support the effectiveness of our approach.

W2: Lack of novelty.

We respectfully disagree with the assessment that EntroBranch lacks novelty or merely replaces existing allocation heuristics. The key distinction is not simply where parameters are placed, but how predictive entropy contraction is used to control additional computation within a frozen Transformer. EntroBranch combines (1) contraction-based layer selection, (2) contraction-guided branch-width allocation, and (3) localized prefix-conditioned parallel transformations with token-wise hidden-state aggregation and shared KV caching. These components together implement a different scaling mechanism from distributing trainable parameters or LoRA experts across layers.

The distinction is also supported by the ablations. In Table 3(d), entropy-guided layer selection outperforms random-layer selection (50.82 versus 48.73 on Qwen3-0.6B-Base). Table 3(a) further examines contraction-guided branch width against alternative widths, supporting selective allocation rather than unconditional expansion. Our additional LPT/MoLA experiments in W1 provide direct empirical comparisons with closely related PEFT approaches.

EntroBranch is also distinct from conventional prediction-level ensembles: it does not combine independently trained checkpoints or complete inference trajectories. Instead, multiple prefix-conditioned transformations of the same incoming hidden state are aggregated at one selected Transformer layer, after which computation returns to a single path. This localized intermediate-state mechanism differs from full-model prediction aggregation in both structure and cost.

We will expand the related-work discussion to make these mechanism-level distinctions explicit.

W3: Paper presentation and readability.

We appreciate the reviewer's suggestions for improving presentation and readability. We clarify the current organization below and outline changes that could make the paper easier to follow.

First, regarding the motivation and overall framework: the Introduction discusses the non-uniformity of predictive entropy contraction across Transformer layers, which motivates the questions of where to scale and how much to scale. Figure 2 summarizes the pipeline from entropy-guided calibration to branch-specific prefix tuning, token-wise aggregation, and shared KV caching. We will improve the accompanying explanation and figure caption to make these connections more accessible.

Second, regarding Figure 3: this figure illustrates that sharp predictive entropy contractions appear consistently across three Qwen3 model scales and provides empirical support for our motivation. We agree that the main-text presentation could be more concise and can move the detailed visualization to the appendix while retaining the key observation in the main text.

Finally, regarding captions and organization: we agree that more descriptive captions and a clearer table organization would improve readability, and we will revise them accordingly. We will also add a dedicated notation table and a brief transformer glossary in the appendix to make the method easier to navigate for readers with different backgrounds.

W4: Formal inconsistencies and missing definitions.

We thank the reviewer for pointing out places where the mathematical notation can be clarified. We address each item below.

First, regarding the logit-lens: The reviewer's claim that the logit lens is introduced but not used is not accurate. As explicitly defined in Lines 150–157, we apply the logit lens to compute (z_{\ell,t}=W_U\mathcal{N}(x_{\ell,t})), from which the top-(k) predictive distribution (\pi_{\ell,t}^{(k)}) and its entropy (H_{\ell,t}^{(k)}) are derived. These entropy values directly determine the contraction score in Eq. (1), which is used to select the scaling layer. Thus, the logit lens is an integral part of our method, rather than an unused concept.

Second, regarding Figure 2: We identified a notation error in the figure. The contraction illustration uses rℓ,tr_{\ell,t}, denoting relative entropy contraction. We will correct these labels and expressions in the final version to match Eqs. (1) and (2). The formal definitions and method described by the equations remain unchanged. For the remaining parts, we have used standard Transformer terminology; therefore, we have not provided specific labels. We will also elaborate them in the appendix.

Third, regarding Line 215: fθjf_{\theta_j} is the branch-specific prefix generator, implemented as a two-layer MLP, and U(j)U^{(j)} is its prefix input. The MLP produces E(j)=fθj(U(j))E^{(j)}=f_{\theta_j}(U^{(j)}), the prefix for branch jj. Thus, U(j)U^{(j)} is not the Transformer hidden state Xℓ⋆−1X_{\ell^\star-1} entering the branched layer. We will explicitly define U(j)U^{(j)} and its role in the revised manuscript.

Finally, regarding branch starting points: All branches receive the same incoming hidden representation Xℓ⋆−1X_{\ell^\star-1}, as stated in Section 3.2. Branch-specific prefixes induce different transformations of this shared input; their initialization does not change the common incoming hidden state. We will restate this distinction for clarity.

Q1

EntroBranch differs from ensembles of independently trained checkpoints or separate inference trajectories in how its branches are constructed and used.

As described in Section 3.2, all branches share the same frozen Transformer backbone and incoming hidden states. They differ only in their branch-specific prefixes generated by independently initialized MLPs. These MLPs are initialized once at the beginning of training and jointly optimized with the token-wise gate in a single training process. Thus, the method does not involve independently trained checkpoints, separate training runs, or trajectory-specific seeds.

Branching occurs only at the selected predictive concentration layer. The branch outputs are then aggregated, and computation resumes along a single path. We will emphasize this localized branching mechanism to distinguish it more clearly from full-model trajectory ensembles.

Q2

Beam search and temperature sampling are independent decoding-time baselines, not components of EntroBranch. As discussed in Section 2.1 (Lines 110–119), they explore alternative decoding candidates at inference time, whereas EntroBranch determines where to scale using predictive entropy contraction and allocates additional computation within the selected Transformer layer.

In particular, neither beam search nor temperature sampling is used to select the prefix insertion layer. Layer selection is performed by EntroBranch's entropy-based mechanism, independently of the decoding baselines. This separation is part of both the method and the experimental setup.

We will add an explicit statement in Section 4.2 distinguishing the role of these comparison baselines from EntroBranch's layer-selection procedure.

Reviewer_Az54OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EntroBranch, an approach to efficient internal scaling that allocates additional computation to Transformer layers exhibiting sharp predictive entropy contraction. The method identifies a predictive concentration layer and determines the branch width using a fixed logit lens on a frozen backbone. It then introduces multiple prefix-conditioned branches at the selected layer, aggregates their outputs using a token-wise gate, and shares the content KV cache to reduce memory overhead.

The authors evaluate EntroBranch on Qwen3 and Gemma4 models across mathematics, code generation, and multiple-choice benchmarks. The reported results show an average relative improvement of 33.70% over single-path prefix tuning with only 4.58% additional estimated per-token FLOPs. The paper also includes extensive ablation studies on branch width, layer selection, aggregation, and KV caching, along with comparisons against other fine-tuning and inference-scaling methods.

Critical strengths

1. Interesting and intuitive approach to selective internal scaling.

The central idea of allocating additional computation to layers exhibiting strong predictive entropy contraction is interesting and well motivated. The layer-wise entropy patterns in Figure 3 provide an intuitive starting point for the proposed method. Although the observation itself is relatively simple, connecting it to selective internal branching is a useful perspective on improving model efficiency.

2. Clear presentation and extensive empirical ablations.

The paper is generally well written and easy to follow. I particularly appreciate the comprehensive ablation studies in Table 3, which examine branch width, branch location, multi-layer branching, aggregation mechanisms, and KV-cache design. The comparisons with other fine-tuning methods and the additional layer-wise analyses help substantiate the proposed design choices. The reported performance improvements under modest computational overhead are also encouraging.

Critical weaknesses

1. Limited evidence of effectiveness on post-trained and reasoning-enabled models.

My main concern is the generalizability of the proposed method beyond base models. Three of the four evaluated models are Qwen3-Base variants. Although the paper includes Gemma4-E4B-it, it remains unclear whether the approach is effective on modern post-trained or reasoning-enabled Qwen3 models, particularly with thinking mode enabled.

This distinction matters because post-training can substantially alter intermediate representations, output distributions, and the interpretability of intermediate logits. Consequently, the predictive entropy contraction patterns used to select branching layers may behave differently after post-training. While Table 3(f) provides a preliminary analysis of entropy contraction after supervised fine-tuning, it does not establish that EntroBranch maintains its performance gains in these settings. Given the practical importance of reasoning-enabled models, I view this as a significant limitation of the current evaluation.

2. The practical efficiency and general applicability claims need further validation.

The computational efficiency results are promising, but the primary comparison relies on estimated per-token FLOPs. The paper does report throughput in Section 4.4 and VRAM usage in Section 4.7, which I appreciate. However, these measurements cover limited experimental configurations. A more detailed evaluation of end-to-end inference latency, prefill/decode overhead, and memory consumption under different batch sizes and context lengths would strengthen the practical efficiency claims.

Furthermore, although multiple-choice benchmarks provide some evidence beyond the code and mathematics training domains, evaluation on more diverse open-ended language tasks would help establish whether the method offers broader benefits beyond the current task settings.

Questions for authors

Q1. Potentially incorrect dataset citations in Table 9.

I noticed two apparent citation mismatches in Table 9 (page 30):

Numina 1.5 Olympiads (Ding et al., 2026): The corresponding reference is "AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMs," which appears unrelated to the Numina dataset.

MathInstruct Hard (Hu et al., 2026): The corresponding reference is "Fine-Grained Data Ordering Improves Fine-Tuning for Large Language Models," which does not appear to identify the original dataset source.

Could the authors verify these references and provide the exact dataset URLs, versions, and subsets used? Accurate dataset provenance is important for reproducing the training setup.

Q2. Evaluation protocol and baseline for MMLU-Pro (and other choice tasks).

The single-path MMLU-Pro scores are close to chance for a 10-option task (Qwen3-1.7B-Base: 14.56; Qwen3-4B-Base: 17.59). EntroBranch roughly doubles them (30.93 and 39.51), even though it is trained only on code and mathematics data. For reference, the Qwen3 technical report lists Qwen3-4B-Base at ~50.6 on MMLU-Pro under 5-shot CoT evaluation. I understand that protocols differ, but the gap makes the baseline difficult to interpret. Could the authors explain why the single-path scores are so low?

https://arxiv.org/abs/2505.09388

8 comments

We thank the reviewer for the constructive feedback on model coverage and evaluation. In this response, we address W1 and Q2 with additional experiments and protocol details.

W1 Limited evidence of effectiveness on post-trained and reasoning-enabled models.

Our training data contain no explicit thinking traces, so the main experiments use non-thinking inference consistently across methods. Prior work also reports that explicit reasoning does not always benefit small models [5].

We also conducted additional experiments using 0.6B model with thinking mode enabled and a maximum reasoning length of 256 tokens.

BenchmarkSingle PathEntroBranch
HumanEval25.00%31.71%
HumanEval+21.95%27.44%
MBPP38.62%56.35%
MBPP+32.01%48.41%

EntroBranch outperforms single-path prefix tuning across all four evaluations by 5.49–17.73 percentage points. These results support its effectiveness under reasoning-enabled inference.

We also applied both methods to Qwen3-0.6B-Instruct:

BenchmarkSingle PathEntroBranch
HumanEval17.68%25.61%
HumanEval+16.46%23.17%
MBPP29.63%33.33%
MBPP+25.66%29.10%

These results extend the evidence to the tested instruction-tuned model. Both methods achieve lower absolute scores when adapted from the Instruct model than from the Base model. Wang et al. report a similar observation for geometric reasoning with LLaMA3-8B [6], suggesting that behavioral biases from prior instruction tuning may affect subsequent adaptation.

Q2 Evaluation protocol and baseline for MMLU-Pro (and other choice tasks).

All prefix-tuning baselines use the same training data. Since the exact evaluation framework used in the Qwen3 paper is unavailable, we used EvalPlus prompts for code evaluation and a 5-shot non-reasoning setting for mathematical reasoning. For MMLU-Pro and other multiple-choice tasks, we scored candidate answers using their logit probabilities.

For reference, we additionally evaluated the untuned Qwen3-0.6B-Base and Qwen3-1.7B-Base models under the same protocol. On HumanEval (+), Qwen3-0.6B-Base achieves 24.4 (22.6), while Qwen3-1.7B-Base achieves 62.2 (54.3). On MBPP (+), they achieve 24.1 (20.9) and 43.9 (37.0), respectively. On GSM8K, their scores are 58.38 and 71.11, respectively, while Qwen3-1.7B-Base achieves 28.88 on MMLU-Pro.

Single-path tuning improves coding performance but reduces performance on GSM8K and MMLU-Pro. Task-specific specialization and reduced generation diversity are possible explanations: MMLU-Pro and GSM8K covers diverse knowledge domains rather than complex reasoning ability [1], [2]. EntroBranch's multiple paths help mitigate these effects.

W2 The practical efficiency and general applicability claims need further validation.

We conducted additional experiments to evaluate practical inference efficiency and generalization beyond the training domains.

1. Inference Efficiency Across Batch Sizes

BatchThroughput (tokens/s) EB / SingleEnd-to-End Time (s) EB / Single
16230.71 / 240.770.814 / 1.050
32373.44 / 382.670.426 / 0.640
1281331.14 / 1369.810.129 / 0.187

Across these batch sizes, EntroBranch reduces average task completion time by 22.5–33.4%, while token throughput decreases by 2.4–4.2%. This indicates that under different batch sizes, the generation speed of EntroBranch is basically the same as that of the single path method. Moreover, since EntroBranch can generate more concise content, its actual generation efficiency is higher than that of the single path method.

2. Efficiency Across Context Lengths

We further evaluated inference efficiency under varying input context lengths.

Input TokensSingle (tok/s)EntroBranch (tok/s)Relative Time (EB / Single)
256418451310.815×
512419547510.883×
1024315432740.963×
2048203120251.003×
4096118411461.033×

EntroBranch achieves up to 18.5% lower inference time at shorter context lengths, with a 3.3% overhead at 4096 tokens. Together with the comparable VRAM consumption reported in Section 4.7, these results support its practical efficiency across the tested settings.

3. Prefill and Decode Efficiency Across Context Lengths

To further address the reviewer's concerns about practical inference efficiency, we additionally measured prefill and decode latency separately across input context lengths ranging from 256 to 4096 tokens.

Prefill efficiency. EntroBranch achieves lower prefill latency across all evaluated context lengths, with reductions ranging from 1.07% to 16.87%.

Context TokensSingle Path (ms/batch)EntroBranch (ms/batch)Relative Overhead
256130.01108.07-16.87%
512240.97203.46-15.56%
1024449.65400.71-10.89%
2048905.65846.36-6.55%
40961940.421919.62-1.07%

Decode efficiency. EntroBranch also achieves lower decode latency for shorter contexts (256–1024 tokens). At longer contexts, the additional decode overhead remains limited to 0.27% at 2048 tokens and 3.29% at 4096 tokens.

Context TokensSingle Path (ms/step)EntroBranch (ms/step)Relative OverheadSingle Path (127 steps, ms)EntroBranch (127 steps, ms)
25615.29512.473-18.45%1942.451584.01
51215.25513.472-11.69%1937.341710.95
102420.29219.548-3.67%2577.102482.62
204831.51731.602+0.27%4002.644013.47
409654.06655.847+3.29%6866.367092.52

These measurements complement our FLOPs-based analysis by directly characterizing the runtime cost of localized branching in both inference phases. While the latency advantage diminishes as context length increases, the results demonstrate that EntroBranch introduces only modest decode overhead at the longest tested contexts.

4. General Applicability to Open-Ended Language Tasks

To evaluate generalization beyond code and mathematics, we additionally evaluated EntroBranch on NQ-Open, an open-domain question-answering benchmark requiring freely generated answers rather than multiple-choice selection.

Model: Qwen3-0.6B-BaseNQ-Open Accuracy
Single Path5.62%
EntroBranch7.26%
Relative Improvement+29.18%

We will incorporate these phase-specific measurements into the revised manuscript to provide a more complete assessment of practical inference efficiency.

Q1 Potentially incorrect dataset citations in Table 9.

Thank you for identifying these citation inconsistencies. We have verified the original dataset sources and will correct the references in Table 9.

Specifically, NuminaMath-1.5 Olympiads originates from the Project Numina dataset (Li et al., 2024), available at https://huggingface.co/datasets/project-numina/NuminaMath-1.5. The relevant source category is olympiads in the training split.

MathInstruct was originally introduced by Yue et al. (2024) in MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning, and is available at https://huggingface.co/datasets/TIGER-Lab/MathInstruct.

We thank the reviewer again for highlighting these issues.

I appreciate the additional clarification and experiments. However, modern LLMs are increasingly dominated by reasoning-enabled (thinking) models. Without convincing experiments demonstrating the effectiveness of the proposed method under realistic reasoning settings, I remain unconvinced of its practical applicability. In particular, a thinking budget of only 256 tokens is relatively small and does not adequately reflect typical reasoning workloads. I would be much more interested in results with substantially larger thinking budgets, such as 32K tokens, as well as evaluations under longer context lengths. Furthermore, I question why relatively old models such as LLaMA 3 are being used as supporting evidence, rather than more recent reasoning-capable models. For example, experiments using Qwen3.5 models, with thinking mode enabled and a sufficiently large reasoning budget, would provide much stronger evidence of the method's effectiveness. If the authors can demonstrate consistent improvements under such realistic reasoning conditions, I would be willing to reconsider and increase my score.

Reviewer_a235OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EntroBranch uses a logit lens on a frozen model to pick the layer with the largest drop in predictive entropy. At that layer it runs P prefix-tuned copies of the frozen layer and mixes their outputs with a token-wise gate; all other layers run a single path. The paper reports a 33.70% average relative gain over single-path prefix tuning with 4.58% extra FLOPs on Qwen3 and Gemma4, and gives a theorem on when reweighting the branches lowers the loss.

Critical strengths
  1. Adding computation at a single chosen layer is a simple and cheap idea, with little extra FLOPs, latency, or memory.

  2. The layer sweep (Tables 10–11) shows that contraction correlates with downstream score across layers (Spearman 0.72 and 0.82).

Critical weaknesses
  1. The main baseline looks broken, and the untuned base model is not reported.
  • Single-path prefix tuning scores far below the Qwen3 technical report's numbers for the same untuned base models. GSM8K: 39.5 vs. 59.6 (0.6B) and 57.2 vs. 75.4 (1.7B). MMLU-Pro: 14.6 vs. 36.8 (1.7B) and 17.6 vs. 50.6 (4B).
  • EntroBranch lands close to the untuned numbers (GSM8K 60.1 and 76.4), so the reported gain may mostly measure how much single-path tuning hurts the model.
  • Beam search and sampling run on top of this weakened model. On choice tasks, which are scored by option likelihood, they give exactly the single-path scores.
  • A frozen, untuned baseline under the same evaluation protocol is needed.
  1. The theory does not support the method.
  • Theorem 1 is the standard descent lemma: if a feasible direction lowers the loss, a small step lowers it. It says nothing about which layer to pick, how many branches to use, or why branching beats a single path, as App. A.2 admits.
  • The free per-token reweighting it relies on needs the label, so it is an oracle.
  • The empirical check is weak. The oracle loss drop is 6.6e-7 against a training loss of about 0.25, and the branch direction captures only 0.45% of the gradient energy.
  • The shared KV cache is exact by construction, because all branches share the same input and weights. Yet the proof promised at the start of Appendix A is missing.
  1. The evidence for the design choices is within noise.
  • All results are single runs on small test sets (e.g., AMC has 83 problems).
Questions for authors
  1. What do the untuned base models score under your protocol, and why does single-path prefix tuning lose about 20 points on GSM8K and MMLU-Pro?

  2. Can you report mean and standard deviation over several training seeds for Table 2 and the width and layer ablations?

  3. Can you apply beam search and sampling to the base model and to EntroBranch, not only to the single-path model?

6 comments

We thank the reviewer for the careful assessment and constructive suggestions. In this response, we address W1 and Q1 concerning the baselines and evaluation protocol.

W1 The main baseline looks broken, and the untuned base model is not reported.

To ensure the fairness of the comparison, all the baseline methods were trained using the same training data to guarantee fairness, rather than using untrained base models. Since the exact evaluation framework used in the Qwen3 technical report is unavailable, we used EvalPlus prompts for code evaluation, a 5-shot non-reasoning setting for mathematical reasoning, and logit-based scoring for multiple-choice questions.

We additionally evaluated the untuned Qwen3-0.6B-Base and Qwen3-1.7B-Base models under the same protocol. On HumanEval (+), Qwen3-0.6B-Base achieves 24.4 (22.6), while Qwen3-1.7B-Base achieves 62.2 (54.3). On MBPP (+), their scores are 24.1 (20.9) and 43.9 (37.0), respectively. On GSM8K, the two models achieve 58.38 and 71.11, respectively, while Qwen3-1.7B-Base achieves 28.88 on MMLU-Pro.

SFT can compromise generalization to other tasks, consistent with prior studies on task-specific adaptation [1], [2], [3]. However, EntroBranch's gains cannot be attributed merely to recovering this degradation. On Qwen3-0.6B-Base, the average code score increases from 23.00 (untuned) to 24.95 (single-path) and further to 43.55 (EntroBranch), demonstrating substantial improvements beyond the untuned model. Meanwhile, EntroBranch achieves 60.12 on GSM8K, exceeding both the untuned model (58.38) and single-path tuning (39.50). These results suggest that EntroBranch achieves stronger adaptation to challenging code tasks while better preserving cross-task performance.

Furthermore, the SMoP comparison in Section 4.6 (Table 3e) indicates that prompt diversity alone cannot fully explain these gains. Although SMoP introduces multiple prompts with a routing mechanism, EntroBranch achieves higher math performance (48.26 vs. 45.56), choice performance (60.64 vs. 53.05) and code performance (43.55 vs. 41.88) with the same proportion of trainable parameters (0.36%). This supports the value of localized internal branching and token-wise aggregation beyond simply introducing diverse prompts.

Q1 What do the untuned base models score under your protocol, and why does single-path prefix tuning lose about 20 points on GSM8K and MMLU-Pro?

As reported in W1, we additionally evaluated the untuned Qwen3 base models under the same protocol.

Previous research has reported that supervised fine-tuning can improve target-domain performance while compromising generalization [3]. Similarly, optimizing a single adaptation path toward the training distribution may reduce generation diversity. This could help explain why coding performance improves while performance on GSM8K and MMLU-Pro declines.

EntroBranch uses multiple adaptation paths and achieves better performance than single-path tuning. The SMoP comparison in Section 4.6 is consistent with the potential benefit of adaptation diversity: increasing prompt diversity provides some scaling benefits, but remains below EntroBranch.

W2 The theory does not support the method.

We agree that Theorem 1 alone does not establish the optimality of layer selection, branch width, or superiority over single-path adaptation. As mentioned in Appendix A.2, our theoretical analysis and empirical evaluation are designed to provide complementary evidence for EntroBranch, rather than an end-to-end optimality guarantee.

  1. Connecting theory with empirical evidence. Our analysis establishes three complementary aspects: Branching feasibility (Appendices A.1–A.3): We characterize the reachable representation subspace, establish sufficient conditions for local loss reduction, and analyze branch-subspace conditioning. These results explain how multiple branches can introduce useful correction directions. Allocation principles (Appendix A.4): Entropy contraction characterizes the reduction in effective predictive breadth, motivating intervention at strongly contractive layers. Meanwhile, rank-limited effective parallelism explains why increasing branch width need not yield proportional benefits. Empirical validation (Sections 4.5 and 4.8): The contraction-selected layers outperform the tested alternatives, with positive correlations between contraction and downstream performance (ρ = 0.7182 and 0.8167). Branch-width ablations show diminishing returns beyond the calibrated width, while gradient-alignment and training-loss analyses provide complementary evidence for the usefulness of the learned branch representations. Given the high-dimensional, nonlinear nature of LLMs, establishing a comprehensive theoretical guarantee for layer selection, branch width, and end-to-end performance would require substantially stronger assumptions. We therefore adopt a theory-guided, empirically validated approach: theoretical analysis motivates the mechanism and allocation principles, while experiments evaluate whether these principles translate into practical improvements. We will clarify this distinction in the revision.

  2. Oracle diagnostic and gradient alignment. We agree that the oracle correction requires supervised gradients. However, it is used exclusively as a post-hoc diagnostic, not during training or inference. Its small absolute loss reduction is not intended to explain the overall training-loss improvement. Although the learned rank-one branch subspace captures only 0.452% of gradient energy, this represents a 4.63× enrichment over an equal-rank random subspace (0.098%). This relative comparison provides evidence of non-random alignment, while the actual performance improvements are established independently by the training-loss comparisons and downstream evaluations.

  3. KV-cache exactness. We agree that content-KV sharing is exact by construction. This property is already explained in Section 3.3. Since all branches share the same input hidden states XX and frozen projection matrices, their content keys and values are identical by construction: Kc(j)=XWK=KcK_c^{(j)} = XW_K = K_c and Vc(j)=XWV=VcV_c^{(j)} = XW_V = V_c, while prefix K/V remain branch-specific. We will explicitly add this derivation to Appendix A in the revised version.

W3 The evidence for the design choices is within noise.

Except in the temperature-sampling experiments, we used a temperature of 0 and evaluated the full test sets. This removes variation from evaluation. Most benchmarks are much larger than AMC, including GSM8K (1,319 questions), MBPP+ (384 questions), and MMLU-Pro (12,032 questions).

To further assess sampling variability on AMC, we evaluated EntroBranch at T=0.7T=0.7 over three runs. The mean scores for the 1.7B and 4B models were 26.51 and 31.33, respectively, compared with 20.48 and 30.12 for single-path temperature sampling. We also report results across random training seeds in Q2 below.

Q2 Can you report mean and standard deviation over several training seeds for Table 2 and the width and layer ablations?

Table 2: We trained and evaluated the 0.6B model using three random seeds, following the training configuration for EntroBranch as described in the paper, and report the mean ± standard deviation below.

DatasetBasePlus
HE36.80 ± 0.6931.70 ± 1.04
MBPP56.40 ± 0.6648.77 ± 0.64

Width Ablation (P = 4)

DatasetBasePlus
HE35.17 ± 1.2929.67 ± 2.47
MBPP54.83 ± 1.0647.07 ± 0.46

Layer Ablation (L = 17) For the layer ablation, we fixed the branch layer at L=17L=17 to assess seed variation for a given configuration. This separates training variability from variation due to layer selection, but does not quantify variability across randomly selected layers.

DatasetBasePlus
HE34.57 ± 1.9929.67 ± 1.99
MBPP55.73 ± 0.9047.73 ± 0.58

Q3 Can you apply beam search and sampling to the base model and to EntroBranch, not only to the single-path model?

Beam search (Beam size=2). For the 0.6B model, the untuned base achieves 33.5 (29.3) on HumanEval (+) and 24.9 (21.7) on MBPP (+) with beam search, while EntroBranch achieves 43.3 (37.2) and 58.5 (48.9), respectively. The untuned model's beam-search scores are also below the reported greedy-decoding results of EntroBranch.

Temperature sampling. At T=0.7T=0.7, the untuned 0.6B model achieves pass@2 scores of 32.9 (29.9) on HumanEval (+) and 32.8 (29.1) on MBPP (+). For 0.6B EntroBranch, temperature sampling yields 39.0 (33.5) on HumanEval (+) and 61.1 (53.4) on MBPP (+). Prior studies report that supervised fine-tuning can reduce generation diversity, which may explain why untuned models can outperform the single-path baseline under sampling [3], [4].

We thank the reviewer again for highlighting these issues.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.