- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
In their work, the authors present EntroBranch, a layer-selection method based on logit-lens entropy for prefix tuning, combined with an ensemble method that uses a learnable gate within the LLM. The method first calibrates and identifies the layer (on a portion of the training set) that shows the largest drop in predictive entropy, and then attaches multiple prefix tuning adapters to this layer. These adapters are then merged into a single adapter using linear combination. The authors further evaluate EntroBranch on various LLM-based tasks, which is sufficient. However, the baseline selection is quite weak and feels like a mix of token sampling and fine-tuning methods.
The paper is hard to read in general. The authors are not using standardized terms from related works (e.g., referring to adding an adapter as scaling, training trajectories as branches, or scaling methods as sampling methods), without defining them beforehand. The motivation for EntroBranch is unclear in the introduction, and its contribution relative to related work is indistinguishable. The general idea of EntroBranch is interesting. However, the execution of the idea lacks the fundamentals to be considered acceptable.
Critical strengths
S1: Introduction of the EntroBranch method. The general idea of using the entropy of login lens logits seems quite straightforward and interesting.
Critical weaknesses
W1: Weak or underfitted baselines. The authors compare EntroBranch only with single-path prefix tuning, ParScale, and decoding strategies such as temperature sampling and beam search. However, many PEFT methods explicitly address where to add trainable parameters and how to allocate them, and the authors neither discuss nor compare to these. This includes:
-
methods that choose the layers where prompts or prefixes are inserted, such as Late Prompt Tuning (Liu et al., 2022), SPT (Zhu & Tan, 2023) and Adaptive Prefix Tuning (Zhang et al., 2023b);
-
methods that allocate the trainable-parameter budget across layers or modules based on importance, such as AdaLoRA (Zhang et al., 2023a), AutoPEFT (Zhou et al., 2024), IST (Yao et al., 2024) and LISA (Pan et al., 2024);
-
methods that give each layer its own number of parallel, gated modules, which is very close to EntroBranch’s contraction-guided branch width, such as MoLA (Gao et al., 2024) and AlphaLoRA (Qing et al., 2024).
Additionally, comparing sampling and beam search with EntroBranch is not an equal comparison. EntroBranch adds and trains new parameters inside the network, while these methods only change how the single-path model is decoded. The two are orthogonal and could be combined. This also applies to the efficiency comparison. The comparison with LoRA and SMoP (Table 3e) is a step in the right direction, but it covers only Qwen3-0.6B-Base.
I suggest reviewing related work or PEFT benchmarks with similar math and code setups (e.g., Hu et al., 2023; Meng et al., 2024) and comparing EntroBranch in a similar setup. At minimum, it should be compared against the layer-selective prompt and prefix methods above on all evaluated models, which would significantly strengthen the paper.
W2: Lack of novelty. As I mentioned in my previous paragraphs, it is unclear from the paper how the authors distinguish EntroBranch from related work that proposes similar ideas using different heuristics. In addition, the paper lacks a related work section on state-of-the-art ensemble methods.
W3: Paper presentation and readability. The paper is generally hard to read. The introduction is confusing, often introducing terms that were not defined earlier or are difficult to understand from context. It is hard to understand the motivation of EntroBranch from the introduction and related work. Moreover, the tables presenting the results are hard to orient within (multiple subtables with vague subsections is not a good pattern), and the same goes for Figure 2. I suggest adding more detailed explanations to the captions, as this would help readers better understand what is going on in the Figures and Tables. Also, I would consider consolidating Figure 3, as it would free up space for other, more important results.
W4: Formal inconsistencies and missing definitions. Mostly in Section 3 and its relevant figures, I have noticed some formal inconsistencies and missing definitions. On Line 150, the authors define the logit lens logits (), but never use them in further equations. In Figure 2, I have noticed that or is never mentioned or defined in the text. On Line 215, the authors never define . Is it the hidden representation from the previous layer?
I would also like to point out that the trajectories may technically not start from the same point in space, since the authors are adding new, randomly initialized parameters for each trajectory. This should also be addressed in the paper.
Questions for authors
Q1: The authors are using an ensemble of different prefix tuning checkpoints during training. However, it is unclear how the checkpoints differ at the beginning. Are the authors using different seeds for each trajectory, or different prefix initializations?
Q2: In the paragraph starting on line 294, the authors mention scaling methods as beam search or temperature sampling. What do the authors mean by scaling methods? Token sampling methods? Or are the authors using these methods to find the best layer to add the prefix to?
5 comments
W3: Paper presentation and readability.
We appreciate the reviewer's suggestions for improving presentation and readability. We clarify the current organization below and outline changes that could make the paper easier to follow.
First, regarding the motivation and overall framework: the Introduction discusses the non-uniformity of predictive entropy contraction across Transformer layers, which motivates the questions of where to scale and how much to scale. Figure 2 summarizes the pipeline from entropy-guided calibration to branch-specific prefix tuning, token-wise aggregation, and shared KV caching. We will improve the accompanying explanation and figure caption to make these connections more accessible.
Second, regarding Figure 3: this figure illustrates that sharp predictive entropy contractions appear consistently across three Qwen3 model scales and provides empirical support for our motivation. We agree that the main-text presentation could be more concise and can move the detailed visualization to the appendix while retaining the key observation in the main text.
Finally, regarding captions and organization: we agree that more descriptive captions and a clearer table organization would improve readability, and we will revise them accordingly. We will also add a dedicated notation table and a brief transformer glossary in the appendix to make the method easier to navigate for readers with different backgrounds.
W4: Formal inconsistencies and missing definitions.
We thank the reviewer for pointing out places where the mathematical notation can be clarified. We address each item below.
First, regarding the logit-lens: The reviewer's claim that the logit lens is introduced but not used is not accurate. As explicitly defined in Lines 150–157, we apply the logit lens to compute (z_{\ell,t}=W_U\mathcal{N}(x_{\ell,t})), from which the top-(k) predictive distribution (\pi_{\ell,t}^{(k)}) and its entropy (H_{\ell,t}^{(k)}) are derived. These entropy values directly determine the contraction score in Eq. (1), which is used to select the scaling layer. Thus, the logit lens is an integral part of our method, rather than an unused concept.
Second, regarding Figure 2: We identified a notation error in the figure. The contraction illustration uses , denoting relative entropy contraction. We will correct these labels and expressions in the final version to match Eqs. (1) and (2). The formal definitions and method described by the equations remain unchanged. For the remaining parts, we have used standard Transformer terminology; therefore, we have not provided specific labels. We will also elaborate them in the appendix.
Third, regarding Line 215: is the branch-specific prefix generator, implemented as a two-layer MLP, and is its prefix input. The MLP produces , the prefix for branch . Thus, is not the Transformer hidden state entering the branched layer. We will explicitly define and its role in the revised manuscript.
Finally, regarding branch starting points: All branches receive the same incoming hidden representation , as stated in Section 3.2. Branch-specific prefixes induce different transformations of this shared input; their initialization does not change the common incoming hidden state. We will restate this distinction for clarity.
Q1
EntroBranch differs from ensembles of independently trained checkpoints or separate inference trajectories in how its branches are constructed and used.
As described in Section 3.2, all branches share the same frozen Transformer backbone and incoming hidden states. They differ only in their branch-specific prefixes generated by independently initialized MLPs. These MLPs are initialized once at the beginning of training and jointly optimized with the token-wise gate in a single training process. Thus, the method does not involve independently trained checkpoints, separate training runs, or trajectory-specific seeds.
Branching occurs only at the selected predictive concentration layer. The branch outputs are then aggregated, and computation resumes along a single path. We will emphasize this localized branching mechanism to distinguish it more clearly from full-model trajectory ensembles.
Q2
Beam search and temperature sampling are independent decoding-time baselines, not components of EntroBranch. As discussed in Section 2.1 (Lines 110–119), they explore alternative decoding candidates at inference time, whereas EntroBranch determines where to scale using predictive entropy contraction and allocates additional computation within the selected Transformer layer.
In particular, neither beam search nor temperature sampling is used to select the prefix insertion layer. Layer selection is performed by EntroBranch's entropy-based mechanism, independently of the decoding baselines. This separation is part of both the method and the experimental setup.
We will add an explicit statement in Section 4.2 distinguishing the role of these comparison baselines from EntroBranch's layer-selection procedure.
W1: Weak or underfitted baselines
EntroBranch is an internal scaling method, rather than a method for redistributing a fixed parameter or computation budget. It introduces additional computation within a selected Transformer layer, using predictive entropy contraction to determine where and how much to scale. This motivates our comparison with inference-time scaling methods such as beam search and temperature sampling, which also improve performance through additional computation, but at the decoding stage rather than within the model. Notably, these baselines require approximately 2× the single-path FLOPs, whereas EntroBranch achieves superior performance with only 4.58% additional FLOPs on average (Table 2).
From the perspective of parameter scaling, Qwen3-0.6B with EntroBranch approaches Qwen3-1.7B single-path performance on shared benchmarks, including HellaSwag (36.07 vs. 36.35) and Math500 (36.40 vs. 37.00), while surpassing it on GSM8K (60.12 vs. 57.24). This further illustrates the effectiveness of localized internal scaling.
We appreciate the suggestion to include additional PEFT baselines, although allocating trainable parameters differs fundamentally from scaling internal computation. We therefore evaluated LPT and MoLA using their official configurations:
EntroBranch consistently outperforms both methods, even though MoLA uses 12.4× more trainable parameters. Together with the existing LoRA, SMoP, and parameter-matched prefix-tuning comparisons (Table 3e), these results further support the effectiveness of our approach.
W2: Lack of novelty.
We respectfully disagree with the assessment that EntroBranch lacks novelty or merely replaces existing allocation heuristics. The key distinction is not simply where parameters are placed, but how predictive entropy contraction is used to control additional computation within a frozen Transformer. EntroBranch combines (1) contraction-based layer selection, (2) contraction-guided branch-width allocation, and (3) localized prefix-conditioned parallel transformations with token-wise hidden-state aggregation and shared KV caching. These components together implement a different scaling mechanism from distributing trainable parameters or LoRA experts across layers.
The distinction is also supported by the ablations. In Table 3(d), entropy-guided layer selection outperforms random-layer selection (50.82 versus 48.73 on Qwen3-0.6B-Base). Table 3(a) further examines contraction-guided branch width against alternative widths, supporting selective allocation rather than unconditional expansion. Our additional LPT/MoLA experiments in W1 provide direct empirical comparisons with closely related PEFT approaches.
EntroBranch is also distinct from conventional prediction-level ensembles: it does not combine independently trained checkpoints or complete inference trajectories. Instead, multiple prefix-conditioned transformations of the same incoming hidden state are aggregated at one selected Transformer layer, after which computation returns to a single path. This localized intermediate-state mechanism differs from full-model prediction aggregation in both structure and cost.
We will expand the related-work discussion to make these mechanism-level distinctions explicit.