- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SELFMORPH, which adapts the internal execution of a frozen LLM by selecting among BASE, REPEAT(increase depth) and BRANCH(increase depth). A lightweight self-supervised morphing predictor is trained to generate the optimal "computation shape" based on the input prompt's hidden trajectories. Evaluations across diverse benchmarks demonstrates consistent accuracy gains or compute-efficiency trade-offs over frozen backbones.
Critical strengths
- The problem is well motivated. Treating serial loop and local branching as input-dependent execution choice is a useful extension of frozen-model inference. The backbone remains unchanged, and the selector is pre-trained to improve the computation of inference.
- The evaluation includes multiple model scales and unseen tasks across multiple benchmarks, moreover provides representation ablations and practical implementation analysis.
Critical weaknesses
- Lack of direct comparison with highly relevant prior work[1,2]. This is the most critical weakness of this paper. The core philosophy of SelfMorph is reconfiguring frozen LLM layers via dynamic branch or looping via a lightweight predictor to map inputs to custom execution paths, which is similar with Dr. LLM[1] and PoLar[2]. While the authors acknowledge PoLar briefly in paper, they omit any direct empirical comparison with both works.
[1] Heakl, Ahmed, Martin Gubri, Salman Khan, Sangdoo Yun, and Seong Joon Oh. "Dr. llm: Dynamic layer routing in llms." In International Conference on Learning Representations, vol. 2026, pp. 73320-73337. 2026.
[2] Li, Ziyue, Yang Li, and Tianyi Zhou. "Skip a Layer or Loop It? Learning Program-of-Layers in LLMs." arXiv preprint arXiv:2606.06574 (2026).
Questions for authors
-
Please provide direct performance comparisons between SelfMorph, Dr.LLM, and PoLar on the same benchmarks.
-
Please provide an in-depth methodological comparison with Dr.LLM and PoLar, clearly identifying SelfMorph’s novel contributions relative to these methods.
-
Please provide controlled ablation studies that isolate the contributions of SelfMorph’s distinguishing components, including cross-layer hidden-state trajectories, the BRANCH operation, and input-dependent selection among BASE, REPEAT, and BRANCH together with the intervention span and intensity.
5 comments
Response to Reviewer tRXJ [2/2]
Q3: Controlled ablation studies. (Question 3)
A3: We describe the controlled ablation studies below. Each ablation changes only one component while keeping all other settings and hyperparameters fixed.
Cross-layer hidden-state trajectories. We compare a single hidden state with trajectories formed from different numbers of observed layers. Single hidden state has the worst performance. This indicates that cross-layer hidden-state trajectories provide richer information for morphology selection.
| Trajectory length | Qwen3.5-27B | Qwen3.5-4B | Gemma4-12B |
|---|---|---|---|
| 1 | 0.5940 | 0.4845 | 0.5286 |
| 3 | 0.5964 | 0.4905 | 0.5333 |
| 5 | 0.5952 | 0.4952 | 0.5345 |
| 10 | 0.6083 | 0.4964 | 0.5369 |
BRANCH operation. We remove BRANCH and train the selector. Performance drops on all three backbones, showing that BRANCH provides gains that are not fully captured by BASE and REPEAT alone.
| Pool (spans × intensities) | Qwen3.5-27B | Qwen3.5-4B | Gemma4-12B |
|---|---|---|---|
| BASE + REPEAT (4 × 3) | 0.5964 | 0.4810 | 0.5321 |
| BASE + REPEAT (4 × 3) + BRANCH (4 × 3) | 0.6083 | 0.4964 | 0.5369 |
Input-dependent selection. We evaluate it at two levels. First, does per-input selection outperform a fixed strategy? BenchPrior selects the best fixed action for each benchmark, while SelfMorph outperforms BenchPrior in every setting. This supports the value of selecting an action for each input. Second, do span and intensity adaptation provide additional gains? The table below shows their improvement.
| Pool (spans × intensities) | Qwen3.5-27B | Qwen3.5-4B | Gemma4-12B |
|---|---|---|---|
| BASE + REPEAT (1 × 1) + BRANCH (1 × 1) | 0.5786 | 0.4738 | 0.531 |
| BASE + REPEAT (4 × 1) + BRANCH (4 × 1) | 0.5869 | 0.4833 | 0.5369 |
| BASE + REPEAT (4 × 3) + BRANCH (4 × 3) | 0.6083 | 0.4964 | 0.5369 |
Thank you for the additional experiments, clarifying the comparisons with prior work and further ablation. These responses have addressed my questions. However, considering the novelty relative to prior work, together with overall strengths and limitations, I maintain my original rating.
Response to Reviewer tRXJ [1/2]
We sincerely thank you for the constructive comments. Below is our point-by-point response.
A1: Thank you for your comment! We have evaluated Dr.LLM [1] and PoLar [2] using their official implementations, trained on the same training set as SelfMorph. As the table below, SelfMorph achieves the highest or joint-highest score in 26/27 settings. On Qwen3.5-27B, its gains over the stronger comparator reach 20.18% on BBH and 23.16% ARC-Challenge. We will add the results and discussion in the revised paper.
[1] Heakl, Ahmed, Martin Gubri, Salman Khan, Sangdoo Yun, and Seong Joon Oh. "Dr. llm: Dynamic layer routing in llms." In International Conference on Learning Representations, vol. 2026, pp. 73320-73337. 2026.
[2] Li, Ziyue, Yang Li, and Tianyi Zhou. "Skip a Layer or Loop It? Learning Program-of-Layers in LLMs." arXiv preprint arXiv:2606.06574 (2026).
A2: Good idea! We compare the methods along three axes: computation topology, decision signal, and supervision.
First, in addition to BASE and REPEAT, SelfMorph introduces BRANCH, an operation absent from both Dr.LLM and PoLar. SelfMorph can therefore choose a distinct computation topology: when to continue transforming representations through serial repetition, when to explore alternative hidden-state trajectories through local branching, and when to change nothing. We also define a parameterized morphology space that jointly covers morphology type, intervention span, and computational intensity.
Second, Dr.LLM performs local routing using hidden states around individual layers, while PoLar uses prompt representations from an external embedding model. In contrast, SelfMorph uses hidden states from multiple depths that the backbone actually produces while processing the raw prompt, thereby capturing how the current input's representation evolves. This design directly connects the internal signals produced by the target model's execution to the global morphology decision. Table 5 in our paper also supports this design: multi-layer trajectories outperform a single hidden state or prompt embedding.
Third, SelfMorph directly learns the trade-off between performance and cost from a utility distribution over the candidate actions. Both Dr.LLM and PoLar use offline MCTS to construct per-layer labels or segment labels. SelfMorph instead evaluates each action in a compact, prefiltered candidate set for both task score and computational cost, combines the two into an action utility, and converts the utilities into a soft supervision distribution. This supervision not only indicates which actions are effective, but also preserves their relative utility, cost differences, and ties, allowing multiple useful actions to contribute to training.