acceptodds
Under review as a conference paper at ICLR 2027

Compute Has Shape: Self-Morphing the Internal Computation of Frozen LLMs

Abstract

Test-time scaling improves large language models through additional computation, but how to adapt the topology of internal computation to individual inputs remains underexplored. Existing approaches largely fix recurrent or parallel architectures during training, or reconfigure frozen models along a single structural axis. We introduce SelfMorph, which makes computational shape an input-dependent inference-time decision. It unifies the original forward path (BASE), serial reuse of a layer span (REPEAT), and local branching and merging of perturbed hidden states (BRANCH) in a unified space parameterized by type, location, and intensity. Since they offer complementary benefits rather than uniform dominance, we use a lightweight selector that reads the backbone’s depth-wise hidden state trajectory on the prompt to choose a morphology before response generation. We train the selector by distilling utility-based distribution from offline morphing that jointly account for task performance and computational cost. Evaluated on three LLMs spanning 4B–27B, SelfMorph achieves a 8.06% mean relative improvement on five in-distribution benchmarks and 4.45% on four unseen benchmarks. It achieves the strongest performance in 24 of 27 settings. These findings identify input-adaptive computational topology as a complementary axis of test-time scaling beyond generating more tokens.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tRXJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SELFMORPH, which adapts the internal execution of a frozen LLM by selecting among BASE, REPEAT(increase depth) and BRANCH(increase depth). A lightweight self-supervised morphing predictor is trained to generate the optimal "computation shape" based on the input prompt's hidden trajectories. Evaluations across diverse benchmarks demonstrates consistent accuracy gains or compute-efficiency trade-offs over frozen backbones.

Critical strengths
  1. The problem is well motivated. Treating serial loop and local branching as input-dependent execution choice is a useful extension of frozen-model inference. The backbone remains unchanged, and the selector is pre-trained to improve the computation of inference.
  2. The evaluation includes multiple model scales and unseen tasks across multiple benchmarks, moreover provides representation ablations and practical implementation analysis.
Critical weaknesses
  1. Lack of direct comparison with highly relevant prior work[1,2]. This is the most critical weakness of this paper. The core philosophy of SelfMorph is reconfiguring frozen LLM layers via dynamic branch or looping via a lightweight predictor to map inputs to custom execution paths, which is similar with Dr. LLM[1] and PoLar[2]. While the authors acknowledge PoLar briefly in paper, they omit any direct empirical comparison with both works.

[1] Heakl, Ahmed, Martin Gubri, Salman Khan, Sangdoo Yun, and Seong Joon Oh. "Dr. llm: Dynamic layer routing in llms." In International Conference on Learning Representations, vol. 2026, pp. 73320-73337. 2026.

[2] Li, Ziyue, Yang Li, and Tianyi Zhou. "Skip a Layer or Loop It? Learning Program-of-Layers in LLMs." arXiv preprint arXiv:2606.06574 (2026).

Questions for authors
  1. Please provide direct performance comparisons between SelfMorph, Dr.LLM, and PoLar on the same benchmarks.

  2. Please provide an in-depth methodological comparison with Dr.LLM and PoLar, clearly identifying SelfMorph’s novel contributions relative to these methods.

  3. Please provide controlled ablation studies that isolate the contributions of SelfMorph’s distinguishing components, including cross-layer hidden-state trajectories, the BRANCH operation, and input-dependent selection among BASE, REPEAT, and BRANCH together with the intervention span and intensity.

5 comments
AuthorsOpenReview ↗

Response to Reviewer tRXJ [1/2]

We sincerely thank you for the constructive comments. Below is our point-by-point response.

Q1: Direct comparisons with Dr.LLM and PoLar on the same benchmarks. (Weakness 1 and Question 1)

A1: Thank you for your comment! We have evaluated Dr.LLM [1] and PoLar [2] using their official implementations, trained on the same training set as SelfMorph. As the table below, SelfMorph achieves the highest or joint-highest score in 26/27 settings. On Qwen3.5-27B, its gains over the stronger comparator reach 20.18% on BBH and 23.16% ARC-Challenge. We will add the results and discussion in the revised paper.

BackboneMethodBBHGSM8KMATH-500ARC‑ChallengeOpenBookQA
Qwen3.5-27BDr.LLM0.57370.40400.52000.63330.8333
PoLar0.51050.41600.56000.62000.8267
SelfMorph0.68950.43600.56000.78000.8400
Qwen3.5-4BDr.LLM0.37890.17600.32000.78670.7867
PoLar0.41050.17200.33000.79330.7867
SelfMorph0.52110.18000.34000.80000.7933
Gemma4-12BDr.LLM0.56320.22400.23000.77330.8800
PoLar0.60530.23200.23000.77330.9000
SelfMorph0.61050.23600.24000.77330.9067
BackboneMethod2WikiMultiHopHellaSwagTruthfulQA-MC1GPQA-Diamond
Qwen3.5-27BDr.LLM0.71360.91000.75000.5500
PoLar0.74630.91000.79000.5800
SelfMorph0.77080.93000.80000.6000
Qwen3.5-4BDr.LLM0.51560.79000.67000.4400
PoLar0.56430.85000.67000.4400
SelfMorph0.58330.89000.67000.4700
Gemma4-12BDr.LLM0.15350.39000.49000.2400
PoLar0.13810.46000.52000.2900
SelfMorph0.16390.46000.48000.3200

[1] Heakl, Ahmed, Martin Gubri, Salman Khan, Sangdoo Yun, and Seong Joon Oh. "Dr. llm: Dynamic layer routing in llms." In International Conference on Learning Representations, vol. 2026, pp. 73320-73337. 2026.

[2] Li, Ziyue, Yang Li, and Tianyi Zhou. "Skip a Layer or Loop It? Learning Program-of-Layers in LLMs." arXiv preprint arXiv:2606.06574 (2026).

Q2: In-depth methodological comparison with Dr.LLM and PoLar. (Question 2)

A2: Good idea! We compare the methods along three axes: computation topology, decision signal, and supervision.

First, in addition to BASE and REPEAT, SelfMorph introduces BRANCH, an operation absent from both Dr.LLM and PoLar. SelfMorph can therefore choose a distinct computation topology: when to continue transforming representations through serial repetition, when to explore alternative hidden-state trajectories through local branching, and when to change nothing. We also define a parameterized morphology space that jointly covers morphology type, intervention span, and computational intensity.

Second, Dr.LLM performs local routing using hidden states around individual layers, while PoLar uses prompt representations from an external embedding model. In contrast, SelfMorph uses hidden states from multiple depths that the backbone actually produces while processing the raw prompt, thereby capturing how the current input's representation evolves. This design directly connects the internal signals produced by the target model's execution to the global morphology decision. Table 5 in our paper also supports this design: multi-layer trajectories outperform a single hidden state or prompt embedding.

Third, SelfMorph directly learns the trade-off between performance and cost from a utility distribution over the candidate actions. Both Dr.LLM and PoLar use offline MCTS to construct per-layer labels or segment labels. SelfMorph instead evaluates each action in a compact, prefiltered candidate set for both task score and computational cost, combines the two into an action utility, and converts the utilities into a soft supervision distribution. This supervision not only indicates which actions are effective, but also preserves their relative utility, cost differences, and ties, allowing multiple useful actions to contribute to training.

AuthorsOpenReview ↗

Response to Reviewer tRXJ [2/2]

Q3: Controlled ablation studies. (Question 3)

A3: We describe the controlled ablation studies below. Each ablation changes only one component while keeping all other settings and hyperparameters fixed.

Cross-layer hidden-state trajectories. We compare a single hidden state with trajectories formed from different numbers of observed layers. Single hidden state has the worst performance. This indicates that cross-layer hidden-state trajectories provide richer information for morphology selection.

Trajectory lengthQwen3.5-27BQwen3.5-4BGemma4-12B
10.59400.48450.5286
30.59640.49050.5333
50.59520.49520.5345
100.60830.49640.5369

BRANCH operation. We remove BRANCH and train the selector. Performance drops on all three backbones, showing that BRANCH provides gains that are not fully captured by BASE and REPEAT alone.

Pool (spans × intensities)Qwen3.5-27BQwen3.5-4BGemma4-12B
BASE + REPEAT (4 × 3)0.59640.48100.5321
BASE + REPEAT (4 × 3) + BRANCH (4 × 3)0.60830.49640.5369

Input-dependent selection. We evaluate it at two levels. First, does per-input selection outperform a fixed strategy? BenchPrior selects the best fixed action for each benchmark, while SelfMorph outperforms BenchPrior in every setting. This supports the value of selecting an action for each input. Second, do span and intensity adaptation provide additional gains? The table below shows their improvement.

Pool (spans × intensities)Qwen3.5-27BQwen3.5-4BGemma4-12B
BASE + REPEAT (1 × 1) + BRANCH (1 × 1)0.57860.47380.531
BASE + REPEAT (4 × 1) + BRANCH (4 × 1)0.58690.48330.5369
BASE + REPEAT (4 × 3) + BRANCH (4 × 3)0.60830.49640.5369

Thank you for the additional experiments, clarifying the comparisons with prior work and further ablation. These responses have addressed my questions. However, considering the novelty relative to prior work, together with overall strengths and limitations, I maintain my original rating.

Reviewer_xt4COpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SelfMorph to choose the internal computation of a frozen language model for each input. Its action space contains the original path, repeated execution of a selected layer span, and branching of perturbed hidden states followed by averaging. Candidate spans are screened on separate diagnostic tasks. A small selector reads the model's hidden-state trajectory during prompt processing and predicts an action before generation. It is trained from offline scores for 25 candidate actions with a computation penalty. The paper reports improvements on five held-out task splits and transfer to four unseen benchmarks across three backbones, with analyses of action diversity, selector inputs, and execution costs.

Critical strengths
  1. Sections 2 and 3 provide a practical formulation of input-dependent internal computation. The method jointly selects morphology, location, and intensity while keeping the backbone frozen. Screening is separate from the evaluation benchmarks, and the selector uses only prompt computation at inference. This makes the contribution more concrete than a claim that additional depth or branches can sometimes help.
  2. The evaluation provides useful evidence for selection and transfer. Table 1 compares against a task-specific fixed action, and Table 3 tests four benchmarks excluded from selector training. Tables 4 and 5 test action diversity and trajectory inputs. Appendix E also discusses real cache and numerical constraints, including the failure of collapsing branch execution shapes to preserve outputs.
Critical weaknesses
  1. My main concern is that the resource measurements do not make the cost of the full decision pipeline clear. Appendix E.3 describes Figures 7 and 8 as morphology execution measurements over the generation call. Section 3.3 also requires trajectory probing and selector execution, and Branch uses a separate prefill after probing. I do not see this additional prompt work explicitly represented by the analytic penalty based on span width and intensity. I find the prefix and cache optimizations valuable, but the total time and memory trade-off for a new input is still unclear to me. I see this as a fixable cost-reporting concern and would like the authors to state the measurement boundary and account for any omitted decision overhead when discussing practical efficiency.
  2. My concern about the component analysis is that most evidence attributing gains to morphology diversity and selector design uses a different answer protocol from the main result. Appendix C.3 uses the 32-token protocol for oracle collection, and Section 4.3 and Appendix G use it for the main ablations. Table 1 instead scores completed responses. Section 4.1 already shows that morphology changes how early the answer appears. I think a 32-token ablation gain could reflect better answer timing as well as better final correctness. I accept the completed-response gains as valid evidence, but I think the component conclusions need this qualification. I would also like the fitting protocol for the fixed actions stated beside Table 1.
Questions for authors
  1. I would like to know whether the SelfMorph measurements in Figures 7 and 8 include probing, selector execution, and the separate Branch prefill. I would also like the authors to clarify the existing timing boundaries and summarize the cost of any omitted stages from available profiling, or state the limitation.
  2. I would like to know whether the fixed actions in Table 1 were chosen using 32-token scores or completed-response scores. I would also like to know whether any existing completed-response ablations support the action-diversity and trajectory conclusions. I would like the authors to distinguish evidence for early-answer behavior from evidence for final correctness if no such results are available.
3 comments
AuthorsOpenReview ↗

Response to Reviewer xt4C

We sincerely thank you for the constructive comments. Below is our point-by-point response

Q1: Measure the full decision pipeline and state the measurement boundary. (Weakness 1 and Question 1)

Thank you for your comment! The cost evaluation in Appendix E.3 compares the execution cost of morphology actions. We therefore add SelfMorph (full), which covers the full pipeline, including probing, selector and any other stage. As shown below, the full pipeline adds a little latency and peak memory relative to execution alone. Probing and selection occur once per response, so their cost can be amortized over decoding. We will make these boundaries explicit in Appendix E and use full-pipeline measurements when discussing practical efficiency.

BackboneMethodTotal Throughput (tokens/s)Latency (s/prompt)Peak Memory (GB)
Qwen3.5-27BBase10.871.216250.394
Repeat8.500.904050.404
Branch10.921.340250.625
SelfMorph (execution)8.821.052750.404
SelfMorph (full)8.481.086950.652
Qwen3.5-4BBase22.600.65338.082
Repeat18.840.77168.094
Branch22.000.66498.410
SelfMorph (execution)21.720.72068.118
SelfMorph (full)19.820.78178.226
Gemma4-12BBase22.491.241222.508
Repeat22.101.263622.511
Branch21.661.286022.665
SelfMorph (execution)22.081.266222.559
SelfMorph (full)21.391.303122.722

Q2: State how the fixed actions were fitted and distinguish short-budget component evidence from completed-response accuracy. (Weakness 2 and Question 2)

A2: The fixed actions were selected by maximizing mean training utility computed from 32-token responses, using the same training set as SelfMorph. Table 1 then evaluates all resulting strategies on completed responses. We will state this fitting protocol explicitly beside Table 1.

The short budget and direct-answer prompt keep utility collection tractable: without a limit, collection time grows linearly with response length. The existing results nevertheless provide evidence that the resulting utilities transfer. In Table 1, utilities are fitted on the short window but evaluated on completed responses, and SelfMorph still has the best performance. Across the two evaluation, the ordering shows a relatively consistent trend. Our ablation is based on this consistency and we will maintain the distinction throughout the component analysis.

Reviewer_xt4COpenReview ↗

Response to Authors: Raising Confidence

Thank you to the authors for the detailed response and additional measurements. The full-pipeline efficiency breakdown and the explicit clarification of the measurement boundaries effectively address my first concern. Additionally, the explanation of the fitting protocol on the short window and how it transfers to completed responses resolves my second question. All of my concerns have been well addressed.

I will raise my confidence score and continue to recommend acceptance.

Thank you for your careful review and for raising your confidence! Your constructive comments helped make our efficiency and evaluation claims more transparent, and we will include these results and clarifications in the final version.

Reviewer_hzmvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a frozen LLM can do better by choosing an input-specific internal computation pattern before it answers. The method defines a small action space over the frozen stack: run normally, repeat a contiguous block of layers, or branch into perturbed replicas through a block and average them. A lightweight selector reads the prompt's hidden states and picks one action for the whole answer. It is evaluated on three frozen backbones (Qwen3.5-27B, Qwen3.5-4B, Gemma4-12B) across several reasoning and science benchmarks, against the base model and fixed single-action choices, plus a set of benchmarks held out from the selector.

Critical strengths

The paper does a good job validating the core idea. A small selector reads the prompt's hidden states and picks an input-specific computation pattern over a frozen model. The gains hold on three backbones and on benchmarks the selector never trained on. It can also choose to do nothing, so it does not force extra compute. This is the central claim, and it is shown well enough to matter.

Critical weaknesses
  • The reported speed and memory only cover running an already-chosen action, not the probe and selector overhead. The cost-accuracy figure uses an analytic cost on the training split, but not measured latency. There is also no comparison to any adaptive-inference method, only the base model and the authors' own fixed actions. The whole point is test-time adaptation. So for me, the key question stays open: are the gains worth the extra compute, or would a simpler fixed or token-based strategy match them?

  • The paper also really needs confidence intervals to capture variance. Since branch actions are stochastic, each action is scored from a single run, and there are no seeds or paired tests. Many gains are only a few correct answers, so we need to check for noise. The largest ones come from a short 32-token budget, where the paper itself says the intervention mainly makes the model answer sooner. So the breadth claim is hard to separate from noise and from answer timing.

Questions for authors
  1. Do the timing numbers include the probe and selector, or only the chosen action? What is the full probe-to-generation latency and peak memory, and on what hardware?
  2. Are the gains and oracle choices stable across branch resampling and selector seeds? Can you report paired uncertainty from the runs you already have?
  3. Do the selector and fixed baselines keep their ordering if utilities are fit on completed responses, not the short window?
5 comments
AuthorsOpenReview ↗

Response to Reviewer hzmv [1/2]

We sincerely thank you for the constructive comments. Below is our point-by-point response.

Q1: What are the full latency and peak memory? (Weakness1 and Question1)

A1: Thank you for your comment! The cost evaluation in Appendix E.3 compares the execution cost of different morphology actions. We therefore add SelfMorph (full), which covers the full pipeline, including probing, selector and all generation. As shown below, the full pipeline adds a little latency and peak memory relative to execution alone. Probing and selection occur once per response, so their cost can be amortized over growing decoding.

BackboneMethodTotal Throughput (tokens/s)Latency (s/prompt)Peak Memory (GB)
Qwen3.5-27BBase10.871.216250.394
Repeat8.500.904050.404
Branch10.921.340250.625
SelfMorph (execution)8.821.052750.404
SelfMorph (full)8.481.086950.652
Qwen3.5-4BBase22.600.65338.082
Repeat18.840.77168.094
Branch22.000.66498.410
SelfMorph (execution)21.720.72068.118
SelfMorph (full)19.820.78178.226
Gemma4-12BBase22.491.241222.508
Repeat22.101.263622.511
Branch21.661.286022.665
SelfMorph (execution)22.081.266222.559
SelfMorph (full)21.391.303122.722

Q2: How does SelfMorph compare with relevant methods? (Weakness 1)

A2: We added Dr.LLM [1] and PoLar [2] using their official implementations, trained on the same training set as SelfMorph. As the table below, SelfMorph achieves the highest or joint-highest score in 26/27 settings. On Qwen3.5-27B, its gains over the stronger comparator reach 20.18% on BBH and 23.16% ARC-Challenge.

BackboneMethodBBHGSM8KMATH-500ARC‑ChallengeOpenBookQA
Qwen3.5-27BDr.LLM0.57370.40400.52000.63330.8333
PoLar0.51050.41600.56000.62000.8267
SelfMorph0.68950.43600.56000.78000.8400
Qwen3.5-4BDr.LLM0.37890.17600.32000.78670.7867
PoLar0.41050.17200.33000.79330.7867
SelfMorph0.52110.18000.34000.80000.7933
Gemma4-12BDr.LLM0.56320.22400.23000.77330.8800
PoLar0.60530.23200.23000.77330.9000
SelfMorph0.61050.23600.24000.77330.9067
BackboneMethod2WikiMultiHopHellaSwagTruthfulQA-MC1GPQA-Diamond
Qwen3.5-27BDr.LLM0.71360.91000.75000.5500
PoLar0.74630.91000.79000.5800
SelfMorph0.77080.93000.80000.6000
Qwen3.5-4BDr.LLM0.51560.79000.67000.4400
PoLar0.56430.85000.67000.4400
SelfMorph0.58330.89000.67000.4700
Gemma4-12BDr.LLM0.15350.39000.49000.2400
PoLar0.13810.46000.52000.2900
SelfMorph0.16390.46000.48000.3200

[1] Heakl, Ahmed, Martin Gubri, Salman Khan, Sangdoo Yun, and Seong Joon Oh. "Dr. llm: Dynamic layer routing in llms." In International Conference on Learning Representations, vol. 2026, pp. 73320-73337. 2026.

[2] Li, Ziyue, Yang Li, and Tianyi Zhou. "Skip a Layer or Loop It? Learning Program-of-Layers in LLMs." arXiv preprint arXiv:2606.06574 (2026).

AuthorsOpenReview ↗

Response to Reviewer hzmv [2/2]

Q3: Could the reported improvements reflect stochastic variation or earlier answer production under the 32-token budget? (Weakness 2 and Question 2)

A3: Good suggestion! We evaluated eight selector training seeds and separately evaluated eight BRANCH noise seeds with the remaining conditions fixed. Using per-example predictions, we performed 10,000 paired bootstrap resamples. The score differences and 95% confidence intervals are reported below.

Source of variationBackboneΔ vs. BASE (pp)Δ vs. BenchPrior (pp)
Selector seedQwen3.5-27B+8.10 [5.55, 10.68]+1.90 [0.46, 3.39]
Qwen3.5-4B+3.21 [1.53, 4.88]+4.17 [2.31, 6.09]
Gemma4-12B+1.07 [0.02, 2.25]+1.19 [0.11, 2.36]
BRANCH seedQwen3.5-27B+8.10 [5.48, 10.71]+1.90 [0.54, 3.36]
Qwen3.5-4B+3.21 [1.59, 4.82]+4.17 [2.41, 5.95]
Gemma4-12B+1.07 [0.00, 2.31]+1.19 [0.03, 2.39]

As for concern about earlier answer, the main accuracy evidence comes from Table 1, which scores completed responses. Table 2 measures whether the answer appears within 32 tokens. Thus, the completed-response improvements establish a gain beyond earlier answer production, while the short-budget results characterize an additional response-behavior benefit.

Q4: Are oracle choices stable across BRANCH resampling? (Question 2)

A4: Since oracle collection evaluates all additional actions per example, we tested oracle-action stability across eight BRANCH noise seeds on two disjoint training subsets: a general set of 300 examples, with 60 per benchmark, and 120 examples whose original oracle selected BRANCH. Agreement with the original oracle action are shown in the table below, proving that the oracle preferences are stable under perturbation resampling.

BackboneGeneral setBRANCH-oracle set
Qwen3.5-27B95.67%85.83%
Qwen3.5-4B97.33%90.83%
Gemma4-12B98.00%88.33%

Q5: Do the selector and fixed baselines keep their ordering if utilities are fit on completed responses? (Question 3)

A5: The short budget and direct-answer prompt keep utility collection tractable: without a limit, collection time grows linearly with response length. The existing results nevertheless provide evidence that the resulting utilities transfer. In Table 1, utilities are fitted on the short window but evaluated on completed responses, and SelfMorph exceeds BenchPrior in all 15 settings. Across the two evaluation regimes, SelfMorph remains first, while REPEAT and BenchPrior outperform BASE and BRANCH.

AuthorsOpenReview ↗

Looking forward to your response

Dear Reviewer,

Thank you again for reviewing our paper! We have posted our responses to your questions and concerns. We are wondering if you have a new evaluation of our paper after reading our responses. If you have other questions, we are very willing to discuss them with you.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.