acceptodds
Under review as a conference paper at ICLR 2027

Geometry-Guided Layer Selection for Language Model Compression

Abstract

Which transformer blocks should be considered for removal? We study a layer-selection criterion that compares centered activation subspaces through projector distance. An analytical construction distinguishes this population statistic from tokenwise cosine, and a rank–angle decomposition makes its selections interpretable. On DeepSeek-R1-Distill-Qwen-14B, we compare geometry, Block Influence, and positional references using matched calibration and separate single-block interventions on 144 metallurgy questions. The geometry-selected set has a mean reference-token loss change of nats, compared with for Block Influence and positive changes for the early and late positional references. The geometry–Block Influence contrast is unresolved by the workload-bootstrap interval. All 15 geometric choices are reproduced by angular changes between retained rank-one subspaces at three calibration seeds. These results characterize an interpretable selection signal and its conditional-loss profile; they do not estimate joint deletion effects or free-generation accuracy. A separate historical compression-and-adaptation study provides resource context: a 33-block geometric configuration uses 28.0% fewer parameters and achieves the observed token throughput of the 48-block original. We distinguish candidate selection from validation of the resulting compact model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
geometry, adaptation, LLM pruning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bENCOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper describes a new "geometry-guided (GG) population-level" ayer-selection method for pruning of Transformer-based Language models. The "GG" method selects blocks for removal by comparison of population set of centered activations before and after block. Authors provided detailed comparison with Block Influence based pruning method using set of metallurgy-related samples.

Critical strengths

None

Critical weaknesses

Limited novelty of proposed method vs Block Influence method.

Limited validation: The matched study covers only one model family, one domain, one calibration pool, and one response- conditioning protocol.

The paper is badly written with many typos and missed definitions . Few examples:

  • Abstract, line 020: "All 15 geometric choices are reproduced by angular changes between retained rank-one subspaces at three calibration seeds. " Should be something like " The choice of15 blocks selected for removal based on activation geometry based method, is reproduced ..."
  • Introduction, line 036 "... Metallurgy provides the domain workload; the selection rule itself does not depend on that domain." I would suggest to change to "We verify new pruning method using the set of sentences from metallurgy domain. We claim that the layer selection for removal is invariant to the probe set" . Btw, this claim was not proved.
  • Equation (1) line 116: $r_l" is not defined. Also the paper doesn't explicitly define how one can compute U_l from X_l in this section, and clarification is given only in "implementation" part
Questions for authors

Suggestions for future work:

  1. It would help to demonstrate that the layer selection as potential canidate removal is test-domain independet using additional samples from other domains (e.g. math, code)
  2. the paper uses "geometry" as method name. I woud recommend using more concrete name, someting like "population activation projector distance"
  3. carefully rewrite the paper making sure that each symbol is defined.
1 comment
AuthorsOpenReview ↗

Response to Reviewer bENC

Thank you for the suggestions. Activation-projector distance compares centered activation subspaces and provides a rank-angle decomposition that explains its block selections. The formula contains no metallurgy-specific features or rules, which is the intended meaning of “the selection rule itself does not depend on that domain.” The activation distribution, estimated subspaces, scores, and selected blocks can all change with the calibration texts. We do not claim invariance to the probe set. Stability at three sampling seeds within one pool supplies no evidence of stability across math, code, or other domains.

The name “activation-projector distance” also describes the computation more precisely than “geometry.” For a token-by-feature activation matrix XℓX_\ell, we subtract its column mean, estimate its leading right singular vectors, and retain the smallest rank meeting the 90% energy target, subject to the stated approximation cap. The retained vectors form UℓU_\ell; rℓr_\ell is their number, and Pℓ=UℓUℓ⊤P_\ell = U_\ell U_\ell^\top is the orthogonal projector. The block score is the Frobenius distance between consecutive projectors. These definitions and the truncation rule belong with Equation (1).

BI averages cosine changes of paired token vectors. Our score compares the retained subspaces of centered activation populations and is insensitive to translations, singular-vector signs, and singular-value weights within the retained subspace. The analytical examples show that the two statistics can disagree. The projector identity is classical. We apply and evaluate it for candidate selection, using the decomposition to explain the actual choices.

All 15 selected transitions have rank one at both endpoints and meet the energy target. Ranking their angular changes alone recovers the entire geometric shortlist at all three calibration seeds. Geometry and BI share only six of the 15 choices, yet their mean singleton-deletion losses are close: −0.1483 and −0.1439 nats, with an unresolved paired contrast. This supports an interpretable alternative signal on the tested model and workload, while superiority and domain-independent selection remain unestablished.

The validation covers one model family, calibration pool, domain, and response-conditioning protocol. Math/code calibration and evaluation would test how both selections and deletion effects change across domains.

Reviewer_FYdJOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a layer-selection score based on changes in centered activation subspaces and compares it with Block Influence using single-block deletion tests on a 48-layer language model. Geometry and BI perform similarly and better than positional baselines on reference-token NLL.

Critical strengths

The paper gives a clear geometric interpretation of its layer score and tests its selections against matched single-block deletion measurements, making the proposed signal and its limits easy to inspect.

Critical weaknesses
  • The scope and the contribution of the paper is really unclear. It introduces a geometric layer-selection score, evaluates selected layers through separate single-block deletions, and reports resource measurements for separately pruned and adapted checkpoints. The relationship among these studies is not sufficiently explained. In particular, the single-block results do not establish the effect of jointly removing the selected layers, while the adapted-checkpoint results do not isolate the proposed selector or establish answer-quality preservation.

  • The results provide limited evidence that geometry improves practical layer pruning. The evaluation averages 15 separate single-block deletions rather than testing their joint removal, so it does not validate the selected set as a compressed model. Moreover, geometry’s advantage over BI is statistically unresolved, and its angular component is only weakly correlated with deletion loss.

Questions for authors

See the weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer FYdJ

Thank you for recognizing the geometric interpretation and the matched interventions. Centered activation-projector distance identifies a reproducible set of individually low-loss deletions, and its rank-angle decomposition explains what determines that set. The earlier compact-checkpoint measurements provide separate resource context.

The first study defines the score and its distinction from tokenwise cosine. The second tests it on the unadapted model using the same calibration positions and the same 48 single-block deletion interventions for both selectors. At shortlist size 15, geometry and BI yield mean reference-token loss changes of −0.1483 and −0.1439 nats, compared with positive changes for the early and late positional references. The paired geometry-BI interval [−0.00970, 0.00064] does not establish superiority. The third study uses earlier 33-block checkpoints with continued pretraining; its geometric removal set differs from the matched set by one block and its calibration sampling differs. Its parameter and throughput measurements describe those earlier checkpoints; they cannot establish joint-deletion performance for the current selector or isolate selection from adaptation.

All 15 selected transitions connect rank-one subspaces, satisfy the energy-retention target, and are recovered by sorting the angular term alone at each of three calibration seeds. Across the larger equal-rank population, the angular term has weak correlation with deletion loss, so the interpretability result concerns which blocks are selected rather than strong global loss prediction.

The selectors make different choices despite their close aggregate results: they share six blocks and each contributes nine others. The geometry-only group has lower loss on strength recall, whereas the BI-only group has lower loss on composition recall. Their primary means are −0.1179 and −0.1106; the shared group's −0.1939 cancels in the paired comparison. This identifies the nine-for-nine substitution to test in a joint-deletion experiment and separates it from the benefit of the shared core.

The matched study measures individual-deletion effects; joint compression remains a separate validation question. In the earlier generation study, the retrospective semantic evaluation provides evidence against preserved answer quality under the tested protocol. A practical pruning claim therefore requires matched joint removal and adaptation, together with independent answer-quality evaluation. The specified statistic, its reproducible rank-one selection mechanism, and its measured intervention profile provide a concrete basis for that test.

Reviewer_BcQAOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a method to evaluate the importance of a layer in the context of layer dropping in Transformer models. The Block Influence (BI) introduced by ShortGPT measures a score of the block by computing 1 minus the average cosine similarity between input and output tokens of a block, given a calibration dataset. As a result, if the input and output activations of the block are highly aligned, BI will be close to 0, meaning that the block doesn't significantly impact the residual stream and is a reasonable candidate for being droped.

This paper, in constrast, points out that this measure has some issues. Let HℓH_\ell and Hℓ+1H_{\ell+1} be the hidden states given as input to blocks ℓ\ell and ℓ+1\ell+1, respectively. Then, the values of BI could be very high if the norm of the means of HℓH_\ell and Hℓ+1H_{\ell+1} was significant . However, if these residual streams were instead both centered, then a completely different measure of alignment could be obtained. For example, two vectors that were aligned because they pointed near the means of their populations, could be in opposite directions when the means of their populations were subtracted from them.

The paper's proposal, then, seems to be to instead use a PCA-based score to measure the importance of a block. First, they apply PCA to the matrices representing the inputs and outputs of a block given a calibration dataset. Then, they compare the norm of the difference between the principal components of the residual streams constructed using a calibration dataset.

Since PCA centers the matrices, the issue described previously is resolved. Using PCA also allows for an interesting decomposition of the transformer block score, namely that it is a sum between the rank change (i.e., how many rank-1 subspaces are retained in PC decomposition of the block outputs, minus how many rank-1 subspaces are retained in the PC decomposition of the block inputs) and a measure of angular change.

The authors evaluate the proposed score in a single model, and on a dataset that seems to be constructed by the authors. They study only the case where a single block is removed at a time (instead of dropping several blocks) and compare the results with the score proposed by ShortGPT.

Critical strengths

S1. Interesting idea for block importance The use of principal components to compare the input and output distributions of the transformer block is interesting, and in principle could lead to better results and to interpretations about how the residual streams are changing across blocks.

Critical weaknesses

W1. Presentation The paper is not easy to follow. In several parts information seems misplaced or missing.

I give a non-exhaustive list of examples from the text to clarify what I mean. I believe this work would benefit from substantial rewriting.

W1.1. In multiple places there is a mention to a "historical" selector. What does historical selector mean? This sounds like a separate method that was done before the main content of the paper.

W1.2. In Section 3.1, the paper mentions that the principal directions are "retained by an energy rule". This rule is seems to be explained in Section 3.4, where they describe how they estimate energy for a candidate rank. A method-level description of this could be closer to Section 3.1, or it could be worth to point, in Section 3.1, that the energy rule is explained in Section X. Also related to the rank retention, the fact that some truncation of the principal components is part of the method appears without "proper notice". It could be worth to have this more explicitly mentioned when describing the method in the Introduction.

W1.3. In the same Section, the paper writes: "Given budget k, we select the k smallest scores, ...". It is not clear for what kk is being selected, specially considering that comparisons are done by always removing a single block (so it is not the case that kk blocks are being selected). Probably, it means that the removal of top-jj block ranked by the proposed method is compared with the removal of the top-jj block ranked by BI.

W1.4. Also in Section 3.1, the sentence "projection discards amplitudes within retained directions" is not clear to me. What is being projected? If something is being projected on the retained directions, why would their amplitudes be discarded?

W1.5. Section 3.4 then seems to touch on several different topics, and the sections organization could benefit from some restructuring. In several parts, it seems closer to an Experiments section, instead of a Method section. For example, it describes that both the proposed approach and BI use "3,000 common token positions sampled from a 9,000-token pool drawn from 100 frozen assistantmessage texts (10,577 nonpadding tokens)". Personally, I believe this is closer to an Experiments section which described the dataset that was used. At the same time, this same section also described their energy rule: "For candidate rank r, estimated energy is the sum of the first r squared singular values divided by the exact ∥X~L∥F2\| \tilde{X}_L \|_F^2 .", which is part of the method description.

W1.6. Also in Section 3.4, a few expressions are introduced which I believe are not explained, such as "Gaussian range finder".

W1.7. Section 3.4 also appears to give information that are implementation details, and are out-of-place. For instance, the seed values of the runs are given ("The main seed is 42; sensitivity seeds are 7 and 123."), and it is mentioned that the implementation uses "explicit block hooks" ("The matched implementation uses paired positions and explicit block hooks;").

W1.8. There are also some examples in Section 4.1. A checkpoint revision value ("1df85071 78afcc1bef68cd8c393f61a886323761") is specified for the model that was used. This seems out-of-place and also unclear.

W1.9. (~ l. 205) The description of the dataset is not clear. In the paper, it is also mentioned that a metallurgy dataset was used. This same paragraph also has information that could be better positioned in an appendix (e.g., "questions are selected deterministically"). The same paragraph closes with "These questions come from the previously studied workload, not an independently held-out benchmark; training overlap is unresolved.", which is not clear.

W1.10. Section 4.2 also presented out-of-place mnetions such as a seed number ("Its percentile interval uses 2,000 stratified bootstrap resamples of the 144 declared question groups, with seed 2026") and a reference to a specific snapshot ("Snapshot 005 preserves inputs, runtime, spectra, losses, ...").

W2. Empirical Rigor The work does not provide sufficient experiments to support claims and a better understanding of the proposed method.

W2.1. They consider only block-by-block removal, this ignores interactions between block removals

W2.2. The method seems to be evaluated on a single model.

W2.3. The dataset used is not clear, and is not part of the datasets that are typically used in the context of block pruning methods.

W2.4. There is a lack of baselines as well, since there are several other related methods on dropping blocks.

W2.5. The results are inconclusive, as the authors themselves mention in the conclusion: "The current evidence supports a characterized selection signal; it does not establish a universally stronger ranking."

W2.6. I also think that the initial premise about having mean shifts in the residual streams should be evaluated empirically (and contrasted with maybe the variance). And if it is indeed the case that the population means are dominating the cosine similarities of BI, then a simpler ablation where just the mean is removed prior to the computation of the cosine could be evaluated.

Questions for authors

Q1. Is PCA applied independently to each population of vectors (that is, to the inputs matrix, and then independently to the outputs matrix of the blocks)? If so, couldn't it happen that two populations that have different means will now share the 0 mean? Then, this could imply that two vectors were, in fact, different, will now become aligned because with respect to the mean of their populations they have a similar direction.

Q2. Could you clarify the datasets that were used? Were they generated by the authors? Why not use more common datasets used in this field, such as Wikitext2 and others?

2 comments
AuthorsOpenReview ↗

Response to Reviewer BcQA

Thank you for the detailed comments. Activation-projector distance selects candidate blocks by the change in their retained, centered activation subspaces. We assess those choices through matched deletion interventions and use the rank-angle decomposition to explain the selected set.

PCA is applied independently to the input and output populations, using the same sampled token positions. For each centered matrix, UℓU_\ell contains the retained right singular vectors and rℓr_\ell is their number. We retain the smallest rank whose estimated squared singular values account for 90% of the centered matrix's total squared Frobenius norm, or the available rank if the estimate reaches its cap. The score compares Pℓ=UℓUℓ⊤P_\ell = U_\ell U_\ell^\top with Pℓ+1P_{\ell+1}, avoiding dependence on the arbitrary signs and bases of principal-component vectors. The “Gaussian range finder” is the randomized sketch used to approximate the leading singular subspace. The rank rule belongs with this score definition; seeds, hooks, checkpoint hashes, and snapshot identifiers can be grouped in the reproducibility details.

Populations with different means can have identical centered subspaces because the score intentionally ignores translation. “Discards amplitudes” means that replacing a retained spectrum with its orthogonal projector preserves the subspace but removes its singular-value weights. Projecting a vector still preserves its coefficients along the retained directions. A small score therefore identifies a geometric candidate without establishing functional equivalence to the identity. In the paper's example, a shared large mean can make raw tokenwise cosine approach one, and BI approach zero, despite orthogonal centered fluctuations. The construction shows a statistical distinction; the role of mean dominance in the evaluated model remains untested.

Measuring the relative energy of the empirical mean and centered fluctuations, together with a mean-centered BI ablation, would test that explanation. Those comparisons remain open. Centering alone also leaves a mathematical distinction: for paired x=ze1x = z e_1 and y=−ze1y = -z e_1 with equally probable z=±1z = \pm 1, both means are zero, the retained projectors coincide, but centered BI is two. Their relative practical value still needs empirical testing.

In the matched experiment, kk is the number of candidates in a shortlist. Its reported loss is the average of kk separate single-block deletions from the original model. With k=15k = 15, it is neither the loss after deleting all 15 nor the effect of the fifteenth-ranked block alone. The analysis evaluates all 48 singleton interventions and compares shortlists drawn from that common matrix. The geometric shortlist has mean loss change −0.1483 nats, compared with −0.1439 for BI; the paired contrast's interval, [−0.00970, 0.00064], leaves the advantage unresolved. All 15 geometric choices are explained by angular changes between rank-one subspaces, recur at all three calibration seeds, and avoid energy-capped endpoints. These results explain the selection mechanism without establishing a stronger ranking than BI.

The calibration data are 100 frozen assistant-message texts, from which both methods use the same 3,000 token positions. The evaluation questions are author-constructed from existing metallurgy values and intervals: 48 composition-recall, 48 strength-recall, and 48 composition-to-property questions, with fixed English references. Fourteen earlier questions are a separate secondary set. We chose this workload to examine selection for a concrete domain task. It was previously studied, so it is not an independently held-out benchmark, and overlap with training is unresolved. The measured endpoint is teacher-forced reference-token likelihood under a common reasoning-segment prompt, not free-generation accuracy. WikiText-2 and additional standard workloads would test broader language-model behavior that this experiment does not cover.

“Historical” refers to an earlier compression-and-adaptation run using different calibration sampling and a geometric set differing by one block. Its resource measurements do not jointly validate the matched shortlist. Multi-block interactions need a separate experiment; SLEB addresses such interactions by recomputing importance after removal (Song et al., ICML 2024). The present evidence characterizes the statistic on one model and domain against BI. Establishing a practical compression advantage requires joint deletion, centered BI, additional pruning baselines, and broader workloads. The earlier resource study serves as supporting context for that focused result.

I appreciate the authors' clarification to some of my raised questions. I'll maintain my rating, since several of the main concerns remain unaddressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.