acceptodds
Under review as a conference paper at ICLR 2027

STRIM: STRucture-aware Low-rank Inference Model

Abstract

Low-rank model compression has become an effective strategy for reducing the size of LLMs while maintaining performance. This paper introduces STRIM (STRucture-aware Low-rank Inference Model), a method for the low-rank compression of large language models. We start from a guiding block-level reconstruction view and derive a greedy, student-input-aware compression procedure. Compared with matrix-wise low-rank compression methods and recent student-input-aware approaches, STRIM explicitly accounts for compression-relevant architectural operators such as residual connections and normalization layers, which helps mitigate cascading compression errors. Experimental results show that STRIM consistently improves task accuracy by over 5% on six classification datasets (e.g., OpenbookQA, WinoGrande, HellaSwag) across a range of model families (e.g., Mistral, Gemma, Qwen) and compression ratios.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model compression, low-rank approximation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_CcYVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes STRIM for low-rank compression of large language models. The key idea is to optimize low-rank approximations at the transformer block level rather than decomposing each weight matrix independently. The method explicitly considers residual connections and normalization layers when constructing the compression objective. It then uses a greedy procedure to obtain efficient closed-form updates for individual weight matrices. The paper evaluates the method on several model families including Mistral, Gemma, and Qwen. Experiments cover different compression ratios and downstream classification tasks. The paper also studies post-compression optimization, instruction fine-tuning, adaptive rank selection, pruning, and quantization.

Critical strengths

1.Existing low-rank compression methods often optimize each weight matrix independently. Modeling the residual path and normalization within the block objective is a meaningful extension of this formulation. 2. The paper evaluates multiple model families and compression settings. The results also include post-compression optimization and adaptive rank allocation. This gives the proposed method reasonable empirical coverage.

Critical weaknesses
  1. The main motivation is that residual connections and normalization create dependencies that are ignored by matrix-wise compression. However, the experiments do not cleanly isolate the contribution of these architectural terms. A direct ablation of the proposed objective without residual connections and normalization would be important. Without this comparison, it is difficult to determine whether the improvement comes from the claimed structural modeling or from the overall optimization procedure.
  2. The paper formulates a joint block-level optimization problem but then replaces it with a greedy sequence of matrix-wise updates. This approximation is central to the method. However, the paper provides limited evidence that the greedy procedure is a good approximation to the original objective. In particular, it is unclear how sensitive the result is to the update order and whether different update orders lead to different compressed models. This weakens the connection between the proposed formulation and the actual algorithm.
Questions for authors
  1. How much of the reported gain comes specifically from modeling residual connections and normalization? Please compare the full method with a version using the same greedy optimization but removing these structural terms.
  2. How sensitive is STRIM to the order of the sequential weight updates? Does changing the update order affect reconstruction error or downstream accuracy?
  3. Can the authors provide evidence that the greedy solution is a reasonable approximation to the original joint optimization objective?
1 comment

We thank the reviewer for highlighting the need to connect the guiding block objective to the actual greedy algorithm.

1. Direct structural ablation

Table 2 already provides the requested comparison: it holds the greedy procedure, ranks, regularization, and OPT settings fixed and removes only structural correction. Since the evaluated Mistral architecture does not instantiate Eq. (6), this is specifically a residual-correction ablation. Without OPT, full STRIM versus no structural correction gives 6.94/52.80 versus 7.39/50.57 at ratio 0.8, 9.79/41.75 versus 13.38/38.71 at 0.6, and 19.47/35.12 versus 41.21/33.93 at 0.4. With OPT, the difference is smaller at mild compression but remains clear in perplexity and at stronger compression; at ratio 0.4 the results are 18.87/38.12 versus 28.77/36.56.

To cover normalization separately, we additionally ran Gemma-3-4B at ratio 0.8. With the same OPT schedule, full STRIM obtains 11.81 perplexity / 49.22 accuracy, compared with 12.37 / 44.00 without normalization treatment. Without OPT, the results are 12.18 / 42.19 and 14.93 / 39.87. We will revise the ablation discussion to make the residual-only interpretation of Table 2 explicit and add the Gemma normalization result.

2. Sequential update order

STRIM follows a topological order of the forward computation graph:

{q,k,v}→o→{gate,up}→down.\{q,k,v\}\rightarrow o\rightarrow\{\mathrm{gate},\mathrm{up}\}\rightarrow\mathrm{down}.

All valid topological orders differ only in the ordering of matrices inside the braces. These matrices are parallel branches that share the same input and do not consume one another's outputs, so their updates construct the same conditional regression problems and commute. Thus, there is no additional ordering heuristic or meaningful sensitivity among valid topological orders. In contrast, a non-topological order would compress a downstream matrix before the upstream matrices on which it depends. For example, compressing o_proj before Q/K/V would fit it before the Q/K/V compression errors exist, thereby omitting precisely the propagated error that the student-input-aware greedy update is intended to correct. Such an order defines a different, causally incomplete procedure rather than another valid ordering of STRIM. We will state this dependency and the allowable parallel permutations explicitly in Algorithm 1.

3. Relation between the greedy solution and the joint objective

The block objective is a guiding formulation; STRIM does not claim to solve the joint rank-constrained multi-matrix problem or to provide a bound on its optimum. Joint optimization couples several low-rank factors through attention, nonlinearities, residual paths, and normalization and does not admit the closed form used by STRIM. Instead, STRIM performs block-coordinate-style greedy compression: at each step, it runs the current partially compressed model, constructs the target induced by the relevant architectural operators, and solves that conditional rank-constrained regression problem in closed form. Thus, each local update is exact for its current conditional objective, but proximity to the global joint optimum is not guaranteed.

We also provide direct empirical evidence for the approximation quality. On Mistral-7B at ratio 0.8, SVD-LLM has 0.46 Rel. RMSE, while STRIM's regularized greedy closed-form initialization reaches approximately 0.27 before iterative refinement. Direct optimization of the block reconstruction objective further reduces this to 0.25 after 10 epochs and 0.2357 after 100 epochs. Thus, the greedy initialization captures approximately 85% of the observed reconstruction-error improvement from SVD-LLM to the 100-epoch refined solution without iterative optimization. The no-OPT ablation provides complementary downstream evidence: the structure-aware greedy initialization consistently improves over the otherwise identical matrix-output target, especially under aggressive compression. This is empirical support rather than a guarantee of proximity to the unknown global joint optimum, and we will state this limitation directly.

Reviewer_1ND4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper introduces STRIM, a structure-aware method for low-rank LLM compression that reduces errors accumulated during sequential compression. It uses inputs from the partially compressed model and accounts for residual connections and normalization, combining a regularized closed-form initialization with local refinement. Experiments on Mistral, Gemma, and Qwen show better task accuracy under the same parameter budget, with support for quantization and different rank-allocation strategies.

Critical strengths
  1. The motivation makes sense, STRIM uses inputs from the partially compressed model and explicitly accounts for residual connections and normalization to reduce accumulated compression errors.
  2. The experiments include comparisons with matched ranks and optimization budgets, along with ablations of structural correction and regularization, helping establish the value of the initialization.
Critical weaknesses
  1. The normalization treatment assumes similar output RMS values before and after compression, but this approximation is not directly validated. The main structural ablation uses Mistral and therefore only tests residual correction.
  2. Structural correction does not always clearly improve final accuracy. After OPT, it adds only 0.05 and 0.16 percentage points at parameter retention ratios of 0.8 and 0.6.
  3. The comparison with the closely related AA-SVD covers only Mistral and uses different regularization and optimization settings, making the source of the gains unclear.
Questions for authors
  1. How different are the output RMS values before and after compression, especially at high compression ratios? Could the authors separately ablate the normalization correction on Gemma or Qwen?
  2. How much of STRIM’s advantage remains if AA-SVD uses the same original-weight regularization and optimization budget? This would help isolate the contribution of structural correction.
  3. In Table 4, the two methods reach almost identical reconstruction errors but still differ in downstream accuracy. Does this pattern hold with a separate calibration set, or does the reconstruction metric miss errors that matter more for downstream tasks?
2 comments

We thank the reviewer for focusing on the normalization approximation and controlled comparison. We ran the requested Gemma normalization ablation and RMS diagnostic.

1. Normalization ablation and RMS diagnostic

At parameter ratio 0.8 on Gemma-3-4B, full STRIM obtains 11.81 perplexity / 49.22 six-task accuracy with OPT, compared with 12.37 / 44.00 when all normalization-aware treatment is disabled. Without OPT, the corresponding results are 12.18 / 42.19 and 14.93 / 39.87. An intermediate variant that disables only QK-Norm treatment obtains 11.81 / 48.31 with OPT and 14.02 / 40.20 without OPT. All variants use the same ranks, regularization, calibration data, and OPT budget.

The fixed full-model RMS is not only a tractability approximation but also anchors the regression to a stable reference: a distorted or collapsed student RMS cannot determine its own loss scaling during compression. Since inference still uses the actual student RMS, we measured the closed-form, pre-OPT models using

∣RMS⁡(zstudent)−RMS⁡(zteacher)∣RMS⁡(zteacher)\frac{|\operatorname{RMS}(z_{\mathrm{student}})-\operatorname{RMS}(z_{\mathrm{teacher}})|}{\operatorname{RMS}(z_{\mathrm{teacher}})}

over 128 independently sampled length-2048 sequences. At ratio 0.8, the mean discrepancy is 5.06%, 4.46%, 8.55%, and 8.33% for q_proj, k_proj, o_proj, and down_proj, respectively, compared with baseline uniform-weight errors of 19.12%, 18.06%, 17.53%, and 18.11%. This baseline omits normalization-aware token weighting. At ratio 0.4, the RMS discrepancies increase to 8.55%, 8.53%, 12.73%, and 14.27%, while the corresponding baseline errors are 19.65%, 18.15%, 16.95%, and 19.05%. This confirms that the full-model RMS is a useful but imperfect proxy, particularly under aggressive compression. We will include both the diagnostic and the component ablation in the revision.

2. Contribution of structural correction under matched settings

To isolate structural correction without confounding it with regularization or optimization differences, Table 2 removes residual correction on Mistral while holding the student-input-aware procedure, original-weight regularization, rank allocation, and OPT budget fixed. Our new Gemma-3-4B ablation similarly isolates normalization treatment: at ratio 0.8, disabling it reduces accuracy from 49.22 to 44.00 with OPT and from 42.19 to 39.87 without OPT, while worsening perplexity from 11.81 to 12.37 and from 12.18 to 14.93, respectively. Together, these matched ablations directly isolate the contributions of residual and normalization correction from the remaining pipeline.

3. Similar reconstruction errors but different downstream accuracy

To assess whether the discrepancy extends beyond the calibration objective, we compare held-out perplexity across the same optimization trajectory. STRIM and SVD-LLM+OPT have identical perplexity through epochs 10, 20, and 50 (6.73, 6.62, and 6.57), while STRIM consistently achieves higher six-task accuracy. At epoch 100, SVD-LLM+OPT has both slightly lower calibration Rel. RMSE (0.2353 versus 0.2357) and lower perplexity (6.52 versus 6.57), yet STRIM retains higher six-task accuracy (56.57 versus 55.14). Thus, local Rel. RMSE appears more aligned with perplexity than with classification accuracy and does not capture all task-relevant differences. Importantly, this divergence strengthens the case for STRIM's initialization: its downstream advantage cannot be explained merely by lower local reconstruction error, and it preserves classification-relevant behavior that prolonged optimization of that error does not recover.

Thank you for the additional experiments and clarifications. They address my main concerns. I have no further questions.

Reviewer_py2ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes STRIM, a low-rank compression method for LLMs that accounts for shifted student inputs and Transformer structures, including residual connections and RMSNorm. The method uses a regularized closed-form solution followed by optional block-level optimization. Experiments across several model families show improvements over existing compression baselines.

Critical strengths
  1. Well-motivated approach. Accounting for shifted student inputs and residual errors is reasonable. The resulting regression formulation is simple and admits an efficient closed-form solution.

  2. Good empirical results. STRIM shows clear improvements over several baselines, especially under aggressive compression. The evaluation covers multiple model families and compression ratios.

Critical weaknesses
  1. Limited novelty and unclear contribution attribution. Student-aware regression and block-level refinement have already been explored in AA-SVD. The main novelty of STRIM therefore seems to lie in its structural corrections and regularization. However, their individual contributions are not fully clear. In Table 2, removing structural correction after OPT reduces classification accuracy by only 0.05 and 0.16 percentage points at parameter ratios of 0.8 and 0.6, respectively, although the improvements in PPL and the no-OPT setting are more noticeable. It remains unclear how much of the final performance gain over existing methods comes specifically from the proposed structural corrections.

  2. Insufficient justification of the structure-aware formulation. The treatment of residual connections and RMSNorm needs further clarification. For architectures such as Gemma3, normalization and residual addition can occur together, but Algorithm 1 prioritizes normalization without explicitly accounting for the residual discrepancy. It is unclear how this choice relates to the original block-level reconstruction objective. In addition, the formulation does not clearly explain how head-wise QK-Norm is handled in models such as Qwen3. The RMS approximation used in Eq. (6) is also not validated, and a separate ablation of RMSNorm correction is missing.

Questions for authors
  1. Could the authors separately ablate residual correction, RMSNorm correction, and regularization under matched rank allocation and OPT settings to better clarify their individual contributions?

  2. How does the method handle cases where normalization and residual connections coexist, as well as head-wise QK-Norm? How accurate is the RMS approximation in Eq. (6) under aggressive compression?

  3. Could the authors report the actual compression time compared with SVD-LLM and AA-SVD, including the costs of closed-form initialization and OPT? This would help assess the additional offline computational overhead of STRIM.

3 comments

We thank the reviewer for identifying the need to separate the structural components and clarify the normalization implementation. We ran additional Gemma-3-4B experiments and provide the requested details below.

1. Separate structural and regularization ablations

The existing Mistral ablation in Table 2 is already a residual-only ablation, because Mistral does not instantiate the normalization case in Eq. (6). It compares the complete method with exactly the same greedy procedure, rank allocation, regularization, and OPT schedule after removing only residual correction. The benefit is especially clear before OPT and under stronger compression. For example, without OPT, removing residual correction changes perplexity/accuracy from 6.94/52.80 to 7.39/50.57 at ratio 0.8, and from 19.47/35.12 to 41.21/33.93 at ratio 0.4. With OPT, the accuracy difference is small at ratios 0.8 and 0.6, as the reviewer notes, but residual correction still improves perplexity from 6.89 to 6.73 and from 10.26 to 9.20; its effect becomes larger at ratio 0.4 (28.77/36.56 versus 18.87/38.12).

To isolate normalization, we added the following Gemma-3-4B ablation at ratio 0.8. All rows use identical ranks, regularization, calibration data, and, where applicable, the same 10-epoch OPT schedule.

MethodWith OPT PPL / Acc.Without OPT PPL / Acc.
STRIM11.81 / 49.2212.18 / 42.19
without QK-Norm treatment11.81 / 48.3114.02 / 40.20
without any normalization treatment12.37 / 44.0014.93 / 39.87

This separates the approximate QK-Norm treatment from the post-projection normalization treatment used for o_proj and down_proj. Both contribute, while the complete normalization treatment is particularly important after OPT.

Finally, Appendix Tables 10 and 11 separately isolate regularization at Mistral ratio 0.8 without and with OPT, respectively. In the matched-OPT setting of Table 11, original-weight-anchored regularization obtains 55.89, compared with 53.62 without regularization and 53.77 with separately tuned conventional zero-centered regularization. We will revise the text to make clear that Table 2 isolates residual correction, the new Gemma experiment isolates normalization, and Appendix Tables 10 and 11 isolate regularization.

2. Residual/normalization coexistence, QK-Norm, and approximation accuracy

We agree that the case distinction should be more explicit. For standard pre-norm blocks such as Mistral, Eq. (5) is applied only to the attention output projection and MLP down projection, where the projected branch is added directly to the residual stream. For Gemma-3, o_proj and down_proj outputs are normalized before residual addition. Our reported implementation therefore applies the normalization-aware target to the normalized branch; it does not simultaneously add the separate residual-discrepancy target used for Mistral. The residual bypass remains in the forward computation, but any teacher-student discrepancy in that bypass is not explicitly corrected by this update. This is a greedy approximation to the block objective rather than an exact joint treatment of both operators, and we will state this explicitly.

Gemma-3 and Qwen3 also use head-wise QK-Norm. The current closed-form solver uses one shared low-rank projection and a practical scale approximation over the concatenated projection output; it does not solve a separate reduced-rank regression for each head. We will correct the presentation so that it does not imply exact head-wise optimization. The new without QK-Norm treatment row above directly measures the value of this approximation.

Using the full-model RMS as the fixed scale is also an intentional robustness choice: it prevents a distorted or collapsed student RMS from determining its own regression weighting or normalizing away magnitude errors during compression, while preserving a closed-form problem. At inference, RMSNorm instead uses the actual student RMS, so we directly measure how well the full-model RMS preserves its relative token weighting.

We further measured the RMS approximation per normalization unit, using the true head partition for Q/K, on 128 independently sampled length-2048 sequences. The table reports the mean relative RMS discrepancy:

Ratioq_projk_projo_projdown_proj
0.85.06%4.46%8.55%8.33%
0.48.55%8.53%12.73%14.27%

For comparison, the baseline uniform-weight reference, which omits normalization-aware token weighting, produces mean relative errors of 19.12%, 18.06%, 17.53%, and 18.11% at ratio 0.8, and 19.65%, 18.15%, 16.95%, and 19.05% at ratio 0.4, in the same projection order. Thus, the full-model RMS provides a closer approximation than this non-normalization-aware reference, although the discrepancy increases under aggressive compression. The component ablation above directly verifies that the resulting normalization-aware targets improve perplexity and accuracy at ratio 0.8.

3. Compression cost

On Mistral-7B at parameter ratio 0.8, using 256 calibration sequences of length 2048 on one NVIDIA A6000 and excluding downstream evaluation, compression took approximately 5 h 45 min for SVD-LLM, 6 h 15 min for STRIM with 10 OPT epochs, and 6 h 49 min for AA-SVD with its recommended 25-epoch refinement. Thus, STRIM adds a modest approximately 9% wall-clock cost over SVD-LLM while remaining comparable to AA-SVD. Because these methods use separate implementations, we treat these measurements as practical cost estimates rather than evidence of an implementation-independent runtime advantage. The closed-form stage has the same asymptotic reduced-rank-regression cost as SVD-LLM, and constructing residual- or normalization-adjusted targets does not add another decomposition. STRIM also changes only the compressed weights and therefore introduces no additional inference-time operations relative to the same low-rank architecture.

Thank you for the detailed response and additional experiments. While the new results address several of my concerns, I still have reservations about the novelty of the method and the approximations used in its structural corrections. So I will maintain my rating.

Reviewer_hgR3OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

STRIM proposes a method to improve student-input-aware low-rank compression for Large Language Models.

The authors propose that these low-rank compression methods need to consider mismatches between the teacher and student hidden states.

The authors argue that compression of linear-layers, residual connections, normalization and regularaization can be casted as regression problems of the form min⁡Wˉhˉ;rank(Wˉhˉ)≤r∥WˉhˉX−Y∥F2+η∥Wˉhˉ−Wh∥F2\min_{\bar{W}_{\bar{h}; \text{rank}(\bar{W}_{\bar{h}})} \leq r} \| \bar{W}_{\bar{h}} X -Y \|_F^2 + \eta \| \bar{W}_{\bar{h}} - W_h \|_F^2. Then, they provide an optimal solution for this formulation in Theorem 4.1.

Their Algorithm 1, uses this optimal solution to obtain optimaly compressed weights Wˉhˉ\bar{W}_{\bar{h}}.

The authors performed experiments using common benchmarks for Mistral, Gemma and Qwen models. They also show examples where STRIM can be combined with other optimization methods.

This work is interesting and the results presented suggest that STRIM is a framework with the potential to improve the performance of compressed models.

Critical strengths

STRIM is a method that is mathematically closed form and provides better initialization. This can be interesting for LoRA/OPT fine tuning. Additionally, compatibility with quantization makes it more attractive.

Critical weaknesses

None

Questions for authors

Regarding Normalization, LLMs exhibit structured activation outliers concentrated in few channels. This channels dominate the RMS in RMSNorm. Low rank compression errors that perturb the outlier magnitude change the normalizations scale. So, small errors in few channels spread across the representation. How well does STRIM handle low-rank compression errors of outliers?

1 comment

We thank the reviewer for the positive assessment and the important question about structured activation outliers.

An important feature of STRIM is that it uses the full-model RMS as a fixed reference scale. To approximate post-RMSNorm reconstruction while retaining a closed-form solution, STRIM minimizes the following weighted error between the student and full-model pre-normalization outputs:

{\operatorname{RMS}(z_{\mathrm{teacher}})}\right\|_F^2.$$ Structured outlier channels are already present in the full-model activations, and STRIM does not remove or separately reweight them. Using the full-model RMS as the denominator means that a mismatch in a dominant student channel remains penalized in the numerator, but cannot also alter the normalization scale used by the closed-form regression. The approximation therefore omits the feedback of this mismatch through the student's actual RMSNorm denominator. At inference, RMSNorm necessarily uses the actual student RMS. We therefore added a diagnostic on the closed-form, pre-OPT Gemma-3-4B models to measure how well the fixed-reference approximation holds. For each normalization unit, we compute $$d=\frac{|\operatorname{RMS}(z_{\mathrm{student}})-\operatorname{RMS}(z_{\mathrm{teacher}})|}{\operatorname{RMS}(z_{\mathrm{teacher}})}.$$ We evaluate 128 independently sampled WikiText-2 sequences of length 2048. At ratio 0.8, the mean RMS discrepancies are 8.55% for `o_proj` and 8.33% for `down_proj`. For comparison, the baseline uniform-weight reference, which omits normalization-aware token weighting, produces mean relative errors of 17.53% and 18.11%, respectively. At the more aggressive ratio 0.4, the RMS discrepancies increase to 12.73% and 14.27%, while the corresponding baseline uniform-weight errors are 16.95% and 19.05%. The full-model RMS therefore provides a substantially closer approximation than the non-normalization-aware reference, although its accuracy decreases under more aggressive compression. We also directly ablated normalization-aware target construction on Gemma-3-4B at ratio 0.8. With the same 10-epoch OPT schedule, full STRIM obtains **11.81 perplexity / 49.22 accuracy**, whereas disabling normalization treatment gives **12.37 / 44.00**. Without OPT, the corresponding results are **12.18 / 42.19** and **14.93 / 39.87**. These results show that the fixed-reference normalization-aware objective materially improves compression quality. We will add both results and clarify that STRIM does not model the omitted student-RMS feedback exactly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.