- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes STRIM for low-rank compression of large language models. The key idea is to optimize low-rank approximations at the transformer block level rather than decomposing each weight matrix independently. The method explicitly considers residual connections and normalization layers when constructing the compression objective. It then uses a greedy procedure to obtain efficient closed-form updates for individual weight matrices. The paper evaluates the method on several model families including Mistral, Gemma, and Qwen. Experiments cover different compression ratios and downstream classification tasks. The paper also studies post-compression optimization, instruction fine-tuning, adaptive rank selection, pruning, and quantization.
Critical strengths
1.Existing low-rank compression methods often optimize each weight matrix independently. Modeling the residual path and normalization within the block objective is a meaningful extension of this formulation. 2. The paper evaluates multiple model families and compression settings. The results also include post-compression optimization and adaptive rank allocation. This gives the proposed method reasonable empirical coverage.
Critical weaknesses
- The main motivation is that residual connections and normalization create dependencies that are ignored by matrix-wise compression. However, the experiments do not cleanly isolate the contribution of these architectural terms. A direct ablation of the proposed objective without residual connections and normalization would be important. Without this comparison, it is difficult to determine whether the improvement comes from the claimed structural modeling or from the overall optimization procedure.
- The paper formulates a joint block-level optimization problem but then replaces it with a greedy sequence of matrix-wise updates. This approximation is central to the method. However, the paper provides limited evidence that the greedy procedure is a good approximation to the original objective. In particular, it is unclear how sensitive the result is to the update order and whether different update orders lead to different compressed models. This weakens the connection between the proposed formulation and the actual algorithm.
Questions for authors
- How much of the reported gain comes specifically from modeling residual connections and normalization? Please compare the full method with a version using the same greedy optimization but removing these structural terms.
- How sensitive is STRIM to the order of the sequential weight updates? Does changing the update order affect reconstruction error or downstream accuracy?
- Can the authors provide evidence that the greedy solution is a reasonable approximation to the original joint optimization objective?
We thank the reviewer for highlighting the need to connect the guiding block objective to the actual greedy algorithm.
1. Direct structural ablation
Table 2 already provides the requested comparison: it holds the greedy procedure, ranks, regularization, and OPT settings fixed and removes only structural correction. Since the evaluated Mistral architecture does not instantiate Eq. (6), this is specifically a residual-correction ablation. Without OPT, full STRIM versus no structural correction gives 6.94/52.80 versus 7.39/50.57 at ratio 0.8, 9.79/41.75 versus 13.38/38.71 at 0.6, and 19.47/35.12 versus 41.21/33.93 at 0.4. With OPT, the difference is smaller at mild compression but remains clear in perplexity and at stronger compression; at ratio 0.4 the results are 18.87/38.12 versus 28.77/36.56.
To cover normalization separately, we additionally ran Gemma-3-4B at ratio 0.8. With the same OPT schedule, full STRIM obtains 11.81 perplexity / 49.22 accuracy, compared with 12.37 / 44.00 without normalization treatment. Without OPT, the results are 12.18 / 42.19 and 14.93 / 39.87. We will revise the ablation discussion to make the residual-only interpretation of Table 2 explicit and add the Gemma normalization result.
2. Sequential update order
STRIM follows a topological order of the forward computation graph:
{q,k,v}→o→{gate,up}→down.
All valid topological orders differ only in the ordering of matrices inside the braces. These matrices are parallel branches that share the same input and do not consume one another's outputs, so their updates construct the same conditional regression problems and commute. Thus, there is no additional ordering heuristic or meaningful sensitivity among valid topological orders. In contrast, a non-topological order would compress a downstream matrix before the upstream matrices on which it depends. For example, compressing
o_projbefore Q/K/V would fit it before the Q/K/V compression errors exist, thereby omitting precisely the propagated error that the student-input-aware greedy update is intended to correct. Such an order defines a different, causally incomplete procedure rather than another valid ordering of STRIM. We will state this dependency and the allowable parallel permutations explicitly in Algorithm 1.3. Relation between the greedy solution and the joint objective
The block objective is a guiding formulation; STRIM does not claim to solve the joint rank-constrained multi-matrix problem or to provide a bound on its optimum. Joint optimization couples several low-rank factors through attention, nonlinearities, residual paths, and normalization and does not admit the closed form used by STRIM. Instead, STRIM performs block-coordinate-style greedy compression: at each step, it runs the current partially compressed model, constructs the target induced by the relevant architectural operators, and solves that conditional rank-constrained regression problem in closed form. Thus, each local update is exact for its current conditional objective, but proximity to the global joint optimum is not guaranteed.
We also provide direct empirical evidence for the approximation quality. On Mistral-7B at ratio 0.8, SVD-LLM has 0.46 Rel. RMSE, while STRIM's regularized greedy closed-form initialization reaches approximately 0.27 before iterative refinement. Direct optimization of the block reconstruction objective further reduces this to 0.25 after 10 epochs and 0.2357 after 100 epochs. Thus, the greedy initialization captures approximately 85% of the observed reconstruction-error improvement from SVD-LLM to the 100-epoch refined solution without iterative optimization. The no-OPT ablation provides complementary downstream evidence: the structure-aware greedy initialization consistently improves over the otherwise identical matrix-output target, especially under aggressive compression. This is empirical support rather than a guarantee of proximity to the unknown global joint optimum, and we will state this limitation directly.