acceptodds
Under review as a conference paper at ICLR 2027

The Behavioral Rate of Fine-Tuning: Compression as a Measure of Useful Information

Abstract

Low-Rank Adaptation (LoRA) adapts a frozen language model through a compact learned update, making it a natural setting for asking how much information fine-tuning actually needs to write. Yet adapter rank, parameter count, or numerical reconstruction error do not directly tell us how many bits are required to preserve the behavior acquired during training. We therefore define an operational behavioral rate as the exact serialized size of a compressed adapter. It is the smallest number of bits that, once reloaded into the frozen base model, retains a fixed fraction of the adapter's held-out fine-tuning gain. Across a broad set of LoRA fine-tuning experiments, we find that this rate increases with distinct training content at fixed optimization budgets, but is not determined by source description length, fine-tuning gain, or ordinary weight-space reconstruction error. Most importantly, the same datasets exhibit substantially different, and even reordered, rates across frozen base models, showing that fine-tuning compressibility is a property of the dataset–receiver pair rather than of the dataset alone. Receiver-relative correction spectra are our strongest predictor of behavior-preserving rate across models and tasks, while rank and layer controls reveal an additional dependence on the adapter carrier. Finally, in controlled experiments with deliberately corrupted reasoning supervision, aggressive compression can recover useful task performance from otherwise harmful updates, beyond what can be explained by simple shrinkage, with substantially stronger recovery when more adaptation rank is available during training. Together, these results suggest that the information cost of fine-tuning is not an intrinsic property of the data or of parameter displacement, but a receiver- and behavior-relative property of how the required corrections are represented in the learned update.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Fine-tuning compression, LoRA, Rate-distortion

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8VzQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper measures the smallest serialized LoRA representation that preserves 90% of a fine-tuning gain. It studies receiver-dependent compression rates, recovery from corrupted supervision, and a retrospective relationship between rate and update shrinkability.

Critical strengths

Near one bit per value, the lower-RMSE code retains about 3% of the gain versus 86% for uniform coding, demonstrating that weight reconstruction error can mislead. Recovery is tested across models with shrinkage, rank, random-projection, and off-task controls. The retrospective shrinkability fit is also promising: RMSE 0.068 bits/value on 42 checkpoints from five additional receivers (Figure 7).

Critical weaknesses
  1. The central measurement needs a fuller audit trail. The complete rate grid, per-adapter retention curves, and raw gains behind Figure 6 are missing. Selected operating points do not establish that corpus reorderings survive threshold uncertainty, especially when fine-tuning gains are small.

  2. The local minimizer claim lacks a curvature condition. Invertibility alone does not make −H_T⁻¹μ a minimum (Appendix B.3, Equation 11). The claim needs positive definiteness on the working subspace; singular cases also need compatible gradients or appropriate damping. This is a repairable theoretical error.

  3. The theory does not explain aggressive compression recovery. The implemented predictor omits the curvature matrices in the derivation, and Table 5 reports failure of the local expansion below one bit per value. The valid Gaussian rate-distortion result therefore remains a motivation rather than an explanation of the measured effect.

  4. Locked pretraining predictors fail to transfer. All three have worse RMSE than the base-code-length comparator on new panels (Appendix A, Table 3). The promising post-training shrinkability fit is retrospective; it does not establish prospective or pretraining prediction.

Questions for authors
  1. Which evaluated rates and retention curves produce Figure 6, and do its ordering changes survive uncertainty in the raw gains and thresholds?
  2. After correcting the curvature assumptions, which theoretical conclusion is supported at the reported operating rates?
  3. Figure 11 shows substantial recovery from shrinkage. Under matched validation-based selection, how much does coding improve over the best scalar-rescaled full-precision rank-1 update?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 8VzQ (part 1 of 3)

Thank you for the detailed review and for checking the theory closely. We answer each point below. Figure and table numbers refer to the revised paper.

W1 and Q1. Audit trail and uncertainty for Figure 7 (Figure 6 in the submission).

The central receiver-dependent ordering holds true in your requested audit of rate conventions, retention thresholds, and evaluation uncertainty.

We reconstructed the complete evaluation grid behind Figure 7. The revised manuscript includes all 60 retention curves (Fig. 12), raw-gain statistics, and the sensitivity analyses across rate conventions and thresholds (Appendix D, Tables 6–8), and a per-checkpoint table of gains and rate thresholds (Table 9). The underlying per-file numerical data will be released upon publication.

We also explicitly separated four operational conventions:

  1. Interpolated threshold crossing, using the best observed gain as reference.
  2. Interpolated threshold crossing, using the uncompressed adapter's gain as reference.
  3. First measured compressed file meeting the threshold, using the best observed gain.
  4. First measured compressed file meeting the threshold, using the uncompressed adapter's gain.

This distinction makes explicit that an interpolated threshold estimate and the smallest measured reloadable file are not interchangeable quantities. We have aligned our revised description of Equation (2) with the reported operational convention and label interpolated estimates separately.

We recomputed the Figure 7 comparisons using four retention thresholds (50%, 75%, 90%, 95%) and these four rate conventions.

We find a stable principal result: XBRL remains the least costly task for the same six of seven receivers in all 16 combinations. Qwen2.5-7B remains the exception, with math cheaper than XBRL in the central estimates.

To evaluate uncertainty, we re-evaluated the 60 checkpoints and performed 10,000 resamples of the evaluation batches. At the 90% retention threshold, depending on the rate convention:

ComparisonOrderings resolved at the 95% level
Task ordering within a receiver17–19 of 19
Receiver ordering within a task51–56 of 57
Task-order reversals between receivers41–51 of 51

These calculations explicitly propagate evaluation-sample uncertainty through the retention threshold. The bootstrap is conditional on the trained checkpoints; variability across training seeds is reported separately in Table 7.

For the specific Qwen2.5-7B exception, the bootstrap support for math requiring fewer bits than XBRL ranges from 0.92 to 1.00 across conventions. Thus, the central ordering is stable, while not every convention resolves this particular comparison at the 95% confidence level.

We have expanded the appendix with the complete rate grid and retention curves so that the ordering claims can be checked directly rather than inferred from selected operating points. These results support the main Figure 7 conclusion while making the limitations of close comparisons and threshold estimation transparent.

Table 7. R* at 90% retention in bits per value: mean over three training seeds, with the lowest and highest seed in parentheses. Qwen3-8B has no math adapter.

ModelXBRLMathCode
Gemma2-9B-IT0.28 (0.25–0.33)0.89 (0.88–0.91)0.99 (0.98–1.00)
Gemma2-9B0.34 (0.32–0.35)0.87 (0.84–0.89)0.88 (0.87–0.90)
Mistral-7B0.37 (0.32–0.45)0.75 (0.70–0.79)0.90 (0.86–0.92)
Llama3.1-8B0.38 (0.36–0.40)0.93 (0.91–0.94)1.02 (0.97–1.07)
Qwen2.5-7B0.48 (0.46–0.49)0.41 (0.38–0.44)0.67 (0.63–0.71)
Qwen3-8B0.53 (0.51–0.55)—0.87 (0.84–0.90)
Qwen2.5-Math-7B0.62 (0.61–0.62)1.03 (0.97–1.15)1.28 (1.23–1.32)
AuthorsOpenReview ↗

Response to Reviewer 8VzQ (part 2 of 3)

Table 8. Number of orderings resolved at the 95% level at 90% retention, under each of the four rate conventions (10,000 resamples of the evaluation batches).

Orderings resolved at 95%Interpolated rate, best referenceInterpolated rate, raw referenceMeasured file, best referenceMeasured file, raw reference
Task order within a model18 of 1919 of 1917 of 1919 of 19
Model order within a task55 of 5756 of 5752 of 5751 of 57
Task-order reversals between models46 of 5151 of 5141 of 5151 of 51

W2. Curvature condition.

Thank you for identifying the missing curvature assumption. We agree that invertibility alone does not guarantee that the stationary point is a minimum.

For the quadratic training-loss approximation on a working subspace SS,

q(δ)=μ⊤δ+12δ⊤HTδ,q(\delta)=\mu^\top\delta+\frac12\delta^\top H_T\delta,

the expression

δ∗=−HT−1μ\delta^\ast=-H_T^{-1}\mu

is the unique minimizer when the restriction of HTH_{T} to SS is positive definite, rather than merely invertible.

We have made this assumption explicit in Appendix B.3.

We also clarify the singular case; if the restricted Hessian is positive semidefinite, a finite minimizer exists only when the restricted gradient lies in its range; in that case the minimizers can be expressed using the pseudoinverse, up to an arbitrary null-space component.

When damping is used, (HT+λI)−1(H_{T} + \lambda I)^{- 1} corresponds to a regularized quadratic problem (which we now distinguish from the original undamped objective).

With these corrections, we repair the local-minimizer statement without changing our empirical measurements or the central interpretation of correction-gradient second moments as geometric descriptors.

W3 and Q2. What does the theory actually explain at the aggressive compression rates used?

With our new results, we can identify a concrete finite-rate mechanism behind the observed recovery, while keeping a precise scope for our theoretical claims. For a rank-1 update DD and its one-bit coded version Q(D)Q(D), define the aligned coefficient

γ=⟨Q(D),D⟩∥D∥F2.\gamma=\frac{\langle Q(D),D\rangle}{\lVert D\rVert_F^2}.

The coded update then admits the exact decomposition

Q(D)=γD+E,⟨E,D⟩=0.Q(D)=\gamma D+E,\qquad\langle E,D\rangle=0.

For the rank-1 sign-coding construction, the gain can also be related to the alignment properties of the quantized factors. Under Gaussian-factor assumptions, the corresponding expected aligned gain is approximately 0.405; empirically, we measure approximately 0.400 across 196 layers (Section 4.3).

This decomposition by itself does not predict task recovery; hence our crucial additional evidence is the observed behavioral response as a function of gain. For corrupted Qwen2.5-7B adapters, the leading full-precision singular direction achieves its best SVAMP performance around gains 0.2–0.3, whereas the original trained gain of 1 is strongly detrimental (Fig. 17a).

Furthermore, validation-selected full-precision rescaling achieves accuracy comparable to coding (Fig. 16), supporting attenuation as the principal observed mechanism. The residual (E) can nevertheless affect task performance.

This provides an empirical explanation of the dominant recovery mechanism, that is an effective attenuation of the learned update, supported by an exact parameter-space decomposition.

A one-dimensional quadratic surrogate can motivate why intermediate gains may outperform both zero gain and the trained gain. However, we do not claim that this surrogate gives a quantitative recovery guarantee for autoregressive language models at one bit per value. In particular, orthogonality in weight space does not imply orthogonality in the task-loss geometry.

We retain the Gaussian rate-distortion derivation as motivation for receiver-relative functional geometry, not as a quantitative theory of the compressed checkpoints or of recovery at extreme rates.

This distinction addresses the regime-of-validity concern raised in the review.

AuthorsOpenReview ↗

Response to Reviewer 8VzQ (part 3 of 3)

W4. Retrospective versus prospective prediction

After additional experiments, we now have prospective evidence for our post-training shrinkability rule, with predictions frozen before measuring the corresponding behavioral rates.

In a new six-checkpoint evaluation, the rule achieves a rate-prediction RMSE of 0.080 bits/value, compared with 0.105 for a constant-rate baseline and 0.093 for a task-mean baseline (Appendix F).

Additional evaluations across further adapters and reparameterizations yield RMSE values between 0.082 and 0.094, consistent with this result. These checks contain correlated evaluations of related adapters, which we clearly distinguish from our six independently evaluated new checkpoints.

The original Figure 8 relationship (Figure 7 in the submission) achieved 0.068 bits/value RMSE retrospectively. Our new results therefore provide an additional prospective test of the already fixed post-training rule.

Of course, this must be clearly separated from prediction before finetuning, since our predictor uses the trained update and a small number of shrinkability evaluations. We also carried out a new prospective pretraining-prediction evaluation, freezing a revised predictor before training six adapters on two new models. Its RMSE was 0.253 bits/value, compared with 0.263 for a text-length baseline. Although the task orderings were correct for both new receivers, the predictors showed systematic underestimation of approximately 0.21 bits/value (Appendix F).

We therefore do not claim accurate absolute prediction before fine-tuning, which remains a substantially harder open problem.

We can still confidently claim that once an update has been learned, the shrinkability-based rule now has prospective evidence beyond the retrospective fit reported in the submission.

We have made this distinction explicit in the revised text.

Q3. Coding vs the best scaled rank-1 update.

Our matched-selection experiments indicate that coding achieves comparable recovery accuracy, rather than a statistically established improvement over tuned and rescaled rank-1 updates.

Under the same validation-based selection protocol, the paired GSM8K differences are:

ComparisonCoded minus scaled accuracy95% CI
1-bit rank-1 code vs selected fp16 rank-1 scaling+0.15 pp[−0.99, +1.29]
2-bit rank-1 code vs selected fp16 rank-1 scaling+0.76 pp[−0.53, +2.05]

Neither interval establishes a statistically significant advantage of coding in task accuracy (Fig. 16a).

This directly answers the question: most of the recovery can be explained by an appropriate reduction of effective gain along the retained direction.

The remaining contribution of compression is nevertheless operationally important. A small coded representation achieves this recovery without any costly search over scalar gains at deployment.

On held-out GSM8K, the untuned rank-1 one-bit code is within one percentage point of the best of 120 searched scalar gains on each of Qwen2.5-7B and Qwen2.5-3B (Fig. 18a). In an illustrative comparison, the coded adapter obtains 82.8%, the best validation-selected scaled update 83.2%, and the clean fine-tune 83.0%.

We therefore distinguish two conclusions:

Mechanistic: gain attenuation accounts for most of the observed recovery, and we do not claim a significant additional accuracy benefit from quantization.

Operational: the compact coded update realizes a favorable behavioral operating point with much lower storage and without a separate gain-optimization procedure.

We believe this is a more precise and informative characterization of the results than attributing recovery to an unverified selective-removal effect.

We believe these additions resolve the principal concerns about measurement transparency and theoretical scope while strengthening the paper's empirical contribution.

Reviewer_26rZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how much information is required to preserve the behavior acquired through fine-tuning. The authors introduce behavioral rate, defined as the minimum compressed LoRA adapter size needed to retain a specified fraction of the held-out fine-tuning gain. Through experiments across different models and datasets, they show that behavioral rate depends on the dataset–receiver pair rather than the dataset alone, and that weight reconstruction error does not reliably reflect behavioral preservation. The paper further explores correction-gradient spectra and update shrinkability as predictors of behavioral rate. Finally, the authors show that aggressive compression can recover performance degraded by corrupted fine-tuning, suggesting that useful and harmful updates are unevenly distributed across adapter directions.

Critical strengths
  1. The paper provides an interesting perspective on fine-tuning by introducing behavioral rate to measure the information required to preserve acquired behavior.

  2. The compression-based recovery experiments provide interesting insights into the structure of LoRA updates, showing that aggressive compression can recover performance degraded by corrupted fine-tuning.

Critical weaknesses
  1. The proposed behavioral rate depends heavily on the compression method. As shown in Table 2, different codecs retain substantially different fine-tuning gains at similar bit rates. It is unclear whether the main conclusions remain consistent across different compression methods or LoRA parameterizations.

  2. The behavioral rate is defined based on retained held-out loss improvement, but the paper does not systematically validate whether this metric reflects final downstream task performance. It remains unclear whether preserving 90% of the loss improvement also preserves the acquired task capabilities.

  3. The paper argues that behavioral rate is a property of the dataset–receiver pair. However, Appendix C shows that increasing the optimization budget can reduce the behavioral rate even with the same training data. The impact of different fine-tuning configurations on the main conclusions is not sufficiently investigated.

  4. The writing of this paper can be improved. e.g.,

  • The Introduction contains only one citation and provides limited discussion of related work, making it difficult to understand the novelty of the proposed perspective.
  • The off-task generalization result (97% vs. 44%) highlighted in the Abstract is only presented in Appendix D.5 rather than in the main paper.
Questions for authors
  1. The paper suggests that compression can filter harmful information while preserving useful information. However, Appendix D shows that random rank-1 projections and update scaling can also recover much of the lost performance. Can the authors further clarify whether the improvement comes from selectively preserving useful information or primarily attenuating harmful updates?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 26rZ (part 1 of 2)

Thank you for the review. Your main concern being what R* measures and how much it depends on our choices, we ran new experiments on each point. Figure and table numbers refer to the revised paper.

W1. Robustness across compression methods.

First, we find that the central receiver effect persists across two substantially different compression schemes, even though their absolute rates differ. We evaluated 12 matched checkpoints, covering two receivers (Qwen2.5-7B and Mistral-7B), two tasks (code and math), and three seeds, with both our original codec and a new SVD-based rank-and-precision codec.

Though the second codec achieves substantially different absolute rates in the tested cases (approximately 1.5–7.7 times lower than the original codec), Qwen2.5-7B consistently requires fewer bits than Mistral-7B on both tasks, across the matched comparisons at retention thresholds from 75% to 95% (Fig. 14a).

This directly supports our main receiver-relative finding: the same training task can have systematically different behavioral compression costs depending on the frozen receiver, and this difference is not specific to our original codec.

We do not claim that the numerical value of R* is invariant to codec choice or LoRA parameterization. Indeed, alternative coding paths can modify both absolute rates and some relative task orderings. Yet, the receiver-dependent difference remains robust in our matched comparison.

This is consistent with the operational definition of behavioral rate in Section 3 and its codec-dependence already acknowledged in Section 7. We have clarified this dependence in the revised manuscript.

W2. Does 90% loss retention preserve task performance?

We want to precise that behavioral rate is defined relative to a specified distortion measure. Preserving a fraction of held-out NLL improvement and preserving the same fraction of downstream accuracy gain are distinct requirements; neither generally implies the other. Even in elementary classification, improving the probability assigned to the correct answer can reduce log-loss while changing the top prediction from correct to incorrect. Thus, the choice of distortion reflects which aspect of behavior one seeks to preserve.

To substantiate this, we investigated this relationship empirically. Across five checkpoints, selecting adapters at 90% loss-gain retention preserved 75–90% of their task-accuracy gain, while increasing the loss-retention threshold to 95% improved this range to 85–102% (Fig. 14b). We further evaluated nine model–task configurations spanning Mistral, Qwen, and Llama on HumanEval, GSM8K, and XBRL. Five retained at least 90% of their task gain. Notably, in three cases, compression improved test NLL beyond the uncompressed adapter while reducing exact-match accuracy, illustrating that teacher-forced likelihood and autoregressive task success need not move together. Crucially, our framework also allows the fidelity criterion to be defined directly in terms of task accuracy. Using 256 task-specific validation examples to select compressed adapters, we obtained the following held-out task-gain retention:

Model / taskTask-gain retention after loss-based selectionAfter accuracy-based selection
Mistral-7B / GSM8K74.3%95.3%
Mistral-7B / XBRL84.1%88.7%
Llama3.1-8B / XBRL82.2%95.6%

For Qwen2.5-7B/GSM8K, no compressed file in the evaluated grid reached the 90% accuracy-retention target. Accuracy-based selection can also require a higher rate: on Mistral/GSM8K, the selected rate increases from 0.649 to 1.507 bits/value, while test accuracy improves from 48.6% to 60.6%, approaching the uncompressed adapter's 63.2%.

These results illustrate that the same behavioral-rate framework can directly target the capability of interest by changing the distortion criterion. We clarify this distinction in Section 4.1 and report the accuracy-based evaluation in Appendix F. The latter remains exploratory, and the NLL- and accuracy-calibration distributions differ for GSM8K, so the rate differences cannot be attributed solely to the choice of distortion.

AuthorsOpenReview ↗

Response to Reviewer 26rZ (part 2 of 2)

W3. A larger training budget lowers R*.

Our new controlled experiments show that receiver dependence persists across optimization budgets. We trained Qwen2.5-7B and Mistral-7B adapters on code and math, and evaluated checkpoints after 125, 250, 500 and 1,000 optimizer updates across two seeds.

At the 90% retention threshold, Qwen2.5-7B requires 27–38% fewer bits than Mistral-7B on code and 56–75% fewer bits on math, across the evaluated budgets (Fig. 13, Table 10).

Thus, changing the training budget changes the learned adapter and its absolute behavioral rate, but does not eliminate the receiver effect in these controlled comparisons. The same receiver ordering is observed at all four budgets.

These experiments also reveal meaningful interactions between training budget and task; e.g., for Mistral, code becomes less costly than math after 250 updates in both seeds, whereas Qwen retained the same task ordering throughout the measured budgets.

Once again, we therefore do not treat R* as a property of the dataset alone or as independent of the training procedure. Rather, for a specified training protocol, it measures the rate required to preserve the behavior of the update that procedure actually learned. We have added these budget curves to the appendix and clarified the training-protocol dependence in the discussion. Our central conclusion that receiver choice changes the cost of preserving learned behavior holds throughout the tested budget range.

W4. Writing.

Thank you for these suggestions. We have revised the introduction to place the contribution more clearly relative to related work on adapter quantization, low-rank representation, spectral adapter structure, and compression under fixed bit budgets, including QLoRA and LoRAQuant.

We also distinguish our contribution from methods whose primary objective is reducing adapter storage, by specifying that our focus is on the minimum operational storage needed to preserve a specified behavioral gain, and how that requirement varies with the frozen receiver.

We also moved the 97% versus 44% off-task preservation result from Appendix D.5 of the submission (now Appendix G.6) into the main text (Fig. 6), with its normalization and evaluation setup made explicit. We hope these changes make the empirical contribution and its relationship to existing adapter-compression work more visible.

Q1. Does the improvement come from keeping useful information or from attenuating harmful updates?

Our new controls identify attenuation of learned directions as the dominant recovery mechanism, while showing that the attenuated directions can retain substantial task-relevant signal.

The leading singular direction of a corrupted update is strongly harmful at its trained amplitude but can become helpful at a lower gain. On Qwen2.5-7B, the leading direction alone reaches approximately 88% on SVAMP at tuned gains of 0.2–0.3, versus 66% at gain 1 (Fig. 17a). One-bit coding retains that direction and induces an effective aligned gain of approximately 0.40 (Section 4.3).

The resulting coded adapter reaches 82.8% on held-out GSM8K, above the 78.2% frozen base and similar to the 83.0% clean fine-tune (Fig. 18a). This effect held among our tests on 12 corrupted adapters spanning four base models and three seeds each, with at least one evaluated coded configuration recovering above both the frozen base and the damaged adapter in every case (Fig. 18b).

The mechanism is thus more nuanced than deleting harmful directions and keeping useful ones, since a learned direction can have a beneficial effect at moderate amplitude and a harmful effect at its trained amplitude. We show that compression can move the updates into a favorable operating range.

Our matched-scaling experiment further supports this mechanism: once the scalar gain of a full-precision rank-1 update is optimized on validation, one-bit coding yields only a +0.15 percentage-point difference on GSM8K (95% CI: −0.99 to +1.29; Fig. 16a). Thus, our experiments support gain recalibration as the principal explanation, rather than quantization selectively discovering and removing a distinct harmful subspace.

Importantly, our compression result remains practically meaningful: the coded representation achieves competitive recovery at a substantially reduced storage cost, without a separate search over scalar gains.


We would welcome your assessment of whether these new controls address the empirical concerns underlying your rating.

Thank you again for your detailed review. We would be grateful to hear whether our additional experiments on codec robustness, optimization budgets, and downstream accuracy address your main concerns. We would be happy to clarify any remaining points before the discussion period closes.

Reviewer_LvY6OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose a LoRA compression metric based on the model's performance. They show that the proposed metric correlates with the dataset's difficulty for a specific model. The main premise of the work is that weight reconstruction error alone cannot be representative of the degradation it produces on the model's performance. Moreover, in the case of corrupted data, the effect can be reversed: compression improves the model's performance instead of degrading it.

Critical strengths
  1. A surprising effect of LoRA compression for corrupted datasets is established.
  2. A model-specific (receiver) view on information compression is well-developed, its value is demonstrated.
Critical weaknesses
  1. The claim that "compression appears to filter" harmful directions is not sufficiently supported. In Section 5.2, the authors show that the strongest singular value contains the most dangerous direction, while the compression experiments in Section 5.1 report the best results while retaining it. Overall, the theory and the results suggest that the effect is due to attenuation of the adapter's most harmful direction, rather than due to filtering.
  2. The current experiments do not clearly delineate the effect of shrinkage along the principled axis from other compression effects. It would be useful to quantify the residual advantage of quantization over optimally scaled low-rank updates and to explain why random projections perform almost as well as the proposed compressed representation.
Questions for authors
  1. How do you choose the retained directions for the experiments in Figure 4, besides rank-1?
  2. Given that the leading singular directions are identified as the most detrimental at full precision, how do the authors reconcile this observation with their strong recovery when retaining and quantizing the leading rank-1 component?
  3. How much of the recovery advantage remains after optimizing the scaling of the corresponding full-precision low-rank updates, rather than only matching their norms?
  4. R^* doesn't show whether the retained information is useful or not for the task, since it is measured relative to the original gain, so it can't detect harmful information. Therefore, I would encourage the authors to distinguish more explicitly between the operational behavioral-rate metric and the broader notion of useful information.
4 comments
AuthorsOpenReview ↗

Response to Reviewer LvY6 (part 1 of 2)

Thank you for the insightful review. We answer each point below. Figure and table numbers refer to the revised paper.

W1.

The term “filtering” is indeed imperfect. We initially chose it to explain that compression does more for performance recovery than a simple uniform attenuation of a harmful update. Reconciling the harm of the leading direction at full precision with our results from section 5.1, we show that the leading direction can indeed carry useful information when attenuated to the right gain:

On Qwen2.5-7B, on the SVAMP benchmark where the frozen base model reaches 79%, the leading direction alone at gain 0.2 to 0.3 reaches 88%, but falls to 66% (Fig. 17a) at gain 1. This shows that the leading direction can indeed carry useful information at a lower gain. Qwen2.5-3B exhibits the same pattern (Fig. 17b). So compression keeps the leading direction and sets it to a better gain. One-bit sign coding scales the kept direction by about 0.40 (Section 4.3) and reaches a performance of 82.8% on held-out GSM8K, above the base model (78.2%) and level with an adapter trained on clean data (83.0%) (Fig. 18a).

W2. Distinguishing shrinkage from quantization and explaining random projections

As explained in the general comment, "shrinkage" designated the drop in the overall norm of the update, and the controls we ran matched overall norms. However, norm reduction is not equivalent to reducing the effective gain. One-bit coding preserves approximately 63% of a rank-1 update's norm, but only 40% of its component along the original direction. This explains the apparent advantage over norm-matched scaling.

We therefore compared quantization against optimally rescaled full-precision rank-1 updates under identical validation-based selection. On GSM8K, one-bit and two-bit coding do not statistically outperform rescaling, beating it by only 0.15 percentage points (95% CI: −0.99 to 1.29) and 0.76 points (−0.53 to 2.05), respectively (Fig. 16a).

Random projections’ performance can be explained by the same mechanism: projecting a rank-16 update onto a random rank-1 subspace produces an expected effective gain of 1/16, or s/16 when scaled by s. Their performance closely follows the corresponding scalar-gain curve, with deviations from projection noise at larger scales (Fig. 20).

We would like to point out that coding reaches this level without any gain search: on GSM8K, the untuned code is within 1 point of the best of 120 searched gains on both Qwen models (Fig. 18a). This is what we used to mean by "filtering", which our abstract and Section 5 now instead describe as “recalibration”.

AuthorsOpenReview ↗

Response to Reviewer LvY6 (part 2 of 2)

Q1. How are the directions in Figure 4 chosen?

At each LoRA layer we take the SVD of B·A. We keep the top r directions, for r = 1, 2, 4, 8 or 16. We quantize them and load them back. In our revision, we added this to the caption of Figure 4.

Q2. The leading direction is harmful, yet rank-1 coding recovers. How do these fit?

As detailed in the answers to W1 and W2, the leading direction is harmful only at its trained gain. Rank-1 coding keeps this direction and scales it by about 0.40, which is a gain where the leading direction is positive. We add additional experimentation on the effects of scaling the leading direction here.

On Qwen2.5-7B, the leading direction alone scores 66% on SVAMP at gain 1. It scores 84% to 88% at gains from 0.1 to 0.5 (Fig. 17a). On held-out GSM8K, one-bit rank-1 coding scores 83%. The base model scores 78% and the damaged adapter 24% (Fig. 18). Qwen2.5-3B gives the same result: 73% for rank-1 coding, 62% for the base model and 18% for the damaged adapter. Adapters trained on clean data are structured similarly in terms of leading directions. On Qwen2.5-7B, a clean adapter’s leading direction alone scores 83.3% on GSM8K, with accuracy comparable to that of the full adapter (83.0%).

Q3. How much advantage remains after optimal scaling?

On our models and dataset combinations, we found little room for it. With the same selection for all methods (see W2), the one-bit code is 0.15 points above the best scaled rank-1 update on GSM8K (−0.99 to 1.29). The two-bit code is 0.76 points above it (−0.53 to 2.05). On GSM-Symbolic the gaps are 0.6 and 1.0 points. Resolving a gap of 0.15 points would need about 150,000 test questions, which none of our datasets are large enough to supply.

It should be noted that both methods already reach the adapter trained on clean data: 82.8% for the compressed adapter, 83.2% for the best scaled update and 83.0% for the clean adapter. Once again, our revision states that coding needs no gain search and only a very small file compared to a full update (Fig. 18a).

Q4. R* and useful information.

This is another valid point, and we have modified Section 3 so as to define R* as the number of bits needed to keep a chosen amount of a declared gain, for a given model and decoder, without claiming absolute usefulness to the task. For the corrupted-training experiments we report absolute accuracy.

CLOSING

We would be glad to hear whether the gain analysis resolves your concern about the filtering interpretation, and we can run further checks if useful.

I thank the authors for their clarifications and condor explaining the contribution. The discovered compression effect is interesting and explained as calibration of the principal direction, therefore I am willing to increase my score.

Thank you very much for your positive feedback and for reconsidering your score. We are glad that the additional experiments and the gain-recalibration explanation addressed your concerns. Your comments helped us substantially clarify the paper's contribution.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful reading. Since all three reviews asked how compression repairs a damaged adapter, and two asked precisely what R* measures; we ran new experiments on both questions. This comment summarizes our answers and the four main changes in the revision.

1. What we meant by shrinkage and filtering: In the submission, "shrinkage" referred to the drop in the overall norm of the update, with our controls matching that norm. We used "filtering" to transmit the intuition that compression does more for recovery than this uniform shrinkage. We did not anticipate that these terms could be read as statements about individual singular directions, which are controls that were introduced much later than our control for recovery via uniform shrinkage. We now separate the two clearly; comparing recovery to a uniform shrinkage of the whole update, and describing the change of gain along singular directions as recalibration rather than filtering.

2. How compression repairs a damaged adapter: The leading direction of the damaged update carries most of its useful information, but training gave it too large a gain, making it harmful on uncorrupted tasks. Compression keeps this direction and lowers its gain: one-bit sign coding scales it by about 0.40. On GSM8K, the coded adapter beats the base model by 4.5 points (82.8% and 78.2%) and matches an adapter trained on clean data (83.0%). On both Qwen models it is within 1 point of the best of 120 searched gains, with no search of its own (Fig. 18a). On 12 damaged adapters from four base models, the best coded file beats the base model in all 12 (Fig. 18b).

3. What R* measures: R* is defined for a given codec, performance measure and training setup, and the definition now says so. Different distortion measures need not induce the same rate: preserving the NLL gain is not generally equivalent to preserving downstream task accuracy. Our main finding holds across these choices: the model that receives the update changes the rate. In matched experiments, Qwen2.5-7B consistently requires fewer bits than Mistral-7B under both tested codecs and across four optimization budgets and two training seeds. We also release the full rate grid behind Figure 7, with its uncertainty (Appendices D–F).

4. Theory and prediction: We corrected the curvature condition. We added an exact aligned-gain decomposition for sign coding that, together with our gain-sweep experiments, helps explain the observed recovery. Our Gaussian rate-distortion result is kept as motivation. Predicting R* from the trained adapter now holds with forecasts frozen in advance (error 0.080 bits per value on new checkpoints), and we note that predicting it before training remains open.

Our point-by-point answers are in each reviewer's thread, and we welcome further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.