- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper measures the smallest serialized LoRA representation that preserves 90% of a fine-tuning gain. It studies receiver-dependent compression rates, recovery from corrupted supervision, and a retrospective relationship between rate and update shrinkability.
Critical strengths
Near one bit per value, the lower-RMSE code retains about 3% of the gain versus 86% for uniform coding, demonstrating that weight reconstruction error can mislead. Recovery is tested across models with shrinkage, rank, random-projection, and off-task controls. The retrospective shrinkability fit is also promising: RMSE 0.068 bits/value on 42 checkpoints from five additional receivers (Figure 7).
Critical weaknesses
-
The central measurement needs a fuller audit trail. The complete rate grid, per-adapter retention curves, and raw gains behind Figure 6 are missing. Selected operating points do not establish that corpus reorderings survive threshold uncertainty, especially when fine-tuning gains are small.
-
The local minimizer claim lacks a curvature condition. Invertibility alone does not make −H_T⁻¹μ a minimum (Appendix B.3, Equation 11). The claim needs positive definiteness on the working subspace; singular cases also need compatible gradients or appropriate damping. This is a repairable theoretical error.
-
The theory does not explain aggressive compression recovery. The implemented predictor omits the curvature matrices in the derivation, and Table 5 reports failure of the local expansion below one bit per value. The valid Gaussian rate-distortion result therefore remains a motivation rather than an explanation of the measured effect.
-
Locked pretraining predictors fail to transfer. All three have worse RMSE than the base-code-length comparator on new panels (Appendix A, Table 3). The promising post-training shrinkability fit is retrospective; it does not establish prospective or pretraining prediction.
Questions for authors
- Which evaluated rates and retention curves produce Figure 6, and do its ordering changes survive uncertainty in the raw gains and thresholds?
- After correcting the curvature assumptions, which theoretical conclusion is supported at the reported operating rates?
- Figure 11 shows substantial recovery from shrinkage. Under matched validation-based selection, how much does coding improve over the best scalar-rescaled full-precision rank-1 update?
5 comments
Response to Reviewer 8VzQ (part 2 of 3)
Table 8. Number of orderings resolved at the 95% level at 90% retention, under each of the four rate conventions (10,000 resamples of the evaluation batches).
| Orderings resolved at 95% | Interpolated rate, best reference | Interpolated rate, raw reference | Measured file, best reference | Measured file, raw reference |
|---|---|---|---|---|
| Task order within a model | 18 of 19 | 19 of 19 | 17 of 19 | 19 of 19 |
| Model order within a task | 55 of 57 | 56 of 57 | 52 of 57 | 51 of 57 |
| Task-order reversals between models | 46 of 51 | 51 of 51 | 41 of 51 | 51 of 51 |
W2. Curvature condition.
Thank you for identifying the missing curvature assumption. We agree that invertibility alone does not guarantee that the stationary point is a minimum.
For the quadratic training-loss approximation on a working subspace ,
the expression
is the unique minimizer when the restriction of to is positive definite, rather than merely invertible.
We have made this assumption explicit in Appendix B.3.
We also clarify the singular case; if the restricted Hessian is positive semidefinite, a finite minimizer exists only when the restricted gradient lies in its range; in that case the minimizers can be expressed using the pseudoinverse, up to an arbitrary null-space component.
When damping is used, corresponds to a regularized quadratic problem (which we now distinguish from the original undamped objective).
With these corrections, we repair the local-minimizer statement without changing our empirical measurements or the central interpretation of correction-gradient second moments as geometric descriptors.
W3 and Q2. What does the theory actually explain at the aggressive compression rates used?
With our new results, we can identify a concrete finite-rate mechanism behind the observed recovery, while keeping a precise scope for our theoretical claims. For a rank-1 update and its one-bit coded version , define the aligned coefficient
The coded update then admits the exact decomposition
For the rank-1 sign-coding construction, the gain can also be related to the alignment properties of the quantized factors. Under Gaussian-factor assumptions, the corresponding expected aligned gain is approximately 0.405; empirically, we measure approximately 0.400 across 196 layers (Section 4.3).
This decomposition by itself does not predict task recovery; hence our crucial additional evidence is the observed behavioral response as a function of gain. For corrupted Qwen2.5-7B adapters, the leading full-precision singular direction achieves its best SVAMP performance around gains 0.2–0.3, whereas the original trained gain of 1 is strongly detrimental (Fig. 17a).
Furthermore, validation-selected full-precision rescaling achieves accuracy comparable to coding (Fig. 16), supporting attenuation as the principal observed mechanism. The residual (E) can nevertheless affect task performance.
This provides an empirical explanation of the dominant recovery mechanism, that is an effective attenuation of the learned update, supported by an exact parameter-space decomposition.
A one-dimensional quadratic surrogate can motivate why intermediate gains may outperform both zero gain and the trained gain. However, we do not claim that this surrogate gives a quantitative recovery guarantee for autoregressive language models at one bit per value. In particular, orthogonality in weight space does not imply orthogonality in the task-loss geometry.
We retain the Gaussian rate-distortion derivation as motivation for receiver-relative functional geometry, not as a quantitative theory of the compressed checkpoints or of recovery at extreme rates.
This distinction addresses the regime-of-validity concern raised in the review.
Response to Reviewer 8VzQ (part 3 of 3)
W4. Retrospective versus prospective prediction
After additional experiments, we now have prospective evidence for our post-training shrinkability rule, with predictions frozen before measuring the corresponding behavioral rates.
In a new six-checkpoint evaluation, the rule achieves a rate-prediction RMSE of 0.080 bits/value, compared with 0.105 for a constant-rate baseline and 0.093 for a task-mean baseline (Appendix F).
Additional evaluations across further adapters and reparameterizations yield RMSE values between 0.082 and 0.094, consistent with this result. These checks contain correlated evaluations of related adapters, which we clearly distinguish from our six independently evaluated new checkpoints.
The original Figure 8 relationship (Figure 7 in the submission) achieved 0.068 bits/value RMSE retrospectively. Our new results therefore provide an additional prospective test of the already fixed post-training rule.
Of course, this must be clearly separated from prediction before finetuning, since our predictor uses the trained update and a small number of shrinkability evaluations. We also carried out a new prospective pretraining-prediction evaluation, freezing a revised predictor before training six adapters on two new models. Its RMSE was 0.253 bits/value, compared with 0.263 for a text-length baseline. Although the task orderings were correct for both new receivers, the predictors showed systematic underestimation of approximately 0.21 bits/value (Appendix F).
We therefore do not claim accurate absolute prediction before fine-tuning, which remains a substantially harder open problem.
We can still confidently claim that once an update has been learned, the shrinkability-based rule now has prospective evidence beyond the retrospective fit reported in the submission.
We have made this distinction explicit in the revised text.
Q3. Coding vs the best scaled rank-1 update.
Our matched-selection experiments indicate that coding achieves comparable recovery accuracy, rather than a statistically established improvement over tuned and rescaled rank-1 updates.
Under the same validation-based selection protocol, the paired GSM8K differences are:
| Comparison | Coded minus scaled accuracy | 95% CI |
|---|---|---|
| 1-bit rank-1 code vs selected fp16 rank-1 scaling | +0.15 pp | [−0.99, +1.29] |
| 2-bit rank-1 code vs selected fp16 rank-1 scaling | +0.76 pp | [−0.53, +2.05] |
Neither interval establishes a statistically significant advantage of coding in task accuracy (Fig. 16a).
This directly answers the question: most of the recovery can be explained by an appropriate reduction of effective gain along the retained direction.
The remaining contribution of compression is nevertheless operationally important. A small coded representation achieves this recovery without any costly search over scalar gains at deployment.
On held-out GSM8K, the untuned rank-1 one-bit code is within one percentage point of the best of 120 searched scalar gains on each of Qwen2.5-7B and Qwen2.5-3B (Fig. 18a). In an illustrative comparison, the coded adapter obtains 82.8%, the best validation-selected scaled update 83.2%, and the clean fine-tune 83.0%.
We therefore distinguish two conclusions:
Mechanistic: gain attenuation accounts for most of the observed recovery, and we do not claim a significant additional accuracy benefit from quantization.
Operational: the compact coded update realizes a favorable behavioral operating point with much lower storage and without a separate gain-optimization procedure.
We believe this is a more precise and informative characterization of the results than attributing recovery to an unverified selective-removal effect.
We believe these additions resolve the principal concerns about measurement transparency and theoretical scope while strengthening the paper's empirical contribution.
Response to Reviewer 8VzQ (part 1 of 3)
Thank you for the detailed review and for checking the theory closely. We answer each point below. Figure and table numbers refer to the revised paper.
W1 and Q1. Audit trail and uncertainty for Figure 7 (Figure 6 in the submission).
The central receiver-dependent ordering holds true in your requested audit of rate conventions, retention thresholds, and evaluation uncertainty.
We reconstructed the complete evaluation grid behind Figure 7. The revised manuscript includes all 60 retention curves (Fig. 12), raw-gain statistics, and the sensitivity analyses across rate conventions and thresholds (Appendix D, Tables 6–8), and a per-checkpoint table of gains and rate thresholds (Table 9). The underlying per-file numerical data will be released upon publication.
We also explicitly separated four operational conventions:
This distinction makes explicit that an interpolated threshold estimate and the smallest measured reloadable file are not interchangeable quantities. We have aligned our revised description of Equation (2) with the reported operational convention and label interpolated estimates separately.
We recomputed the Figure 7 comparisons using four retention thresholds (50%, 75%, 90%, 95%) and these four rate conventions.
We find a stable principal result: XBRL remains the least costly task for the same six of seven receivers in all 16 combinations. Qwen2.5-7B remains the exception, with math cheaper than XBRL in the central estimates.
To evaluate uncertainty, we re-evaluated the 60 checkpoints and performed 10,000 resamples of the evaluation batches. At the 90% retention threshold, depending on the rate convention:
These calculations explicitly propagate evaluation-sample uncertainty through the retention threshold. The bootstrap is conditional on the trained checkpoints; variability across training seeds is reported separately in Table 7.
For the specific Qwen2.5-7B exception, the bootstrap support for math requiring fewer bits than XBRL ranges from 0.92 to 1.00 across conventions. Thus, the central ordering is stable, while not every convention resolves this particular comparison at the 95% confidence level.
We have expanded the appendix with the complete rate grid and retention curves so that the ordering claims can be checked directly rather than inferred from selected operating points. These results support the main Figure 7 conclusion while making the limitations of close comparisons and threshold estimation transparent.
Table 7. R* at 90% retention in bits per value: mean over three training seeds, with the lowest and highest seed in parentheses. Qwen3-8B has no math adapter.