acceptodds
Under review as a conference paper at ICLR 2027

Colinearity Decay: Training Quantization-Friendly Transformers via Decoupled Weight-Space Regularization

12 views

Abstract

Low-bit quantization enables efficient Transformer deployment, yet activation outliers complicate fully quantized inference. Quantization methods act after or during training: post-training methods adapt pretrained models, while many training-time approaches address outliers through architectural changes, or by regularizing activations. Yet minimizing activation magnitude is incomplete: excessive suppression can degrade full-precision accuracy without consistently improving quantized models. We argue that quantization-friendly training should target harmful structural artifacts that cause extreme activations. To this end, we introduce Colinearity Decay (CD), a weight-space regularizer for matrix pairs within Transformer blocks. CD penalizes detrimental cross-matrix alignment and mitigates extreme activations. The decoupled update leaves architecture and task loss unchanged, adding little training overhead and no inference-time cost. Across ImageNet-1K pre-training, object detection, downstream fine-tuning, and small-scale LLM pre-training, CD improves quantized task performance across algorithms and bit widths while preserving unquantized performance. These results support decoupled weight-space regularization for training quantization-friendly Transformers.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Quantization-Friendly Training, Model Compression, Transformers

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mUSiOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
1
Presentation
1(−1)
Soundness
2
Read review
Paper summary

This paper introduces a regularization method in PTQ for penalising the activation outliers through ||W_F1 W_F2 ||_F where W_F1 and W_F2 are the consecutive matrices in FFN of Transformers. This regularizer comes from existing literature TWEO and the existing Colinearity Decay literature.

Critical strengths
  1. The topic this paper introduced is interesting.
  2. The experiments make sense to me.
  3. The bibliography is good.
Critical weaknesses

I carefully read this work for managing to understand its logic, evidences, and method.

The presentation quality is below the bar. There are rooms to be further improved. Please see the "Questions for authors".

This work lacks experiments to defense the arguments/motivations/claims made by authors.

As far as my understanding, it has limited novelty. This regularizer comes from existing literature TWEO and the existing Colinearity Decay literature.

Questions for authors

I first highlight the points I found. I'll state details after these.

  1. Presentation Quality. Presentation is very limited and I believe there is a significant room to be improved. I carefully read the Abstract, Introduction, related work, and Method. Despite the topic is interesting, the presentation.

For example

"We empirically show that excessive suppression of activation magnitudes can degrade both unquantized and quantized accuracy. Quantization-friendly training should instead target harmful structural artifacts that cause extreme activations".

These claims are crucial to justify/motivate this work, however they are never supported by evidences.

  1. Motivation Logic.

There is a logic problem in the introduction. In the first paragraph, it argues quantization is crucial for deploying transformer-based models.

In the second paragraph, the work argues PTQ/QAT are a way to reverse the performance degeneration due to quantization.

However, in the third paragraph, the work introduces the attention sink/artefacts/activation outliers. I don't understand how this correlates to quantization and how the activations are not represented by quantized values. Can the authors clarify and improve the presentation in future?

Also the attention sink should cite the Meta's paper that firstly introduced the concept.

  1. Setting May Not Hold. Another question is, since nowadays attention sink is solved by QWen's group in [1]. What is the significance of the proposed method?

[1] Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free, NIPS25, https://openreview.net/forum?id=1b7whO4SfY

  1. The work should clarify/present that why quantization can't represent outliers is a culprit.

  2. Over claims or claims not supported by evidences.

For example

"in this paper, we argue that the latter strategy offers a better trade-off between full-precision and quantized accuracy."

this is not supported by evidence or ant experiments.

For example

L86-88

"Our key argument is that quantization- friendly training should reduce harmful structural artifacts that cause extreme activations, but should not suppress activation magnitudes to a level that potentially degrades task performance."

Claims. not supported by evidence or experiments.

I don't check more claims. But regarding the claims I read, they are not supported by evidence.

  1. L81

"Figure 1 compares TWEO with our Colinearity Decay (CD) and provides task and block-level".

The work doesn't introduce CD yet. This confuses me in reading. The engagement is very low.

  1. Notations are not introduced.

for example L179-180. The mathematical notations must be properly introduced.

  1. Typos.

"Figure 1: Smaller activations do not necessarily improve quantization."

improve quantization?

  1. Limited novelty. This work contributes little to what we already knew. This regularizer comes from existing literature TWEO and the existing Colinearity Decay literature.
4 comments

Thank you for your reviewing.

We first clarify two key misunderstandings concerning the originality and scope of our method stated in the paper summary:

  • Originality of CD: We introduce CD in this work and affirm its originality. If the reviewer questions this originality, we respectfully request specific and verifiable evidence. Sections 3.1 and 3.2 explicitly distinguish the regularizers: CD targets weight-matrix pairs, whereas TWEO targets block-output activations. We credit TWEO for its colinearity analysis, which is distinct from the CD regularizer.
  • Method positioning and scope: As stated in the Abstract, Introduction, and Sections 3.2 and 3.3, CD is a decoupled weight-space regularizer applied during training to improve subsequent quantization, rather than a PTQ method. Section 3.2 describes its application to normalization, attention, and FFN matrix pairs, extending beyond WF1W_{F1} and WF2W_{F2}.

We have devoted substantial effort to making the presentation clear. Given the conference page limit, we cannot discuss every background detail and assume a working familiarity with model quantization. We address your specific questions below:

  • Experimental support for the accuracy claims: Table 1 compares CD with TWEO and KurtReg under matched training and evaluation settings. Despite substantially reducing maximum activations, these activation regularizers lower FP accuracy in most comparisons and yield inconsistent quantized gains, whereas CD offers a better overall trade-off. Appendices A.3 and A.4 report implementation details and hyperparameter sweeps (Table 5), documenting the selection of comparison settings. Section 4.2 explicitly connects these observations to our claims about activation suppression and FP/quantized accuracy.

  • Attention sinks and quantization difficulty: We would appreciate clarification of the criterion for considering attention sinks “solved”. In our view, attention sinks and quantization difficulties are overlapping issues, with neither subsuming the other. Table 1 already shows that CD further improves quantized accuracy with AttnBias. To address the cited method directly, we additionally reproduced headwise gated attention. The results below show further gains from CD under both quantization pipelines while preserving FP performance (accuracy in %; gains in percentage points).

EvaluationGated attentionGated attention + CDGain
FP3281.76881.902+0.134
RepQ-ViT W4A476.33279.320+2.988
Percentile W4A678.83281.206+2.374

We appreciate the suggestions for improving clarity and completeness. We address the specific presentation and citation comments below:

  • Introduction of CD and placement of Figure 1. CD is already defined in the Abstract, and the central argument is stated before Figure 1 is introduced. The figure provides a direct visual illustration of that argument. Presenting a motivating comparison in the Introduction and reserving the detailed method for the Method section follows a conventional paper structure.
  • Figure caption. We agree that “quantized accuracy” is more precise and easier to understand. We will revise the caption to “Smaller activations do not necessarily improve quantized accuracy.”
  • Attention sink reference. We will add a citation to Xiao et al., Efficient Streaming Language Models with Attention Sinks (ICLR 2024), to make the discussion of attention sinks more complete.
Reviewer_mUSiOpenReview ↗

I thank the authors for the replies.

I thank the authors for the replies. Unfortunately, the replies appear just repeating for each reply and fails to answer my concerns.

AuthorsOpenReview ↗

Respond to Your Feedback

Thank you for your feedback. We remain open to further technical discussion of the paper and would sincerely welcome specific evidence supporting your concerns, so that we can address them directly.

Dear Reviewer mUSi,

As the discussion period draws to a close, we would like to thank you for your time and consideration.

We hope that our responses have clarified CD's contribution and the experimental support for our claims. We would appreciate your consideration of these clarifications and additional results in your final assessment.

We wish you all the best.

Best regards,
The Authors

Reviewer_EekYOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Colinearity Decay (CD) uses training-time weight regularization to make Transformers easier to quantize. It penalizes the product of selected matrix pairs and applies a separate weight update to reduce amplification across matrices. The paper reports gains on ImageNet-1K, COCO, downstream fine-tuning, and small-scale language-model pre-training, without adding inference cost.

Critical strengths
  1. The quantization gains are consistent across the tested vision settings. Appendix Table 6 covers four backbones and four quantization settings, with three training seeds. Mean accuracy improves in all 16 comparisons. For ViT-B with RepQ-ViT W4A4, it rises from 76.286% to 78.615% while full-precision performance is maintained.

  2. CD leaves the inference architecture unchanged and can be followed by different quantization methods. Figure 2 reports less training overhead than TWEO in the tested ViT setups. This is useful for models with a fixed deployment pipeline. The paper provides evidence beyond a single vision classification task, including the early language-model results in Table 3. The observation that better quantization does not require the smallest possible activation outliers is also useful. The core matrix derivation is mostly correct, and the figures and tables are clear.

Critical weaknesses

Contribution beyond closely related regularization. My main concern is how much the specific CD design adds over existing methods. QuantTune (Chen et al., MIPR 2025) already targets activation outliers during Transformer fine-tuning to improve PTQ without changing inference. The shared goal alone does not settle CD's novelty, but it is relevant context.

The closer mathematical precedent is Frobenius Decay (Khodak et al., ICLR 2021), which regularizes the Frobenius norm of a matrix product. FLAMBé also uses a decoupled update from this product regularizer. CD's Eq. (2) has the same basic regularization form. Its one-sided update, WD-aligned scaling, Transformer pairing, and quantization use are differences, but their added value over that close alternative is not isolated. Neither work is cited. I would judge the whole design by what these changes achieve, rather than require each component to be new.

Evidence for the update rule and structural explanation. The decoupling comparison changes more than the update rule. Table 4(c) compares decoupled CD with CD-loss, but Appendix A.9 gives them different normalization and strength control. The gain therefore cannot be assigned to decoupling alone. The normalization also differs between main Section 3.3 and Appendix A.9, and the WD-aligned scaling in Appendix A.2, Eq. 26 is not fully presented in the main text.

The statement that loss-based CD cannot use WD-aligned scaling is too strong. A stop-gradient scale factor can give the same raw regularization direction in a loss. How that direction is processed by the optimizer is a separate difference that needs to be tested or explained.

The structural explanation also needs a more limited claim. For an FFN, ‖W₂W₁‖²_F omits the nonlinearity; for multi-head attention, the product also differs from the full computation. It is a reasonable surrogate, but it does not directly guarantee lower activation outliers or quantization error. Appendix Figures 4–5 support the proposed explanation, yet do not isolate it from a general regularization effect during training.

There is also a boundary case in the scaling derivation: first-order Frobenius norm matching needs a positive denominator. At B = 0, the fallback rule no longer gives the same guarantee. The paper should state this limit.

Overall assessment

I lean toward rejection. The gains and practical value are clear, but the contribution of CD's specific update and pairing choices over the closest product regularizer is still uncertain. A controlled comparison and a narrower causal claim could change my assessment. The issue is the evidence for these choices, not whether every component was invented here.

Basis for confidence

I have research experience with Transformer quantization, activation outliers, weight regularization, and optimization. I checked the relevant prior work and equations, but did not reproduce the training runs. I am more certain about the comparison gaps than about the cause of each reported gain.

Questions for authors
  1. Compared with Frobenius Decay / FLAMBé, what benefit comes from CD's one-sided update, scaling, and matrix pairing? Please explain the formula-level differences and any direct evidence for them. How does this contribution differ from the quantization-friendly fine-tuning studied in QuantTune?

  2. Can Table 4(c) compare the update rules with the same pairs, normalization, and similar update scale? If the existing results cannot separate these factors, please limit the conclusion to the full CD implementation beating the tested loss-based variants.

  3. What do the existing experiments show about the connection between reduced cross-matrix alignment and lower quantization error, beyond a smaller product norm? Please also explain the scaling rule when its denominator is zero or very small. These questions concern the current evidence; I am not requesting additional full-training seeds or larger models.

3 comments

Thank you for the careful review and constructive questions. We appreciate your recognition of the consistent quantization gains and practical value without inference overhead.

1. Contribution relative to Frobenius Decay / FLAMBé and QuantTune.

We acknowledge that Frobenius Decay (FD) already uses the product-norm penalty and FLAMBé applies its gradients through decoupled updates. CD's contribution concerns pairing and update design for quantization-friendly training.

Pairing and update allocation: FD regularizes factorized products, including per-head QK and OV. CD uses two pair types at four positions (Section 3.2): normalization with QKV or FC1 (composable), and V with O or FC1 with FC2 (functional). FLAMBé updates both matrices in each pair; CD updates only the downstream matrix. Using only functional pairs with two-sided updates performs worse under otherwise identical settings.

Budget and scaling: WD provides a practical reference strength. Matching CD's budget to the WD-only baseline enables controlled comparisons. Appendix A.2, Eq. (26), defines this scaling, which also helps adapted FLAMBé.

FLAMBé on quantization: We compare per-head OV updates under the same ViT-B setting: FLAMBé's default full WD replacement versus CD's budget and scaling.

ConfigurationFP (%)W4A4 (%)
Baseline81.93076.950
CD82.07678.712
CD: two-sided functional-pair variant81.45276.522
Adapted FLAMBé: full WD replacement81.4240.106
Adapted FLAMBé: CD budget and scaling81.78677.148

Full WD replacement severely hurts quantized accuracy. CD's budget and scaling yield a modest gain over baseline, while full CD performs better.

QuantTune shares the fine-tuning goal and uses an activation-based outlier loss. CD regularizes weight pairs outside the task-loss graph, with evaluation in both pre-training and fine-tuning.

We recognize the relevance of both works and will cite them and clarify the differences.

2. Matched loss-based versus decoupled CD.

The variants in Table 4(c) differ in both update rule and scaling: decoupled CD uses WD-aligned scaling, whereas CD-loss uses task-loss-relative scaling.

We then matched all other settings, including detached WD-aligned scaling, so that only CD's inclusion in AdamW's moment estimates and preconditioning differed. This matched loss-based run failed to converge, reaching only 6.07% accuracy at epoch 100, with task loss above 6.5, versus approximately 3.7 in normally converging training. Disabling CD-loss during warmup and linearly restoring it afterward did not resolve this failure. These results motivated task-loss-relative scaling to stabilize the original CD-loss comparison.

We will add this context and the matched experiment, and limit the conclusion from Table 4(c) to the full CD implementation outperforming the tested loss-based variants.

3. Reduced cross-matrix alignment and lower quantization error.

We directly analyze local and global quantization errors to assess CD's effect on quantization.

Both metrics use relative RMSE. Local error compares a layer's outputs before and after quantizing its input and weights. We focus on FC2, where quantization error is largest (Figure 1(b)). Global error compares the quantized and FP models' outputs at the same block, capturing propagated errors along the quantized path. The table reports relative error reductions from baseline to CD:

ModelTop-3 local errorTop-6 local errorMean global error
ViT-B16.6%9.9%7.84%
ViT-S17.2%11.6%7.26%
Swin-S26.9%20.6%8.08%

Top-3/Top-6 select layers with the largest baseline local errors, showing that CD improves the hardest local positions. Global error decreases in all 42 analyzed blocks, by 7--8% on average, indicating lower accumulated error along the quantized path.

We will clarify and narrow our claims: CD models with reduced cross-matrix alignment do not necessarily produce smaller activation outputs, but we observe lower local and global quantization errors.

4. Zero and near-zero scaling denominators.

We agree this boundary needs clarification. For weights WW and raw CD direction DD, define c=⟨W,D⟩F/(∥W∥F∥D∥F)c=\langle W,D\rangle_F/(\|W\|_F\|D\|_F). The quadratic-to-first-order squared-norm change ratio is t/(2c2)t/(2c^2) for effective step size tt. Since t≤10−5t\leq10^{-5} in our runs, c>10−2c>10^{-2} keeps this ratio below 5%. We therefore propose full baseline WD when the denominator is zero or c≤10−2c\leq10^{-2}.

The minimum cc values observed for ViT-B and Swin-S were 0.1113 and 0.1287, respectively, both above the threshold. We will clarify this boundary and document the proposed safeguard, which was not used in these runs.

Dear Reviewer EekY,

Thank you for your thoughtful review and constructive suggestions, which have helped us clarify the contribution and supporting evidence.

With approximately three days remaining in the discussion period, we would like to kindly follow up on our rebuttal. Our response covers:

  • CD's relationship to Frobenius Decay, FLAMBé, and QuantTune, with controlled comparisons of pairing and update design.
  • A matched loss-based versus decoupled comparison.
  • Reduced cross-matrix alignment and lower quantization error, with appropriately narrowed claims.
  • The scaling rule when its denominator is zero or very small.

Could you please let us know whether these responses address your concerns, or whether any questions remain? We would be happy to provide further clarification during the remaining discussion period.

Thank you again for your time and careful consideration of our work.

Best regards,
The Authors

Dear Reviewer EekY,

As the discussion period draws to a close, we would like to thank you for your careful review, continued engagement, and willingness to reconsider your assessment in light of our responses. We appreciate your recognition of CD's contribution and your revised rating.

Your constructive questions helped us clarify the roles of matrix pairing and update allocation, strengthen the quantization analysis, and present the supporting evidence more precisely. We are grateful for the time and thought you devoted to this discussion.

We wish you all the best in your research and future work.

Best regards,
The Authors

Reviewer_QVZaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  1. The authors introduce Colinearity Decay (CD) a regularizer with a simple intuition: squared Frobenius norm of the matrix product, and its downstream update suppresses directions emphasized by the upstream matrix. The main implementation replaces part of weight decay with a separately applied CD update. A separate quadratic proxy extends the approach to SwiGLU.

  2. The method adds no inference time overhead. Experiments cover ImageNet pretraining, downstream classification, COCO detection, and smaller language-model pretraining, with multiple quantizers.

Critical strengths
  1. The empirical results are strong across varying architectures, task and datasets (as well as seed for variance). Indicating that the regularizer improves quantization performance consistently.

  2. The proposed scaling matches the first-order radial effect of weight decay and the final checkpoint have a upstream/downstream norm ratio of 1. Combined with the absence of added inference operations, this makes the weight-space intervention a practically interesting alternative to activation penalties.

Critical weaknesses
  1. The benefit of decoupling is not cleanly isolated. The provided ablation changes normalization as well as optimizer coupling. Momentum/preconditioning would still distinguish coupled and decoupled optimization, but the direction and scaling can be matched. The current comparison leaves a central design claim insufficiently supported.

  2. The evidence for the norm-matched algorithm across tasks is unclear. Detection and fine-tuning use a simpler upstream rescaling so the central first-order norm-matching argument does not directly apply to those variants. The exact task-specific strengths also need clearer specification. The SwiGLU extension constructs a Gram matrix and lacks a corresponding overhead measurement.

Questions for authors
  1. Can the loss vs. decoupled comparison be done with identical detached WD-aligned scaling, matrix pairs, updated matrices, and regularization coefficients, so that the comparison isolates optimizer coupling? What prevents using the detached scale in the auxiliary loss formulation?

  2. Which experiments use the matrix-aligned update versus the simpler rescaling, and what are the exact coefficients for detection and fine-tuning?

  3. What are the training-time and peak-memory costs of the SwiGLU extension on ModernLM, and how does its post-down proxy relate to the activation sites quantized in those experiments?

3 comments

Thank you for the careful review and constructive questions. We appreciate your recognition of the consistent empirical gains and the practical value of CD without inference overhead.

1. Matched loss-based versus decoupled CD.

The original ablation already matched matrix pairs and downstream-only updates, but used task-loss-relative scaling. We now compare the two implementations using identical pairs, updated matrices, detached WD-aligned scaling (Appendix A.2), and coefficients: CD 0.01 and residual WD 0.04. The difference is whether the CD direction enters AdamW's moment estimates and preconditioning with the task gradient.

The matched loss-based run reaches only 6.07% accuracy at epoch 100, with task loss above 6.5, versus approximately 3.7 in normally converging training. We also disabled CD-loss during warmup and then linearly restored its coefficient over an equally long interval. Training still failed to converge. This is consistent with the optimizer-processed regularization dominating task optimization at this strength. We therefore adopted task-loss-relative scaling to stabilize the original ablation in Table 4(c).

We will add this matched comparison and clarify the original ablation's scope.

2. Update variants and task-specific coefficients.

The settings are:

TaskCD updateBaseline WDCDResidual WD
Vision pre-trainingMatrix-aligned0.050.010.04
LM pre-trainingMatrix-aligned0.100.020.08
COCO detectionUpstream rescaling0.050.010.04
Downstream fine-tuningUpstream rescaling0.050.020.03

Fine-tuning uses stronger CD because its schedule is shorter. Both variants aim to align CD's regularization strength with the replaced WD component, enabling a fair comparison with the WD-only baseline.

To verify that the two alignment methods yield similar results, we compare them under the same detection setting:

Training variantRepQ-ViT W4A4Percentile W4A4
Baseline29.7 / 29.025.6 / 25.2
CD with upstream rescaling34.4 / 33.129.6 / 28.9
Matrix-aligned CD34.1 / 33.030.0 / 29.3

Entries are box / mask mAP (%). Across 12 FFN pairs, matrix-aligned-to-rescaling ratios of layer-averaged Frobenius norms are 1.006, 1.016, and 0.987 for FC1, FC2, and their product. Thus, the variants yield similar weight scales and quantized performance in this setting.

We will document the coefficients and distinguish their norm-matching properties.

3. SwiGLU overhead and activation quantization sites.

Overhead: Under the ModernLM setup in Appendix A.8, the optimized full CD implementation, including SwiGLU, increases step time from 880.12 to 892.05 ms (+1.36%) and peak allocated memory from 66,323.11 to 66,514.36 MiB (+0.29%). Both runs use eight GPUs and 1,400 steps, with statistics collected over the final 200 steps. Independent CD computations are distributed across GPUs, followed by grouped weight broadcasts.

Activation quantization: The proxy in Appendix A.2 targets post-down structural amplification. CD applied to the down-projection matrix does not directly modify the pre-down matrices or their outputs. However, comparing the full ModernLM 60B-token CD and baseline runs shows smaller activations and lower quantization errors at both locations. At layer 24, which has the largest module output:

DiagnosticBaselineCD
Maximum pre-down activation17699
Maximum post-down activation3,984280
Pre-down relative error, SmoothQuant W8A89.59%7.03%
Post-down relative error, SmoothQuant W8A815.76%7.76%

The reported relative errors compare module outputs computed from the same input, with input and weight quantization enabled versus disabled. The preceding parts of the model are already quantized in both cases. These results connect CD to lower errors at the quantized input and resulting output.

We will add these measurements and clarify this distinction.

Dear Reviewer QVZa,

We appreciate your careful assessment of our work and your helpful questions about the experimental comparisons and implementation details.

As the discussion period closes in about three days, we are writing with a gentle reminder about our rebuttal. In our response, we provide:

  • A matched comparison of loss-based and decoupled CD.
  • Task-specific update settings and a comparison of the two alignment methods.
  • SwiGLU overhead measurements and clarification of the proxy's relation to activation quantization.

We would be grateful to hear whether our rebuttal resolves your concerns. If any point remains unclear or you have further questions, we would welcome the opportunity to respond before the discussion ends.

Thank you once more for your time and constructive feedback.

Kind regards,
The Authors

Dear Reviewer QVZa,

As the discussion period draws to a close, we would like to thank you for your time, your recognition of the merits of our work, and your constructive suggestions.

We hope that our responses have addressed your questions and reinforced your positive view of the paper. We are grateful for the support your favorable assessment provides as our paper is considered for acceptance.

We wish you all the best in your research and future work.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response: Quantization Behavior and CD Design

Thank you for following up on our response and for your positive assessment of CD's design. Below, we provide a clearer explanation addressing your questions.

Question 1. Why does full replacement retain good FP accuracy but fail under quantization, and what does CD budget change?

We refer to the FLAMBé checkpoint as full replacement and the per-head CD checkpoint as CD budget.

Part 1. Locating the accuracy collapse. Quantizing only O projections severely degrades full replacement, whereas quantizing QKV or MLP alone largely preserves accuracy:

Quantized positionsfull replacement Top-1 (%)CD budget Top-1 (%)
QKV only78.51682.422
O projections only16.79782.422
MLP only79.29782.031

These interventions retain the same calibrated W4A4 scales.

Part 2. Why the first block already fails. We measure relative output error as RRMSE=100∥yq−yFP∥2∥yFP∥2\mathrm{RRMSE}=100\frac{\lVert y_q-y_{\mathrm{FP}}\rVert_2}{\lVert y_{\mathrm{FP}}\rVert_2}. In block 0 (B0), errors along the quantized path diverge at AV and increase further through O:

B0 outputfull replacement RRMSECD budget RRMSE
V linear21.64%22.14%
AV209.59%40.92%
O projection831.61%39.59%

At AV, large V activations in a few heads dominate the shared per-tensor quantization scale, reducing precision for smaller-head signals. O then propagates/amplifies this upstream distortion.

Part 3. Relation to regularization strength. On the same checkpoints, we compare each factor's radial regularization with baseline WD at that position:

rW=⟨W,Dreg⟩FλWD∥W∥F2,r_W=\frac{\langle W,D_{\mathrm{reg}}\rangle_F}{\lambda_{\mathrm{WD}}\|W\|_F^2},

where DregD_{\mathrm{reg}} is the total regularization direction, including WD, and λWD\lambda_{\mathrm{WD}} is the baseline WD strength. Thus, rW=1r_W=1 matches baseline WD. B0 medians across heads are:

B0 metric (median across heads)full replacementCD budget
V radial strength rVr_V6.8451.017
O radial strength rOr_O0.014361.000
Factor-scale ratio ∥Oh∥F2/∥Vh∥F2\|O_h\|_F^2/\|V_h\|_F^2497.141.403

The factor-scale ratio is measured directly from weights, without WD normalization. Full replacement shows both unequal relative shrinkage and severe factor imbalance. CD budget stays close to the WD reference and has more balanced O/V factors, supporting WD strength as a useful prior for allocating regularization.

Question 2. How do matrix pairing and downstream-only updates explain the remaining gap?

As shown in Figure 3, CD uses functional (func.) pairs, (V,O)(V,O) and (FC1,FC2)(\mathrm{FC1},\mathrm{FC2}), and composable (comp.) pairs, normalization scales with QKV or FC1. CD updates each pair's downstream matrix; adapted FLAMBé updates both factors of per-head OV pairs.

Each error column measures final-logit RRMSE after quantizing only the named positions across all blocks. All positions combines FC1/FC2/QKV/O. Errors use a fixed 256-image subset; full-network W4A4 Top-1 uses the complete validation set.

ModelFC1-onlyFC2-onlyQKV-onlyO-onlyAll positionsRepQ W4A4 Top-1
Baseline44.847%48.689%35.473%26.562%83.211%76.950%
Func., both sides41.419%52.879%41.135%25.823%86.598%76.522%
Func. + comp., downstream each (CD)35.499%43.117%29.359%25.634%69.562%78.712%
FLAMBé OV, WD-aligned40.893%45.848%34.583%22.930%72.471%77.148%

Func., both sides versus CD. Two-sided functional updates increase FC2-only logit error and leave larger FC1/QKV errors than CD's func. + comp. allocation. At the upstream matrices, composable-pair regularization is therefore more effective than applying the functional objective.

FLAMBé OV versus CD. FLAMBé has the lowest O-only error but larger FC1/FC2/QKV errors. Its factorization-based OV objective omits CD's FFN and normalization-to-projection pairs. CD's quantization-oriented coverage gives lower combined error and higher W4A4 accuracy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.