acceptodds
Under review as a conference paper at ICLR 2027

CARE-Attn: Component-Aware Residual Error Correction for FP4 Attention QAT

Abstract

Recent advances in attention quantization-aware training (QAT), notably Attn-QAT, have substantially narrowed the quality gap of FP4 attention. Nevertheless, a measurable gap to higher-precision attention remains, and end-to-end recovery alone does not reveal what QAT actually adapts to inside the attention operator. We revisit FP4 attention from a component-wise perspective and find that Attn-QAT adapts highly unevenly: it substantially reduces the error from quantized queries and keys (), while errors from attention probabilities () and values () barely change and a non-negligible residual remains. These residuals also have distinct structures: the residual is sparse and heterogeneous across layers and heads, the error is limited by the quantizer, and the error is associated with channel-wise offsets. We therefore propose CARE-Attn (Component-Aware Residual Error Correction for Attention), which corrects each residual separately: it recomputes predicted high-error tiles in higher precision under adaptive budgets, decouples training and inference precision through -exact QAT, and removes channel offsets from with exactly compensated mean-centering. On video diffusion transformers, CARE-Attn improves generation quality over Attn-QAT, produces outputs closer to those of the BF16 fine-tuned model, and reduces QAT overhead; on language models, it consistently lowers perplexity and substantially improves long-context retrieval. These results show that attention QAT alone is not enough: explicitly correcting the residuals it leaves is key to accurate FP4 attention. Code is available at https://anonymous.4open.science/r/CARE-Attn.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
QAT, attention quantization, quantization-aware training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FmV3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CARE-Attn analyzes residual FP4 attention errors after QAT and applies separate corrections to QK, probabilities P, and values V. It reports improved VBench quality for Wan2.1, lower perplexity on four language models, and reduced training overhead from removing auxiliary probability accumulation.

Critical strengths

The component-wise analysis motivates distinct corrections, and the paper includes ablations, comparisons against the released Attn-QAT checkpoint and after-training corrections, plus training and attention-kernel throughput measurements.

Critical weaknesses
  • P-exact QAT leaves probabilities unquantized during training but applies two-level FP4 quantization at inference. This deliberate train–test mismatch is supported by the reported results, but the paper needs to distinguish gains from P-exact training from gains due to changing the inference probability quantizer.

  • The full method's efficiency evidence is split between training-step time and attention-kernel throughput; it does not report end-to-end video-generation latency including online tile selection. The evidence therefore supports low kernel-level overhead more directly than faster generation.

Questions for authors
  1. In the main comparison and ablations, do Attn-QAT and CARE-Attn use the same two-level P quantizer at inference, with P-exact training as the only difference? Please identify the exact baseline configuration.
  2. Does the reported attention throughput include CARE-Attn's online tile scoring and selection, or only the attention kernel after the correction mask has been generated?
4 comments
AuthorsOpenReview ↗

Response to Reviewer FmV3 (1/2)

We thank the reviewer for identifying two important ambiguities. We now separate P-exact training from two-level P inference with controlled experiments, and report end-to-end generation latency including online tile selection.

Q1 / W1. Baseline configuration and attribution

The reviewer is correct that Attn-QAT and CARE-Attn use different P quantizers during inference. Attn-QAT trains and infers with single-level P quantization, whereas CARE-Attn uses P-exact training followed by two-level P quantization at inference. Consequently, the original “P-exact QAT” row in Table 3 changed both the training procedure and the inference quantizer. To isolate their contributions, we conducted controlled experiments on Wan2.1-14B, Qwen3-8B, and Llama-3.1-8B.

Table R1. Effect of P training and inference quantization on Wan2.1-14B. Each entry reports weighted VBench / PSNR (dB) over the same 326 prompts and sampling seed. PSNR is measured against the BF16 fine-tuned reference. All configurations exclude V-centering and QK correction.

Training configurationSingle-level P inferenceTwo-level P inference
Attn-QAT, released0.8390 / 11.340.8404 / 11.60
Attn-QAT, reproduced (matched recipe)0.8435 / 12.600.8430 / 12.61
Two-level P fake-quantization—0.8460 / 14.87
P-exact training0.8447 / 13.320.8464 / 14.53

Using the reproduced Attn-QAT checkpoint as a controlled baseline, switching to two-level P inference alone barely changes PSNR (12.60 to 12.61 dB). In contrast, P-exact training improves PSNR by 0.72 dB under single-level inference and by 1.93 dB under two-level inference. These results show that the improvement cannot be attributed to the inference quantizer alone. Rather, P-exact training contributes independently, and its benefit is greater when paired with two-level inference.

We observe the same interaction on language models, where we additionally trained P-exact-only checkpoints without training-time QK correction. All results below use the same three-tile QK correction at inference, including the Attn-QAT checkpoints, so the training configurations can be compared under identical inference settings.

Table R2. Controlled P-pathway ablation on LLMs (C4 perplexity at 8,192 tokens; lower is better).

Model / training configurationSingle-level P inferenceTwo-level P inference
Qwen3-8B: Attn-QAT11.393111.4216
Qwen3-8B: P-exact only11.374411.3401
Llama-3.1-8B: Attn-QAT (seeds 42 / 7)9.3589 / 9.36859.4549 / 9.4602
Llama-3.1-8B: P-exact only (seeds 42 / 7)9.4398 / 9.44269.3371 / 9.3389

Simply applying two-level inference to Attn-QAT weights increases perplexity on both models, whereas it reduces perplexity for P-exact weights. With inference held fixed at two-level P, P-exact training improves perplexity from 11.4216 to 11.3401 on Qwen3-8B and from 9.4549 to 9.3371 on Llama-3.1-8B. The Llama result is reproduced with a second independent training seed (9.4602 to 9.3389). This confirms that P-exact training and two-level inference are complementary, rather than either being sufficient on its own.

We also evaluated training directly with two-level P fake quantization. This achieves comparable video quality to P-exact training (VBench 0.8460 vs. 0.8464) and slightly higher PSNR (14.87 vs. 14.53 dB), but increases training time by 28% (93.4 vs. 72.8 s/step) due to probability fake quantization and auxiliary output accumulation. P-exact therefore provides a more favorable quality–training-cost trade-off, without requiring the more expensive training path.

We have clarified the baseline configurations and renamed the original Table 3 row to “P-exact training + two-level P inference” in the revised PDF. The controlled LLM results are included in Table 7 (Appendix B.2), and the additional Wan2.1-14B comparisons are reported in Tables 11–12 (Appendix B.3).

AuthorsOpenReview ↗

Response to Reviewer FmV3 (2/2)

Q2 / W2. Online selection and end-to-end efficiency

We confirm that the throughput measurements in Fig. 4(d) have already included the complete CARE-Attn operator, with online tile scoring, top-k selection, FP4 quantization, sparse BF16 recomputation, and result merging. No selection masks or indices are precomputed. The original benchmark was conducted at 24k tokens with a uniform 1% QK correction budget.

To further assess deployment efficiency, we measured the complete operator at Wan2.1-14B's deployed input shape (1280×720, 81 frames, 75,600 tokens) using the same uniform 1% budget as Fig. 4(d). Including online selection, CARE-Attn retains 94.7%, 94.5%, and 94.4% of Attn-QAT's attention throughput at SP=1, 4, and 8, respectively, and matches SageAttention3.

We also measured the complete video-generation latency, including text encoding, 50 denoising steps with classifier-free guidance, online selection, and VAE decoding.

Table R3. End-to-end Wan2.1-14B generation latency (s/video; lower is better).

MethodSP=4SP=8
BF16 (FlashAttention-2)1013.8515.0
Attn-QAT775.5395.7
CARE-Attn795.9407.5

All measurements use one eight-GPU RTX PRO 5000 node. SP=4 runs two concurrent requests, with per-request latency reported, while SP=8 runs one request.

CARE-Attn adds only 2.6%–3.0% end-to-end latency relative to Attn-QAT, while maintaining a 1.26–1.27× speedup over BF16. These results confirm that online selection is included in our efficiency measurements and that the additional correction cost remains modest in the complete video-generation pipeline.

We have clarified the benchmark configuration in Fig. 4(d) and added the deployment-resolution operator measurements and end-to-end latency results to Table 16 in the revised PDF, with the main findings summarized in Sec. 5.4.

We sincerely thank the reviewer for the careful evaluation and constructive feedback. We hope the new controlled experiments have clarified the respective contributions of P-exact training and two-level P inference, while the end-to-end measurements have addressed the concerns about online selection and practical efficiency. We would be grateful if the reviewer would reconsider the overall assessment in light of these additional results.

Thank the authors for the additional experiments, which resolve my questions on attribution and online tile selection. However, against the reproduced Attn-QAT baseline, the VBench gain is smaller than the gap between the released and reproduced baselines themselves, so much of the reported improvement reflects the training recipe rather than the method. Moreover, training with two-level fake-quantized P matches P-exact QAT in quality, so P-exact mainly reduces training cost; I therefore maintain my original score.

We thank the reviewer for the follow-up and for acknowledging that our additional experiments resolve the attribution and online-selection concerns.

We would like to clarify that both observations regarding the baseline recipe difference and the comparable quality of two-level P fake-quantized training were already reported in our original submission (Table 11 in Appendix B.3 and Sec. 5.3, respectively).

Importantly, we reproduced Attn-QAT precisely to ensure a reliable and controlled comparison. Since the exact training data and conditions underlying the released checkpoint of Attn-QAT were not fully available to us, comparing against it alone could conflate differences in the training recipe with the contributions of our method. We therefore trained Attn-QAT using the same data, training recipe, GPU count and topology, and batch configuration as CARE-Attn, enabling a more controlled comparison. Under these matched conditions, CARE-Attn still improves weighted VBench from 0.8435 to 0.8472. Although this gain is more modest than the comparison against the released checkpoint, it is the more credible estimate of the improvement attributable to CARE-Attn, precisely because the major training-related confounding factors have been controlled.

We would also appreciate the reviewer considering the complementary fidelity results in Table 12 of the original submission. Against the reproduced Attn-QAT baseline, CARE-Attn improves PSNR from 12.60 to 15.76 dB, SSIM from 0.4913 to 0.6162, and LPIPS from 0.4807 to 0.3147. These matched-baseline results provide further evidence that the benefits of our method extend beyond what is captured by the weighted VBench score alone.

We agree that two-level P fake-quantized training achieves comparable quality to P-exact training, as already discussed in Sec. 5.3. The advantage of P-exact training is its lower training cost while maintaining comparable quality.

We appreciate the reviewer's careful consideration and feedback.

Reviewer_448MOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes the error of FP4 attention component by component and proposes CARE-Attn, which applies a separate correction to each error source. The existing method, Attn-QAT, greatly reduces the error from QK, i.e., the relevance scores between tokens, but barely reduces the error from the attention probabilities P and the values V that carry the actual information. CARE-Attn recomputes only a small part of the high-error QK computation in higher precision, reduces the problem of small P values vanishing to zero, and subtracts the channel-wise mean from V before adding it back. As a result, it generally outperforms Attn-QAT on both video and language models. It also keeps about 97% of Attn-QAT’s attention kernel throughput.

Critical strengths
  1. Instead of comparing only end-to-end performance, the paper first analyzes where the error comes from among QK, P, and V.
  2. It applies a different solution to each of the distinct problems found in QK, P, and V. The link between the analysis and the method design is clear.
  3. It compares against both the released Attn-QAT model and an Attn-QAT model retrained under the same conditions.
  4. An ablation that adds each correction one at a time shows which part contributes to the improvement.
  5. It evaluates on several language models as well as video models, and reports both accuracy and attention speed.
Critical weaknesses
  1. The video models use a single training seed. For the language models, too, repeated runs with different seeds are limited to only some models. Mean performance and the range of variation are not reported.
  2. Inference speed is measured only for the attention kernel, not the full model. The time to generate a full video or the end-to-end response speed of the language models is unknown.
  3. Video quality is evaluated only with automatic metrics, and some sub-metrics are lower than Attn-QAT.
  4. The V correction actually increases the error on language models, so it is not used there. Since the set of corrections depends on the model, a clear criterion is needed for choosing which corrections to apply to a new model.
Questions for authors

See Weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer 448M (1/2)

We thank the reviewer for the constructive suggestions. We have added independent training runs, a blinded human GSB evaluation, and end-to-end latency measurements, and clarified how CARE-Attn selects corrections. We address the four concerns below.

W1. Repeated runs and variability

To assess the stability of our results, we conducted additional independent training runs for both language and video models.

Language models. We trained a second seed for Qwen3-14B, in addition to the two training seeds already reported for Qwen3-8B and Llama-3.1-8B. The table reports C4 perplexity at 8,192 tokens as the mean over the two seeds, with the two per-seed values in brackets.

ModelAttn-QATCARE-AttnImprovement per seed
Qwen3-8B11.5409 [11.5407, 11.5411]11.3182 [11.3178, 11.3185]0.222 / 0.223
Llama-3.1-8B9.5518 [9.5480, 9.5555]9.2920 [9.2908, 9.2932]0.255 / 0.265
Qwen3-14B10.5998 [10.5989, 10.6007]10.4496 [10.4480, 10.4512]0.151 / 0.150

CARE-Attn consistently achieves lower perplexity across both seeds of all three models, with the variation between seeds substantially smaller than the improvement over Attn-QAT. We observe a similar trend in long-context retrieval, where CARE-Attn raises exact-match accuracy on Qwen3-14B from 86.6% to 91.9% and from 85.8% to 92.5% for the two seeds.

Video model. We also repeated the complete Wan2.1-14B CARE-Attn training with a different random seed (2000 instead of 1000), keeping the training recipe and adaptive QK budget unchanged. Evaluated on the same 326 prompts and generation seed, the two runs achieve nearly identical weighted VBench scores of 0.8472 and 0.8480. Both outperform the Attn-QAT baseline (0.8435), with improvements of +0.0037 [0.0001, 0.0075] and +0.0044 [0.0012, 0.0078], respectively (paired 95% bootstrap confidence intervals).

These additional experiments show that the improvements are reproducible across different training seeds for both LLMs and video models. We have included the per-seed LLM results in Table 6 and the additional Wan2.1-14B training run in Table 11 of the revised PDF.

W2. End-to-end efficiency

To address the reviewer's concern, we measured the end-to-end generation latency of Wan2.1-14B on eight RTX PRO 5000 GPUs, including text encoding, 50 denoising steps, online QK tile selection, and VAE decoding. At SP=4/8, CARE-Attn takes 795.9/407.5 seconds per video, compared with 775.5/395.7 seconds for Attn-QAT and 1013.8/515.0 seconds for BF16. CARE-Attn therefore maintains a 1.26–1.27× end-to-end speedup over BF16, with only 2.6%–3.0% additional latency relative to Attn-QAT. These end-to-end measurements use the deployed adaptive QK budgets. In a separate operator benchmark at the same input shape, the CARE-Attn operator retains 94%–95% of Attn-QAT's throughput, including online selection.

For LLMs, we additionally benchmarked causal prefill with a native CUDA operator. With the fixed three-tile budget, the complete attention operator adds 4.3%–11.6% over uncorrected FP4 attention at 8k–64k tokens and remains 2.07–2.52× faster than BF16 FlashAttention-2. In full-model Llama-3.1-8B prefill, the correction adds 0.7%–1.5%, while CARE-Attn is 1.04–1.22× faster than BF16 at 8k–32k. These are prefill measurements; serving-engine integration, paged KV cache, and decode remain future work.

We have added the video end-to-end results to Table 16 and the LLM operator and full-model prefill results to Table 17 in Appendix B.5.

AuthorsOpenReview ↗

Response to Reviewer 448M (2/2)

W3. Human evaluation and video sub-metrics

To complement the automatic video-quality metrics, we conducted a blinded human preference evaluation comparing CARE-Attn with Attn-QAT baseline. We sampled 100 prompts from VBench, stratified by dimension group.

CARE-Attn was preferred on 23 prompts, compared with 10 for Attn-QAT, while the remaining 67 were judged as ties. This corresponds to a net GSB of +13 percentage points, with a two-sided sign-test p-value of 0.035 over the 33 non-tied comparisons. These results provide additional human-evaluation evidence in favor of CARE-Attn, complementing the automatic metrics and qualitative comparisons.

We also acknowledge that CARE-Attn does not improve every VBench sub-metric. Relative to the reproduced Attn-QAT baseline, temporal flickering decreases slightly by 0.0014 (95% CI: [−0.0023, −0.0005]), while the difference in motion smoothness is negligible (−0.0001, 95% CI: [−0.0007, +0.0004]). The revised PDF reports these trade-offs rather than claiming consistent improvement across all dimensions.

We have added the human preference study and its evaluation protocol to Appendix B.3 of the revised PDF, alongside the additional video-quality analyses.

W4. Correction-selection criterion

We agree that the criteria for selecting different corrections should be made explicit. We have therefore added a calibration-guided procedure based on the component-wise error analysis in Eq. (3). Using representative inputs, we measure the isolated quantization errors of QK, P, and V to determine which corrections are beneficial for a given model.

For the P pathway, we adopt two-level quantization when it substantially reduces P error. For QK, we calibrate the correction budget after QAT using residual-error recovery curves, assigning per-(layer, head) budgets when errors vary across heads, as in Wan, or a small uniform budget when improvements saturate quickly, as in the evaluated LLMs.

For V, mean-centering is enabled only when it reduces quantization error and supports exact compensation at inference. Our measurements show that centering reduces V error by 6.3% on Wan2.1-14B but increases it by 1.68–2.34× on the evaluated LLMs, explaining why we apply it only to video models.

We have clarified this selection procedure in Sec. 5.1 and added the corresponding component-wise error measurements to Table 4 of the revised PDF.

We sincerely thank the reviewer for the constructive feedback. We hope the additional training-seed evaluations, human preference study, end-to-end latency measurements, and clarified correction-selection procedure have addressed the concerns raised in the review. We would be grateful if the reviewer would consider these new results when reconsidering the overall assessment.

Reviewer_8tueOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper builds on Attn-QAT and investigates how attention QAT adapts to different sources of quantization error. On Wan2.1-14B, the authors find that QAT substantially reduces QK-induced error but produces little improvement in P- and V-induced errors. They propose CARE-Attn, which combines sparse high-precision QK correction, P-exact QAT that eliminates the additional auxiliary output O′O', and mean-centered V quantization with exact offset compensation. At inference, P is quantized using two-level scaling. The configuration is adapted to each model family: video models use adaptive QK budgets and V-centering, whereas LLMs use uniform QK budgets without V-centering. The method improves generation quality on Wan and perplexity and long-context retrieval on LLMs relative to Attn-QAT.

Critical strengths
  1. The component-wise analysis is insightful and provides a clear motivation for the method. Understanding which quantization errors QAT reduces, and which remain, is a useful contribution.
  2. The proposed corrections are well motivated. In particular, adaptive QK precision and P-exact QAT follow naturally from the analysis. Eliminating the auxiliary output also reduces training overhead.
  3. The experimental results are convincing overall. CARE-Attn improves aggregate video quality and fidelity to the BF16 reference, and consistently improves LLM perplexity and long-context retrieval over Attn-QAT.
  4. Overall, this is a crispy paper with a coherent connection between its analysis, method, and empirical results.
Critical weaknesses
  1. Some individual techniques have precedents in the SageAttention series, including two-level P scaling and V-centering. This limits the novelty of those components in isolation. However, their use within QAT is well motivated, and I consider the component-wise analysis and integration meaningful contributions rather than viewing this as a major weakness.
  2. Although the paper includes LLM quality evaluations and ablations, the efficiency evaluation focuses on Wan. Integration into an LLM serving engine, with end-to-end latency and throughput measurements across context lengths and batch sizes, would strengthen the practical evaluation.
Questions for authors
  1. Could the authors report single-level P quantization error both before and after Attn-QAT on Wan, alongside the corresponding two-level results? Figure 1(a)’s P-error values appear consistent with the two-level setting, but the quantizer used there is not explicit. Since Attn-QAT trains with single-level P quantization, this comparison would help clarify the conclusion that QAT produces little adaptation in the P pathway.
  2. Could the training-component and QK-budget ablations in Appendix B.2 be extended beyond Qwen3-8B, for example to Llama-3.1-8B? This would help establish whether the contributions of P-exact QAT, two-level P quantization, and training-time QK correction generalize across architectures.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 8tue (1/2)

We thank the reviewer for the positive assessment and constructive questions. We have measured single-level and two-level P errors before and after Attn-QAT, extended the training-component and QK-budget ablations to Llama-3.1-8B, and clarified the scope of our LLM efficiency evidence.

W1. Relation to SageAttention

We thank the reviewer for noting the connection to SageAttention. While two-level P scaling and V mean subtraction build on techniques introduced in SageAttention3 and SageAttention2, respectively, our work focuses on understanding the residual quantization errors after attention QAT and designing the training procedure accordingly. In particular, P-exact training enables effective two-level P inference without probability fake quantization during training, while sparse QK correction addresses the remaining QK quantization error. The revised PDF's related-work discussion now clarifies these connections and the distinction from prior work.

W2. LLM serving efficiency

We agree that an end-to-end LLM serving evaluation would strengthen the practical assessment of CARE-Attn. We therefore benchmarked causal prefill with a native CUDA kernel; integration with a serving engine, paged KV cache, and decode remains future work.

With the paper's fixed three-tile budget, the CUDA attention operator, including online tile selection, quantization, and mask construction, adds 4.3%–11.6% over the uncorrected FP4 operator at 8k–64k tokens and remains 2.07–2.52× faster than BF16 FlashAttention-2. In full-model Llama-3.1-8B prefill, the correction adds only 0.7%–1.5%, while CARE-Attn is 1.04–1.22× faster than BF16 at 8k–32k. For video, CARE-Attn achieves a 1.26–1.27× end-to-end speedup over BF16, including online selection.

We have added the video end-to-end results to Table 16 and the causal-operator and full-model prefill measurements to Table 17 in Appendix B.5.

Q1. Does the P-pathway conclusion hold with single-level quantization?

Yes. Our additional measurements confirm that Attn-QAT does not reduce P quantization error, even when evaluated with the single-level quantizer used during its training and inference. Figure 1(a) originally reported P error under two-level quantization without specifying this setting in the caption. To address the reviewer's concern, we repeated the Wan2.1-14B component-isolation analysis using both P quantizers (Table 15 in the revised PDF).

Table R1. Relative attention-output error before and after Attn-QAT (lower is better).

Attn-QAT with single-level PeQKe_{QK}ePe_PeVe_V
Before QAT0.05610.10760.0240
After Attn-QAT0.03650.11430.0235
Attn-QAT with two-level PeQKe_{QK}ePe_PeVe_V
Before QAT0.05610.02200.0240
After Attn-QAT0.03700.02140.0237

While Attn-QAT reduces eQKe_{QK} by approximately 35%, single-level ePe_P actually increases by 6.2%, and both two-level ePe_P and eVe_V change by less than 3%. The underlying quantization difficulty also persists: approximately 77% of P micro-block scales underflow with single-level quantization, compared with 21% under two-level scaling. Switching to two-level scaling reduces ePe_P from 0.1143 to 0.0214 (5.3×).

These results establish that the limited adaptation of the P pathway still holds under the single-level quantization setting. We have clarified the quantization setting in the revised figure caption and included the additional measurements in Table 15.

AuthorsOpenReview ↗

Response to Reviewer 8tue (2/2)

Q2. Do the QK-budget and training ablations generalize to Llama-3.1-8B?

(a) QK correction budget. We extended the QK-budget sweep from Qwen3-8B to Llama-3.1-8B and observed the same trend: most of the improvement is obtained with only a few corrected tiles.

Table R2. Llama-3.1-8B C4 perplexity at 8,192 tokens (training seed 42; lower is better).

Corrected key tiles per query tile0123468All
Attn-QAT weights9.54809.38609.36439.35899.35859.35529.35419.3501
CARE-Attn weights9.85979.34549.30259.29329.28869.28529.28149.2765

For CARE-Attn, increasing the budget from one to three tiles reduces perplexity from 9.3454 to 9.2932, whereas increasing it further to eight tiles brings only a small additional improvement to 9.2814. This diminishing return supports our choice of a three-tile budget for LLM inference, consistent with the Qwen3-8B results.

We have added the Llama-3.1-8B results to Table 8 in Appendix B.2 of the revised PDF.

(b) Training-component ablations. To examine whether the contributions of P-exact training and QK correction also generalize, we trained additional Llama-3.1-8B checkpoints using P-exact training without training-time QK correction. For a controlled comparison, all configurations below use the same three-tile QK correction at inference, including the Attn-QAT checkpoints. We also report both single-level and two-level P inference to distinguish the effect of training from that of the inference quantizer.

Table R3. C4 perplexity at 8,192 tokens (training seed 42; lower is better).

Training configurationInference PQwen3-8BLlama-3.1-8B
Attn-QATSingle-level11.39319.3589
Attn-QAT (same weights)Two-level11.42169.4549
P-exact onlySingle-level11.37449.4398
P-exact onlyTwo-level11.34019.3371
Training-time QK correction onlySingle-level11.37479.3367
Full CARE-Attn (P-exact + QK correction)Two-level11.31859.2932
Two-level P fake-quantized training + QK correctionTwo-level11.34499.2969
BF16 fine-tune (reference)BF16 attention11.23639.2347

The Llama results confirm the contributions of both training components under matched inference settings. With two-level P inference and the same QK correction budget, replacing Attn-QAT training with P-exact training reduces perplexity from 9.4549 to 9.3371 on Llama and from 11.4216 to 11.3401 on Qwen. Adding QK correction during training further reduces perplexity to 9.2932 and 11.3185, respectively. Importantly, P-exact training is beneficial in combination with two-level P inference, rather than under every inference setting; on Llama, it performs worse than Attn-QAT when both use single-level P inference. Although training with two-level P fake quantization achieves comparable perplexity, P-exact avoids the additional training cost (9.2 vs. 10.4 s/step on Qwen3-8B and 72.8 vs. 93.4 s/step on Wan2.1-14B).

We have added the Llama-3.1-8B ablations to Table 7 in Appendix B.2 of the revised PDF.

We sincerely appreciate the reviewer's positive assessment and constructive suggestions. We hope the additional P-error measurements and Llama-3.1-8B ablations have addressed the two questions raised in the review. We would be grateful if the reviewer would consider these new results when reconsidering the overall assessment.

Thanks for the additional results, which addressed my questions. I raised my assessment to clear acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.