- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
With the backbone frozen, the quantized weights determine where adaptation starts. Prior methods reconstruct pretrained weights or activations; the authors argue post-adaptation target risk matters instead. TAQI weights input second moments by BF16 output-gradient energy, blends them with general calibration statistics, and reuses this geometry for the low-rank basis and GPTQ-intrinsic scan, adding an output-group gate and persistent mask. Two backbones, three bit widths, three QA tasks.
Critical strengths
1. Well posed. Treating the quantized point as the object of study is useful, and translating with rather than nesting blocks the assumption that higher precision gives a better start. Eq. (11)'s Schur complement, tying a code's cost to the residual low-rank coordinates can absorb, is the strongest technical content.
2. Solid coverage. Five of six comparisons per bit width favor TAQI; appendices give full metrics, and Table 7 shows error reallocated, not merely reduced. MedQA deficits and the 2-bit macro-F1 of 10.855 are reported as measured.
Critical weaknesses
1. Task awareness is not isolated. TAQI uses 128 labeled in-domain examples and BF16 backward statistics that GPTQ-intrinsic, calibrated on general data, never sees. Each Table 1 gap therefore admits two readings: gradient weighting steered reconstruction toward task-sensitive directions, or calibration merely moved in-domain. Appendix E.6's XJ control separates these but appears only in Sec. 4.4's static analysis. The missing control is an in-domain, unweighted calibration baseline; without it, Table 1 supports in-domain initialization, not the mechanism the title promises.
2. The motivation does not establish that reconstruction fidelity misaligns with adaptation. Fig. 1 compares BF16 with NF4, a precision difference rather than a ranking of initializers at fixed width; page 2 concedes it does not measure reconstruction error, and Fig. 1(b)'s axes share the BF16+LoRA score, which C.1 calls mathematically coupled. Three of five reversals also involve base models whose outputs are almost entirely invalid, so those gaps mainly reflect output validity. This weakens the reversal evidence without touching the other wins, but it leaves the premise resting largely on Proposition 1's toy counterexample.
Questions for authors
Q1. When would Eq. (8)'s initialization-time sensitivity still predict Eq. (2)'s ?
Q2. Could GuidedQuant and Fisher-weighted LQ-LoRA be run under TAQI's joint initialization and fine-tuning protocol?
Q3. Table 2's “Select.” bundles the output-group gate with the persistent mask; can their contributions be separated?
Q4. In Table 1, 2-bit GPTQ-intrinsic scores 29.081 on TAT-QA, below plain GPTQ+LoRA's 34.737; was it tuned, and what fraction of groups does the gate accept?
3 comments
Response to Reviewer 6oYD (Part 2/2): Baselines, gate/mask, tuning and acceptance (Q2, Q3, Q4)
GuidedQuant and Task-Fisher LQ-LoRA under our protocol (Q2). Yes; both have been run. Task-Fisher LQ-LoRA (Guo et al., 2024) uses the official implementation, with Fisher computed on our exact 128 task calibration examples and fine-tuned under our protocol (rank 16, α = 32, effective batch 64, three epochs). Over three training seeds on Granite-350M at 4 bits (mean ± std over training seeds 42/43/44), TAQI achieves a higher mean on all three tasks, by 0.50 / 1.37 / 1.49 points on MedQA / TAT-QA / QASPER:
| Task | Metric | Task-Fisher LQ-LoRA | TAQI |
|---|---|---|---|
| MedQA | Accuracy | 32.522 ± 0.491 | 33.019 ± 1.192 |
| TAT-QA | F1 | 44.154 ± 0.573 | 45.526 ± 0.109 |
| QASPER | Answer F1 | 22.038 ± 0.486 | 23.532 ± 0.700 |
Because both methods receive the same task data, this shows that TAQI's advantage comes from its construction rather than from the calibration set. GuidedQuant+LoRA (Kim et al., 2025) has also been run, at 4 bits on Qwen3.5-0.8B: it uses all 128 in-domain calibration examples, keeps the guided quantized weights frozen and trains a fresh adapter with rank 16, α = 32 and three epochs, and reaches 45.247 / 62.219 / 21.830 (MedQA accuracy / TAT-QA F1 / QASPER answer F1). GuidedQuant is a weight-only quantizer with its own calibration pipeline, and its quantized weights have to be prepared separately for every backbone and task. Within the rebuttal period we completed all three tasks on Qwen3.5-0.8B, so this is the setting we report, under GuidedQuant's own evaluation pipeline; the matched three-seed comparison above is the one run fully under our protocol.
Separating the gate and the persistent mask (Q3). The gate and the mask are two stages of one selection protocol: the gate decides which output groups keep the joint candidate, and the persistent mask keeps the rejected groups inactive during fine-tuning. The component ablation in the paper (Table 2) separates this protocol from the task-conditioned construction: TAQI-core, without the protocol, improves all three primary metrics over GPTQ-intrinsic, and the full protocol improves them further. To separate the two stages, we built three arms for this control (Granite-350M, 4 bits):
| Arm | Gate | Mask | MedQA Acc. | TAT-QA F1 | QASPER Ans. F1 |
|---|---|---|---|---|---|
| No selection | off | off | 32.522 | 45.576 | 22.864 |
| Gate only | on | off | 32.993 | 45.153 | 22.865 |
| Gate + mask (TAQI) | on | on | 34.093 | 45.400 | 22.894 |
These arms are rebuilt for this control and are compared with each other. The last arm is evaluated with batched generation, the other two with single-example generation.
The two gate-on arms start from identical , , and initial adapter tensors in all 72 modules and differ only in the mask, so the gate alone determines the initialization and the mask acts only during fine-tuning. Across the three arms, TAT-QA and QASPER change by at most 0.42 and 0.03 points, while MedQA accuracy rises from 32.522 to 32.993 with the gate and to 34.093 with the gate and mask together. The selection protocol therefore acts as a safeguard on top of the task-conditioned construction, with its effect concentrated on MedQA. Below 4 bits, TAQI's gains come mainly from the task-conditioned construction (Appendix A.2, Table 4).
GPTQ-intrinsic tuning (Q4). No method in Table 1 was tuned individually: all methods, including TAQI, used one shared, preset learning rate, so the comparison gives no method a tuning advantage. In the 2-bit regime this shared setting affects the baselines. In the same 2-bit TAT-QA setting, plain GPTQ+LoRA reaches 44.278 F1 at its best tested learning rate (5e-5), which remains below TAQI's 48.190 at the shared default. TAQI's 2-bit advantage therefore holds even against a baseline with a tuned learning rate.
Gate acceptance (Q4). The gate accepts nearly all output-channel groups (size 64, β = 0.5, τ = 0.05):
| Model | Bits | MedQA | TAT-QA | QASPER |
|---|---|---|---|---|
| Qwen3.5-0.8B (6,240 groups) | 2 | 100.00 | 100.00 | 100.00 |
| Qwen3.5-0.8B | 3 | 99.79 | 99.38 | 99.38 |
| Qwen3.5-0.8B | 4 | 99.62 | 99.13 | 99.18 |
| Granite-350M (1,948 groups) | 4 | 99.95 | 99.69 | 99.79 |
The gate compares the candidate with the source . At 2 bits, the source error is large and the joint candidate improves both task and general error in every group, so all groups are accepted and the 2-bit TAQI model coincides with its ungated construction. At 3–4 bits, is already accurate and the 5% general-error tolerance is tighter, so fewer than 1% of groups are rolled back. This shows that the gate acts as a sparse safeguard and that TAQI's 2-bit gains come from the task-conditioned construction itself.
Thank you for your detailed response and the additional experiments. I will maintain my original rating.
Response to Reviewer 6oYD (Part 1/2): Mechanism and premise (W1, W2, Q1)
We thank the reviewer for the precise analysis and for identifying the controls that separate our mechanism from its confounders. Part 1 addresses W1/W2/Q1 and Part 2 addresses Q2–Q4.
Separating gradient weighting from in-domain calibration (W1). To address your concern that TAQI's gains might come from in-domain calibration alone, we ran the requested in-domain, unweighted control through the full pipeline. Both arms use the same 128 labeled in-domain examples, supervised positions, general statistic, basis, scan, fine-tuning and evaluation; the only difference is weighting each position by ∥g∥22 or uniformly. Both arms are built for this control with the gate disabled (Granite-350M, 4 bits):
Gradient weighting improves all three tasks, by +1.100 / +0.432 / +0.736 points, which shows that the gain comes from the weighting mechanism rather than from moving calibration in-domain. This matches the static analysis in Section 4.4 (Table 7), where, with identical examples, basis and scanner, weighting reduces the task-weighted error DT by 52.16–60.40% and the first-order task-loss perturbation Γ by 32.87–70.56% relative to the unweighted control. The mechanism is also visible in the data: task-gradient energy is concentrated on the answer positions rather than spread over the context (Appendix B), so uniform weighting dilutes the task signal with context tokens. Together, these results confirm that the task-aware weighting named in the title is what drives TAQI's improvement.
Reconstruction fidelity versus adaptation (W2). We agree that Figure 1 compares precision settings rather than initializers at a fixed width, and that Figure 1(b) serves as a descriptive summary for selecting benchmarks with real adaptation headroom; we will state this scope explicitly. The premise, however, is supported by two further lines of evidence. First, as Appendix C.1 separates, not all reversals come from invalid base outputs: on AttackQA, quantization lowers unadapted token F1 by 12.87 points, yet 4-bit+LoRA exceeds BF16+LoRA by 0.47 points after adaptation; on PubMedQA, a 2.60-point loss becomes a 2.60-point gain, and both tasks have informative base predictions. Second, at a fixed 4-bit width on real Granite layers (Section 4.4, Table 7), the task-guided initialization has 7.71–9.20% higher general reconstruction error while lowering the first-order task-loss perturbation by 77.40–81.19%. This shows directly, at the same width, that lower reconstruction error and lower task sensitivity need not coincide, which is the misalignment our premise describes.
When Eq. (8) predicts JK (Q1). Eq. (8) bounds the first-order task-loss perturbation at the reference model, whereas JK is the risk after K adaptation steps. Eq. (8) is informative about JK under four conditions, which we will state in Section 3.2:
These conditions describe the regime TAQI targets: low-bit backbones adapted with limited rank and fine-tuning budgets, which is the standard setting of quantized PEFT.