acceptodds
Under review as a conference paper at ICLR 2027

Searching the Quantized Weight Space for Adaptation: Task-Aware Joint Quantization and Low-Rank Initialization

Abstract

Quantized low-rank fine-tuning enables memory-efficient adaptation of large language models by training low-rank adapters on a frozen low-bit backbone. Since the backbone remains fixed during fine-tuning, the quantized weights selected at initialization define the starting point from which the adapter must learn. Existing methods mainly select quantized weights to reconstruct the pretrained model’s weights or activations. However, reconstruction quality before fine-tuning does not necessarily identify the best starting point for adaptation. We therefore view initialization as a search for quantized weights that perform well after adaptation. However, this poses a major challenge: the adapted model is not yet available during the calculation of the quantized initialization. To address this challenge, we propose Task-Aware Quantized Initialization (TAQI), which uses target-task output gradients measured before fine-tuning to guide this search. TAQI weights layer-wise reconstruction errors by gradient energy and mixes the resulting task-sensitive curvature with general calibration curvature. We show that the task-weighted error bounds the energy of local first-order task-loss perturbations. Using IBM Granite-4.0-H-350M at 4 bits, TAQI improves the primary metric by 0.45–1.33 points over GPTQ-intrinsic across medical, financial, and scientific question-answering benchmarks. These results highlight the promise of adaptation-oriented quantized initialization for quantized low-rank fine-tuning.

Primary area:
optimization
Keywords:
Quantization, Quantized Low-Rank Adaptation, Parameter Efficient Fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6oYDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

With the backbone frozen, the quantized weights determine where adaptation starts. Prior methods reconstruct pretrained weights or activations; the authors argue post-adaptation target risk matters instead. TAQI weights input second moments by BF16 output-gradient energy, blends them with general calibration statistics, and reuses this geometry for the low-rank basis and GPTQ-intrinsic scan, adding an output-group gate and persistent mask. Two backbones, three bit widths, three QA tasks.

Critical strengths

1. Well posed. Treating the quantized point as the object of study is useful, and Mr(Q)M_r(Q) translating with QQ rather than nesting blocks the assumption that higher precision gives a better start. Eq. (11)'s Schur complement, tying a code's cost to the residual low-rank coordinates can absorb, is the strongest technical content.

2. Solid coverage. Five of six comparisons per bit width favor TAQI; appendices give full metrics, and Table 7 shows error reallocated, not merely reduced. MedQA deficits and the 2-bit macro-F1 of 10.855 are reported as measured.

Critical weaknesses

1. Task awareness is not isolated. TAQI uses 128 labeled in-domain examples and BF16 backward statistics that GPTQ-intrinsic, calibrated on general data, never sees. Each Table 1 gap therefore admits two readings: gradient weighting steered reconstruction toward task-sensitive directions, or calibration merely moved in-domain. Appendix E.6's XJ control separates these but appears only in Sec. 4.4's static analysis. The missing control is an in-domain, unweighted calibration baseline; without it, Table 1 supports in-domain initialization, not the mechanism the title promises.

2. The motivation does not establish that reconstruction fidelity misaligns with adaptation. Fig. 1 compares BF16 with NF4, a precision difference rather than a ranking of initializers at fixed width; page 2 concedes it does not measure reconstruction error, and Fig. 1(b)'s axes share the BF16+LoRA score, which C.1 calls mathematically coupled. Three of five reversals also involve base models whose outputs are almost entirely invalid, so those gaps mainly reflect output validity. This weakens the reversal evidence without touching the other wins, but it leaves the premise resting largely on Proposition 1's toy counterexample.

Questions for authors

Q1. When would Eq. (8)'s initialization-time sensitivity still predict Eq. (2)'s JKJ_K?

Q2. Could GuidedQuant and Fisher-weighted LQ-LoRA be run under TAQI's joint initialization and fine-tuning protocol?

Q3. Table 2's “Select.” bundles the output-group gate with the persistent mask; can their contributions be separated?

Q4. In Table 1, 2-bit GPTQ-intrinsic scores 29.081 on TAT-QA, below plain GPTQ+LoRA's 34.737; was it tuned, and what fraction of groups does the gate accept?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6oYD (Part 1/2): Mechanism and premise (W1, W2, Q1)

We thank the reviewer for the precise analysis and for identifying the controls that separate our mechanism from its confounders. Part 1 addresses W1/W2/Q1 and Part 2 addresses Q2–Q4.

Separating gradient weighting from in-domain calibration (W1). To address your concern that TAQI's gains might come from in-domain calibration alone, we ran the requested in-domain, unweighted control through the full pipeline. Both arms use the same 128 labeled in-domain examples, supervised positions, general statistic, basis, scan, fine-tuning and evaluation; the only difference is weighting each position by ∥g∥22\Vert g\Vert_2^2 or uniformly. Both arms are built for this control with the gate disabled (Granite-350M, 4 bits):

TaskMetricIn-domain unweightedGradient-weightedΔ
MedQAAccuracy32.28633.386+1.100
TAT-QAF145.22445.657+0.432
QASPERAnswer F122.89923.635+0.736

Gradient weighting improves all three tasks, by +1.100 / +0.432 / +0.736 points, which shows that the gain comes from the weighting mechanism rather than from moving calibration in-domain. This matches the static analysis in Section 4.4 (Table 7), where, with identical examples, basis and scanner, weighting reduces the task-weighted error DTD_T by 52.16–60.40% and the first-order task-loss perturbation Γ\Gamma by 32.87–70.56% relative to the unweighted control. The mechanism is also visible in the data: task-gradient energy is concentrated on the answer positions rather than spread over the context (Appendix B), so uniform weighting dilutes the task signal with context tokens. Together, these results confirm that the task-aware weighting named in the title is what drives TAQI's improvement.

Reconstruction fidelity versus adaptation (W2). We agree that Figure 1 compares precision settings rather than initializers at a fixed width, and that Figure 1(b) serves as a descriptive summary for selecting benchmarks with real adaptation headroom; we will state this scope explicitly. The premise, however, is supported by two further lines of evidence. First, as Appendix C.1 separates, not all reversals come from invalid base outputs: on AttackQA, quantization lowers unadapted token F1 by 12.87 points, yet 4-bit+LoRA exceeds BF16+LoRA by 0.47 points after adaptation; on PubMedQA, a 2.60-point loss becomes a 2.60-point gain, and both tasks have informative base predictions. Second, at a fixed 4-bit width on real Granite layers (Section 4.4, Table 7), the task-guided initialization has 7.71–9.20% higher general reconstruction error while lowering the first-order task-loss perturbation by 77.40–81.19%. This shows directly, at the same width, that lower reconstruction error and lower task sensitivity need not coincide, which is the misalignment our premise describes.

When Eq. (8) predicts JKJ_K (Q1). Eq. (8) bounds the first-order task-loss perturbation at the reference model, whereas JKJ_K is the risk after KK adaptation steps. Eq. (8) is informative about JKJ_K under four conditions, which we will state in Section 3.2:

  1. Limited repair capacity. Freezing QQ fixes the offset of the reachable family {Q+D:rank(D)≤r}\lbrace Q+D:\mathrm{rank}(D)\le r\rbrace. With bounded rank rr and steps KK, the adapter cannot both absorb large quantization errors in task-critical directions and learn the task shift; the two compete in the spectral tail beyond rank rr (Eq. 17).
  2. Persistence of task-sensitive directions. Directions with high gradient energy at the reference model remain relevant during adaptation, which is most plausible when the backbone is frozen and the updates are low-rank.
  3. Local validity. The quantization perturbation is small enough for first-order sensitivity to be informative; the general term (β = 0.5) and the general-error tolerance (τ = 0.05) keep candidates in this regime.
  4. Comparable adapter parameterization. The compared initializations use the same factor scaling and active output support, because rescaling A↦tAA\mapsto tA, B↦B/tB\mapsto B/t preserves the initial product but changes the finite-step trajectory (Appendix D.5).

These conditions describe the regime TAQI targets: low-bit backbones adapted with limited rank and fine-tuning budgets, which is the standard setting of quantized PEFT.

AuthorsOpenReview ↗

Response to Reviewer 6oYD (Part 2/2): Baselines, gate/mask, tuning and acceptance (Q2, Q3, Q4)

GuidedQuant and Task-Fisher LQ-LoRA under our protocol (Q2). Yes; both have been run. Task-Fisher LQ-LoRA (Guo et al., 2024) uses the official implementation, with Fisher computed on our exact 128 task calibration examples and fine-tuned under our protocol (rank 16, α = 32, effective batch 64, three epochs). Over three training seeds on Granite-350M at 4 bits (mean ± std over training seeds 42/43/44), TAQI achieves a higher mean on all three tasks, by 0.50 / 1.37 / 1.49 points on MedQA / TAT-QA / QASPER:

TaskMetricTask-Fisher LQ-LoRATAQI
MedQAAccuracy32.522 ± 0.49133.019 ± 1.192
TAT-QAF144.154 ± 0.57345.526 ± 0.109
QASPERAnswer F122.038 ± 0.48623.532 ± 0.700

Because both methods receive the same task data, this shows that TAQI's advantage comes from its construction rather than from the calibration set. GuidedQuant+LoRA (Kim et al., 2025) has also been run, at 4 bits on Qwen3.5-0.8B: it uses all 128 in-domain calibration examples, keeps the guided quantized weights frozen and trains a fresh adapter with rank 16, α = 32 and three epochs, and reaches 45.247 / 62.219 / 21.830 (MedQA accuracy / TAT-QA F1 / QASPER answer F1). GuidedQuant is a weight-only quantizer with its own calibration pipeline, and its quantized weights have to be prepared separately for every backbone and task. Within the rebuttal period we completed all three tasks on Qwen3.5-0.8B, so this is the setting we report, under GuidedQuant's own evaluation pipeline; the matched three-seed comparison above is the one run fully under our protocol.

Separating the gate and the persistent mask (Q3). The gate and the mask are two stages of one selection protocol: the gate decides which output groups keep the joint candidate, and the persistent mask keeps the rejected groups inactive during fine-tuning. The component ablation in the paper (Table 2) separates this protocol from the task-conditioned construction: TAQI-core, without the protocol, improves all three primary metrics over GPTQ-intrinsic, and the full protocol improves them further. To separate the two stages, we built three arms for this control (Granite-350M, 4 bits):

ArmGateMaskMedQA Acc.TAT-QA F1QASPER Ans. F1
No selectionoffoff32.52245.57622.864
Gate onlyonoff32.99345.15322.865
Gate + mask (TAQI)onon34.09345.40022.894

These arms are rebuilt for this control and are compared with each other. The last arm is evaluated with batched generation, the other two with single-example generation.

The two gate-on arms start from identical QQ, LL, RR and initial adapter tensors in all 72 modules and differ only in the mask, so the gate alone determines the initialization and the mask acts only during fine-tuning. Across the three arms, TAT-QA and QASPER change by at most 0.42 and 0.03 points, while MedQA accuracy rises from 32.522 to 32.993 with the gate and to 34.093 with the gate and mask together. The selection protocol therefore acts as a safeguard on top of the task-conditioned construction, with its effect concentrated on MedQA. Below 4 bits, TAQI's gains come mainly from the task-conditioned construction (Appendix A.2, Table 4).

GPTQ-intrinsic tuning (Q4). No method in Table 1 was tuned individually: all methods, including TAQI, used one shared, preset learning rate, so the comparison gives no method a tuning advantage. In the 2-bit regime this shared setting affects the baselines. In the same 2-bit TAT-QA setting, plain GPTQ+LoRA reaches 44.278 F1 at its best tested learning rate (5e-5), which remains below TAQI's 48.190 at the shared default. TAQI's 2-bit advantage therefore holds even against a baseline with a tuned learning rate.

Gate acceptance (Q4). The gate accepts nearly all output-channel groups (size 64, β = 0.5, τ = 0.05):

ModelBitsMedQATAT-QAQASPER
Qwen3.5-0.8B (6,240 groups)2100.00100.00100.00
Qwen3.5-0.8B399.7999.3899.38
Qwen3.5-0.8B499.6299.1399.18
Granite-350M (1,948 groups)499.9599.6999.79

The gate compares the candidate Q+LRQ+LR with the source Q0Q_0. At 2 bits, the source error is large and the joint candidate improves both task and general error in every group, so all groups are accepted and the 2-bit TAQI model coincides with its ungated construction. At 3–4 bits, Q0Q_0 is already accurate and the 5% general-error tolerance is tighter, so fewer than 1% of groups are rolled back. This shows that the gate acts as a sparse safeguard and that TAQI's 2-bit gains come from the task-conditioned construction itself.

Thank you for your detailed response and the additional experiments. I will maintain my original rating.

Reviewer_ZboaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that in QLoRA-style fine-tuning, the frozen quantized backbone and initial LoRA factors should be chosen for post-adaptation performance rather than reconstruction fidelity. The argument is supported by a 14-benchmark NF4-vs-BF16 study and a toy counterexample. The proposed method, TAQI, measures output-gradient energy once on the BF16 model using target-task data, uses it to weight the input second moment, and mixes the result with GPTQ’s general curvature. This mixed curvature then defines both the low-rank basis and the rounding metric in GPTQ-intrinsic’s joint scan. An output-group gate keeps the new codes only where task-weighted error improves, and masks the adapter elsewhere during SFT. On Granite-350M at 4 bits, TAQI improves over GPTQ-intrinsic by 0.45–1.33 points on MedQA, TAT-QA, and QASPER, and it wins 15 of 18 comparisons across two backbones and three bit widths.

Critical strengths
  • Useful framing. The paper treats quantization as an initialization problem for later adaptation, rather than as pure compression, which is a sensible perspective.
  • Clean integration. Task information enters both the low-rank basis and the rounding decisions within a single joint scan.
  • Honest and detailed reporting. Regressions are reported, results are explicitly labeled as single-run, and the implementation details are thorough.
  • Supportive diagnostic. The initialization-level analysis shows that error is reallocated toward task-sensitive directions as intended.
Critical weaknesses
  • The method does not match the framing. The paper motivates optimizing performance after adaptation, but TAQI still reconstructs the pretrained weights, only under a reweighted metric. Its gradients are measured once before training and carry no information about adaptation dynamics. The counterexample in Proposition 1 favors initializations closer to the task target, which TAQI never uses, so it does not support the method.
  • Limited novelty. Gradient- or Fisher-weighted quantization already exists (GuidedQuant, SqueezeLLM, data-aware LQ-LoRA), and the joint scan comes directly from GPTQ-intrinsic. What remains new is mainly the weighting choice, the trace-matched mixing, and the group gate.
  • The evidence is weak. All results are single runs on two sub-1B models. Most 4-bit gains are under one point, and the largest 4-bit effect is a regression (−3.0 on Qwen MedQA). Without seeds or confidence intervals, the “15/18 wins” count is not convincing.
  • The evidence is weak. All results are single runs on two sub-1B models. Most 4-bit gains are under one point, and the largest 4-bit effect is a regression (−3.0 on Qwen MedQA). Without seeds or confidence intervals, the “15/18 wins” count is not convincing.
  • Key baselines are missing. LoftQ, LQ-LoRA (Fisher), quantized PiSSA, ApiQ, GuidedQuant+LoRA, LoRA-GA, and CorDA are all absent. There is also no ablation that applies task curvature only to the basis or only to the scan.
Questions for authors

see above

3 comments
AuthorsOpenReview ↗

Response to Reviewer Zboa (Part 1/3): Framing and novelty (W1, W2)

We thank the reviewer for the precise critique of our framing, novelty and evidence. Part 1 addresses W1/W2, Part 2 addresses W3, and Part 3 addresses W4.

Framing and Proposition 1 (W1). To address your concern that the method does not match the framing, we explain why a one-time, task-weighted construction is the right tool for a post-adaptation objective. In quantized PEFT, the backbone QQ is frozen and receives no updates during fine-tuning; only the adapter factors evolve, and their gradients are recomputed at every step. Any task information that should shape QQ must therefore enter before training, and the task loss at the reference model is the information available at that point. Directly optimizing the post-adaptation objective JKJ_K (Eq. 2) would require fine-tuning every candidate backbone, which is not tractable. TAQI therefore combines a task-informed construction of the frozen backbone with fully dynamic adaptation of the adapter.

As for Proposition 1, it establishes the principle that TAQI implements. It shows that, at fixed bit width, rank, optimizer and step budget, an initialization with smaller reconstruction error to the pretrained weights can have higher post-adaptation risk. Reconstruction fidelity alone is therefore an insufficient criterion: errors in directions that govern the task loss matter more than errors elsewhere. Since the adapted target WT⋆W_T^\star is not available before training, TAQI keeps the pretrained weights as the reconstruction anchor and uses task-loss gradients to decide where reconstruction and correction capacity are spent; Eq. (8) bounds the first-order task-loss perturbation this allocation controls. Proposition 1 thus motivates the objective, and TAQI is a computable estimator of it, whose effect is tested by the downstream comparisons.

Novelty (W2). TAQI differs from each cited method as follows.

  • SqueezeLLM (Kim et al., 2024) and data-aware LQ-LoRA (Guo et al., 2024) use parameter-wise diagonal Fisher weights FijF_{ij}, which omit input cross-correlations. TAQI's scan propagates rounding error into unprocessed rows and into the continuous low-rank coordinates through one shared dense input-space form tr(E⊤HE)\mathrm{tr}(E^\top H E) per layer, which an element-wise FijF_{ij} cannot directly provide.
  • GuidedQuant (Kim et al., 2025) uses end-loss-guided input geometry for weight-only post-training quantization, without a low-rank branch that absorbs rounding error during the pass.
  • GPTQ-intrinsic (Zhang and Saab, 2026) couples the scan with the low-rank coordinates, but its geometry carries no task information. TAQI places a task-sensitive geometry, the trace-matched mixture H=(1−β)HG+βHˉTH=(1-\beta)H_G+\beta\bar H_T, in both the basis L=Vr(H)L=V_r(H) and the scan. This changes what the joint construction optimizes. For the paper's objective (Appendix E.3, Eq. 21), the exact minimum over RR at fixed QQ is min⁡RS(Q,R)=∥W−Q∥Heff2,Heff=H+λI−HL(L⊤HL+λIr)−1L⊤H.\min_R \mathcal{S}(Q,R)=\Vert W-Q\Vert_{H_{\mathrm{eff}}}^2,\qquad H_{\mathrm{eff}}=H+\lambda I-HL(L^\top HL+\lambda I_r)^{-1}L^\top H. Rounding is thus evaluated in a metric that discounts what the continuous coordinates can correct along the leading directions of HH. Once HH contains the task statistic, both what the adapter absorbs and what the rounding must protect are determined by task sensitivity rather than by general activation statistics.

Our new experiments show that this formulation, rather than the calibration data or the gate, carries the gain. Against the official Task-Fisher LQ-LoRA on identical task data, TAQI achieves a higher three-seed mean on all three tasks (Part 2). In an in-domain unweighted control, both arms use the same 128 labeled in-domain examples, supervised positions, general statistic, basis and scan, fine-tuning and evaluation; the only difference is weighting each position by ∥g∥22\Vert g\Vert_2^2 (TAQI) or uniformly. Both arms are built for this control with the gate disabled (Granite-350M, 4 bits).

TaskMetricIn-domain unweightedGradient-weightedΔ
MedQAAccuracy32.28633.386+1.100
TAT-QAF145.22445.657+0.432
QASPERAnswer F122.89923.635+0.736

Changing only the position weights from uniform to ∥g∥22\Vert g\Vert_2^2 improves all three tasks by +1.100 / +0.432 / +0.736 points. In the submitted ablation (Tables 2 and 5), TAQI-core with the gate disabled improves TAT-QA and QASPER at every bit width, by 2.154 and 3.102 points at 2 bits. Together, these results show that the task-conditioned geometry is the source of TAQI's advantage.

AuthorsOpenReview ↗

Response to Reviewer Zboa (Part 2/3): Evidence, seeds and scale (W3)

Evidence, seeds and scale (W3). To address your concern on single runs, the matched comparison against Task-Fisher LQ-LoRA now uses three training seeds on all three tasks. We ran the official LQ-LoRA with Fisher computed on our task data, under a matched protocol with identical calibration examples, test sets and scorer (Granite-350M, 4 bits, mean ± std over training seeds 42/43/44):

TaskMetricTask-Fisher LQ-LoRATAQI
MedQAAccuracy32.522 ± 0.49133.019 ± 1.192
TAT-QAF144.154 ± 0.57345.526 ± 0.109
QASPERAnswer F122.038 ± 0.48623.532 ± 0.700

TAQI achieves a higher mean on every task, by 0.50, 1.37 and 1.49 points, and on TAT-QA and QASPER its gains exceed the seed standard deviation of either method. This confirms that TAQI's gains are not artifacts of a single run.

As for scale and the size of the 4-bit gains, we trained and evaluated TAQI on Qwen3.5-4B at 2, 3 and 4 bits (primary metric of each task in the rightmost column of its group; best per bit width in bold):

BitMethodMedQA M-F1MedQA Acc.TAT-QA EMTAT-QA ScaleTAT-QA F1QASPER Evid. F1QASPER Ans. F1
4GPTQ-LoRA70.5970.7576.5096.2883.2910.2328.19
4QA-LoRA65.5665.7256.0089.7563.512.6624.03
4QBLoRA69.4569.5875.6096.2282.489.7328.45
4GPTQ-intrinsic67.0467.0676.9896.8283.8610.6626.82
4TAQI71.0771.2378.7896.5284.9811.2430.15
3GPTQ-LoRA62.3462.4276.0296.2882.6010.0827.28
3QA-LoRA57.4257.5551.3887.9559.063.1825.65
3QBLoRA63.1763.2972.4295.8679.479.2324.64
3GPTQ-intrinsic62.5562.6674.8296.1681.4010.0226.49
3TAQI64.5164.7076.8096.9483.2710.2529.86
2GPTQ-LoRA25.4026.1059.2994.9666.478.3819.56
2QA-LoRA24.9326.101.6238.612.927.761.04
2QBLoRA23.4325.0853.9693.8861.277.6715.40
2GPTQ-intrinsic10.7627.4462.6595.9869.028.9323.69
2TAQI26.6129.6463.6796.2270.968.1623.60

At 4 bits, TAQI achieves the best primary metric among all five methods on all three tasks and outperforms GPTQ-intrinsic by 4.17 MedQA accuracy, 1.12 TAT-QA F1 and 3.33 QASPER answer F1 points. At 4B, TAQI is the strongest method on MedQA at every bit width. This shows that TAQI's advantage persists at multi-billion scale.

We also note that uniform wins across every task and precision are not typical in this literature, even for widely adopted methods. Counting strict wins in the published tables, LoftQ (Li et al., 2024) outperforms QLoRA in 13 of 17 comparisons at 4 bits (Tables 3, 5 and 11), losing, for example, 35.0 versus 35.1 on LLaMA-2-7B GSM8K; Fisher-weighted LQ-LoRA (Guo et al., 2024) outperforms its unweighted variant in 46 of 60 comparisons (Tables 2 and 6), scoring, for example, 86.4 versus 87.1 at 2.75 bits on GLUE. TAQI outperforms GPTQ-intrinsic in 15 of 18 backbone–bit-width–task settings (83%), a rate comparable to or higher than these methods (76% and 77%).

AuthorsOpenReview ↗

Response to Reviewer Zboa (Part 3/3): Baselines and ablation (W4)

Baselines (W4). To address your concern on missing baselines, we evaluated the requested methods with rank 16, α = 32, three epochs and the same 128 in-domain calibration examples wherever calibration applies (Granite-350M, 4 bits, training seed 42):

MethodMedQA Acc.TAT-QA F1QASPER Ans. F1
Task-Fisher LQ-LoRA (Guo et al., 2024)32.67944.79121.494
LoftQ (Li et al., 2024)– ‡40.18819.930
Quantized PiSSA (Meng et al., 2024)33.54343.28622.715
CorDA (Yang et al., 2024)31.57943.99622.556
TAQI (paper, Table 1)34.09345.72123.217

‡ LoftQ uses its official PEFT implementation with the default settings (NF4, one alternating iteration) and the same fine-tuning protocol as all other methods. On MedQA, its training diverged at the start, so this run did not yield a valid model; we leave the cell empty rather than report a failed run, and it is not counted in favor of TAQI. We will report a completed MedQA run in the revision.

TAQI scores highest on all three tasks. LoftQ minimizes the unweighted reconstruction error of the pretrained weights, Quantized PiSSA and CorDA initialize the adapter from a fixed quantized backbone, and Task-Fisher LQ-LoRA weights a decomposition by a diagonal Fisher; none of them conditions the quantized backbone and the adapter jointly on task geometry. These results show that TAQI's joint construction yields a better starting point than reconstruction-driven, adapter-only and Fisher-weighted initializations. We also ran GuidedQuant+LoRA (Kim et al., 2025) at 4 bits on Qwen3.5-0.8B, where it reaches 45.247 / 62.219 / 21.830 (MedQA accuracy / TAT-QA F1 / QASPER answer F1). GuidedQuant is a weight-only quantizer with its own calibration and evaluation pipeline, and its quantized weights have to be prepared separately for every backbone and task.

Basis/scan ablation (W4). We applied the mixed geometry MM to neither site, the scan only, the basis only, or both, with the gate disabled (Granite-350M, 4 bits):

BasisScanMedQA Acc.TAT-QA F1QASPER Ans. F1
HGH_GHGH_G †33.38644.39522.772
HGH_GMM33.38644.78922.642
MMHGH_G31.42244.65222.819
MMMM †33.62145.18822.826

† From the paper (Table 2): the general geometry at both sites is GPTQ-intrinsic, and the task-conditioned geometry at both sites is TAQI-core. The two single-site arms are new runs. Full TAQI, which adds the selection protocol to TAQI-core, reaches 34.093 / 45.721 / 23.217 (baseline table above).

Applying the task-conditioned geometry to both the basis and the scan gives the highest score on all three tasks, with the largest margins on TAT-QA: 0.40–0.54 F1 points over either single-site arm and 0.79 over the purely general arm. Each single-site placement already improves TAT-QA over the purely general arm. This supports the design choice in TAQI: the basis and the scan are most effective when they share the same task-aware geometry.

Reviewer_eYUcOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

For QLoRA-style adaptation, the paper argues the quantized backbone should be chosen as a good starting point for adaptation, not to faithfully reproduce the original model. TAQI steers quantization choices by how much the target task's loss depends on each output direction, blended with a generic calibration signal, keeping only changes that help the task without hurting general error. On a few QA tasks at low bits and sub-1B scale, it mostly beats the GPTQ-intrinsic baseline, with the largest gains at 2 bits.

Critical strengths
  1. Correct, well-motivated problem shift. "Reconstruction quality ≠ post-adaptation quality" is a genuine conceptual observation for frozen-backbone adaptation, precisely formulated, with an honest counterexample.
  2. Honest/Transparent reporting. Single runs are flagged as "descriptive," MedQA regressions are admitted, the surrogate is admitted not to be a Taylor approximation, and unflattering diagnostics are reported.
  3. The 2-bit result is the interesting finding: when quantization capacity is scarce, allocating it by task sensitivity appears to matter.
Critical weaknesses

W1 — One quantized backbone per task breaks the deployment model of the entire method family. Every published method here -- QLoRA, LQ-LoRA, LoftQ, GPTQ-intrinsic -- constructs "one" task-agnostic artifact on generic data and reuses it for all tasks; the per-task cost is the adapter (approx. 0.5–1% of model size). TAQI's construction is task-conditioned, so it requires "one quantized backbone per task". The math: at 7B/4-bit, approx. 3.6 GB per task vs. approx. 80 MB for an adapter ("approx. 45x" the marginal per-task cost); at 70B, approx. 35 GB vs. approx. 0.4 GB (approx. 86x). Serving: one 80 GB GPU at 70B/4-bit holds one backbone plus KV cache -- a second concurrent task leaves no room, so capacity drops from tens of tasks per GPU (adapter multiplexing) to 1-2. The paper never reports or confronts this cost.

W2 — The core idea is one calibration-set swap away from published LQ-LoRA, and that baseline is never run. LQ-LoRA (ICLR 2024) already performs Fisher-weighted joint quantization + low-rank reconstruction — and the Fisher diagonal of the cross-entropy loss is mathematically the same quantity as TAQI's squared output-gradient energy. LQ-LoRA simply computes it on generic text (C4) and reuses the result for all tasks. Swapping the calibration set to task data reproduces TAQI's task-sensitive curvature — a trivial change that is never evaluated. What remains uniquely TAQI: the 50/50 general/task data mix (β=0.5, never swept), the shared-geometry scan (adopted from the GPTQ-intrinsic baseline), and the acceptance gate (τ=0.05, never swept). The dilemma: either the swapped baseline matches TAQI — then the claimed niche was never a contribution — or TAQI wins — then the surviving novelty is two unswept knobs on top of two existing methods. Note this is the same design choice as W1: task-conditioned construction is both the claimed novelty and the source of the deployment cost.

W3 — Single runs at sub-1B scale. Every result is a single run, and the 4-bit deltas the abstract headlines are within typical LoRA seed-to-seed noise. All backbones are sub-1B — far below the 7B–70B regime of the methods the paper positions against. (Fairness: the GPTQ-intrinsic baseline is small too, but the abstract's generality claim needs at least one ≥7B run.)

Questions for authors

Major:

  1. Run LQ-LoRA with the Fisher computed on task data (by your own Eq. 5 identity this equals your H_T) under your pipeline. If it matches TAQI, state exactly which component you claim as the contribution — the β mix, the shared-geometry scan, or the acceptance mask — and provide the ablation showing that component alone carries the gain.2. Address the per-task deployment footprint of one-backbone-per-task construction (W1: ~45–86× the marginal per-task cost of shared-backbone QLoRA/LQ-LoRA; 1–2 concurrent tasks per 80 GB GPU at 70B). For example: report cross-task transfer of a single TAQI-constructed backbone (e.g., the MedQA backbone evaluated with TAT-QA and QASPER adapters, and permutations) to test whether one task-aware artifact can amortize the cost — or justify the footprint for your intended use case.

Minor:

  1. Report mean±std over ≥3 seeds for the 4-bit primary comparisons, and evaluate at least one 3–8B backbone.
  2. The 2-bit headline gains coexist with near-degenerate scores (macro-F1 ≈ 11 on MedQA): report answer-type distributions and invalid rates to rule out format artifacts, and report WikiText/C4 perplexity for TAQI vs. GPTQ-intrinsic backbones to quantify off-task cost.
  3. Sweep β and τ and report sensitivity.
7 comments
AuthorsOpenReview ↗

Response to Reviewer eYUc (1/4): Deployment Cost and Backbone Reuse (W1, Q2)

We thank the reviewer for the detailed and quantitative critique. Part 1 addresses W1/Q2, Part 2 addresses W2/Q1/Q5, Part 3 addresses W3/Q3, and Part 4 addresses Q4.

To address your concern on the per-task deployment footprint, we first clarify the setting that task-aware adaptation targets. Task-agnostic construction is designed for the case where the downstream tasks are unknown or numerous: the backbone is quantized once, and each task is served by its own adapter. For cloud deployment, where one server multiplexes many unrelated tasks on a shared backbone, this is the more economical design, and the 45–86× and 70B/80 GB estimates describe that setting. TAQI addresses the complementary case, in which the target task is known before quantization: a resource-constrained device serves one known professional workflow with a single compact model under tight bit-width and rank budgets. Here the backbone serves only that task, so using the task's information to construct it gives up no sharing and directly improves the deployed model. Such deployments are practical: Google's HAI-DEF guidance and MedGemma documentation (Google, 2025) list offline devices and intermittently connected mobile clinics among the scenarios for self-hosting and customizing open medical models, and Medicine on the Edge (Nissen et al., 2025) evaluates 4-bit clinical LLMs running locally on mobile devices. Task-specific quantized artifacts are also established in this method family: QA-LoRA (Xu et al., 2024) can merge its adapter into the quantized weights to export a standalone model per task, L4Q (Jeon et al., 2025), PEQA (Kim et al., 2023) and Task-Circuit Quantization (Xiao et al., 2025) produce task-specific quantized models, and LoftQ (Li et al., 2024, Sec. 6) notes that calibrating on training data yields "task-specific quantized models." In this setting, the relevant cost is the size and accuracy of one task-specialized model, which we quantify below.

Storage cost of an additional task. The 45–86× estimate assumes that each task stores a full copy of the quantized backbone. TAQI modifies only the targeted weights, so unmodified components are shared across tasks and an additional task stores only its modified integer weights, mask and adapter. We measured this on the exported Granite-350M artifacts (on-disk size, decimal MB):

BitsShared runtimeTask Q/maskTask adapterTask increment
2230.56228.14912.91241.060
3263.94042.13812.91255.050
4297.31756.12812.91269.040

At 4 bits, duplicating the full quantized model for each task would cost 353.4 MB, or 27× the adapter, whereas TAQI's measured increment is 69.0 MB, or 5.3× the adapter. This shows that TAQI's marginal cost is a task-specific payload rather than a duplicated backbone.

Cross-task reuse. Following your suggestion, we reused a TAQI backbone constructed on one task for a different target task, training only a new adapter (Qwen3.5-0.8B, 2 bits):

Initialization sourceTAT-QA F1
TAQI initialized on MedQA45.150
TAQI initialized on QASPER44.586
TAQI initialized on TAT-QA (task-specific)48.190†

† Task-specific TAQI from the paper (Table 1). Other rows reuse the source-task backbone and initial factors and train a new adapter on TAT-QA.

The MedQA- and QASPER-built backbones retain 93.7% and 92.5% of task-specific TAQI. This shows that one task-aware artifact can amortize the deployment cost across tasks, while task-specific construction remains the best choice when accuracy on a single workflow matters most.

Shared initialization without task data. We also built one Granite-350M initializer from 128 WikiText windows, without any downstream data, and reused the same frozen backbone for all three tasks (4 bits):

MethodMedQA Acc.TAT-QA F1QASPER Ans. F1
TAQI, task-specific (paper)34.09345.72123.217
TAQI, shared WikiText32.12944.72522.568

With a single artifact serving all three tasks, the shared initializer retains 94–98% of task-specific TAQI on every task. TAQI therefore offers two deployment modes: a task-specialized backbone when accuracy on one workflow matters most, and one shared backbone when cost must be amortized across many tasks. We will add the storage measurements and this scope statement to the paper.

AuthorsOpenReview ↗

Response to Reviewer eYUc (2/4): Relation to LQ-LoRA, Contribution, and β/τ Sensitivity (W2, Q1, Q5)

To address the dilemma you raise, we ran the calibration-swapped baseline and swept both hyperparameters. The swapped LQ-LoRA does not match TAQI, and the gain is carried by TAQI's task-conditioned geometry rather than by β, τ or the acceptance gate.

The calibration swap does not reproduce HTH_T. Eq. (5) defines only a scalar output-gradient energy, ωnp=∥gnp∥22\omega_{np}=\Vert g_{np}\Vert_2^2; HTH_T (Eq. 6) uses it to weight input outer products. With the input-first convention E∈Rdin×doutE\in\mathbb{R}^{d_{\mathrm{in}}\times d_{\mathrm{out}}}, LQ-LoRA penalizes DLQ(E)=∑i,jFijEij2,Fij=1N∑n(∂ℓn∂Wij)2,D_{\mathrm{LQ}}(E)=\sum_{i,j}F_{ij}E_{ij}^2,\qquad F_{ij}=\frac{1}{N}\sum_n\Big(\frac{\partial\ell_n}{\partial W_{ij}}\Big)^2, whereas TAQI uses DT(E)=tr(E⊤HTE),HT=1Zs∑n,p∥gnp∥22 xnpxnp⊤.D_T(E)=\mathrm{tr}(E^\top H_T E),\qquad H_T=\frac{1}{Z_s}\sum_{n,p}\Vert g_{np}\Vert_2^2\thinspace x_{np}x_{np}^\top. HTH_T retains the cross-input terms xixkx_ix_k that determine whether rounding errors cancel or amplify at the layer output, whereas the diagonal Fisher discards them. The two quantities are also used differently: LQ-LoRA alternates a Fisher-weighted SVD with NF quantization, while TAQI places the geometry inside both the low-rank basis L=Vr(H)L=V_r(H) and the coupled error-feedback scan. Swapping LQ-LoRA's calibration data therefore changes its weights, but not its quadratic form or its construction.

The swapped baseline does not match TAQI. We ran the official LQ-LoRA with Fisher computed on our task data, under a matched protocol with identical calibration examples, test sets and scorer (Granite-350M, 4 bits, mean ± std over training seeds 42/43/44):

TaskMetricTask-Fisher LQ-LoRATAQI
MedQAAccuracy32.522 ± 0.49133.019 ± 1.192
TAT-QAF144.154 ± 0.57345.526 ± 0.109
QASPERAnswer F122.038 ± 0.48623.532 ± 0.700

TAQI achieves a higher mean on all three tasks, by 0.50, 1.37 and 1.49 points. Because both methods receive the same task data, this shows that TAQI's advantage comes from its construction rather than from the calibration set.

The gain comes from TAQI-core. Our contribution is TAQI-core: task-sensitive geometry HTH_T that governs both the low-rank basis and the coupled scan. Two ablations isolate its effect. In the submitted ablation (Tables 2 and 5), TAQI-core alone improves over GPTQ-intrinsic on TAT-QA and QASPER at every bit width, and the gain grows as precision decreases, reaching 2.15 TAT-QA F1 and 3.10 QASPER answer F1 points at 2 bits. A new control that changes only the position weights from uniform to ∥g∥22\Vert g\Vert_2^2 improves all three tasks by +1.100 / +0.432 / +0.736 points. Because the gate is disabled in both experiments, these results show that the gain comes from the task-conditioned geometry itself.

β and τ are robust settings, not the source of the gain. β is not a 50/50 data mix: it is the weight of the task statistic after trace matching, with β = 0.5 giving equal contributions. τ is the allowed relative increase in general reconstruction error, with τ = 0.05 permitting at most 5% while task error improves. We varied each around its default (Granite-350M, 4 bits, MedQA accuracy):

β (τ = 0.05)00.250.50.75
Accuracy32.3632.6833.6232.13
τ (β = 0.5)00.020.050.1
Accuracy32.6833.5433.6232.84

All seven checkpoints of the sweep are evaluated under one common setting and are compared within the sweep. The default (β, τ) = (0.5, 0.05), fixed in advance, gives the highest accuracy in both sweeps. On TAT-QA and QASPER, the default τ lies within 0.32 and 0.26 points of the best tested τ. Removing the task statistic from the construction geometry (β = 0) lowers MedQA accuracy by 1.26 points, consistent with the ablations above. This shows that TAQI's defaults transfer across tasks without per-task tuning. As shown in Part 1, task-conditioned construction is an option rather than a requirement: one TAQI initializer built from general WikiText data can be shared across all three tasks.

AuthorsOpenReview ↗

Response to Reviewer eYUc (Part 3/4): Scale and seeds (W3, Q3)

Scale (W3, Q3). To address your concern on scale, we trained and evaluated TAQI on Qwen3.5-4B, within the requested 3–8B range, at 2, 3 and 4 bits on all three tasks (primary metric of each task in the rightmost column of its group; best per bit width in bold):

BitMethodMedQA M-F1MedQA Acc.TAT-QA EMTAT-QA ScaleTAT-QA F1QASPER Evid. F1QASPER Ans. F1
4GPTQ-LoRA70.5970.7576.5096.2883.2910.2328.19
4QA-LoRA65.5665.7256.0089.7563.512.6624.03
4QBLoRA69.4569.5875.6096.2282.489.7328.45
4GPTQ-intrinsic67.0467.0676.9896.8283.8610.6626.82
4TAQI71.0771.2378.7896.5284.9811.2430.15
3GPTQ-LoRA62.3462.4276.0296.2882.6010.0827.28
3QA-LoRA57.4257.5551.3887.9559.063.1825.65
3QBLoRA63.1763.2972.4295.8679.479.2324.64
3GPTQ-intrinsic62.5562.6674.8296.1681.4010.0226.49
3TAQI64.5164.7076.8096.9483.2710.2529.86
2GPTQ-LoRA25.4026.1059.2994.9666.478.3819.56
2QA-LoRA24.9326.101.6238.612.927.761.04
2QBLoRA23.4325.0853.9693.8861.277.6715.40
2GPTQ-intrinsic10.7627.4462.6595.9869.028.9323.69
2TAQI26.6129.6463.6796.2270.968.1623.60

TAQI achieves the best primary metric in 8 of 9 bit-width–task settings and all three at 4 bits, outperforming GPTQ-intrinsic by 4.17 MedQA accuracy, 1.12 TAT-QA F1 and 3.33 QASPER answer F1 points. This shows that the benefit of task-conditioned construction persists at multi-billion scale. A full 7B evaluation, covering curvature collection, joint construction and fine-tuning for every task and bit width, exceeds the rebuttal period; we are running it and will add the results to the revision.

Seeds (W3, Q3). As for the concern that the 4-bit gains lie within seed-to-seed noise, the matched comparison in Part 2 uses three training seeds on all three tasks. TAQI's mean is higher on every task, and on TAT-QA and QASPER its gains (1.37 and 1.49 points) exceed the seed standard deviation of either method (at most 0.70 points). This shows that the gains are not artifacts of a single run. We will report all seeds in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.