- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RoleQ, which places quantization scales according to reader/writer roles rather than using a uniform axis. The conversion pipeline combines this rule with learned rotations, ternary GPTQ, and scale distillation. An optional second ternary plane is assigned to units selected by Fisher-weighted quantization error.
Critical strengths
The scale-placement rule is simple, and the conversion results are promising. In Table 1, on Qwen3-8B-Base before distillation, RoleQ-T with secondary planes achieves 56.04 MMLU at 2.157 bpw, compared with 34.82 at 2.281 bpw for QTEA under the paper's accounting.
Critical weaknesses
-
The novelty is limited. As acknowledged in Section 2, the rotation setup follows QuaRot and SpinQuant, the l4 objective comes from OptRot, and the Fisher allocator follows FGMP. The main additions are therefore the role-based scale rule and functional-unit allocation. The new rotation objective also shows no clear advantage over the existing alternative: in Table 13(b), s2(opt) and s2(l4) achieve nearly identical perplexity after distillation (20.26 vs. 20.27), while l4 performs better before distillation.
-
The contribution of role-based placement to the final results remains unclear. Table 2 isolates placement under random rotation without distillation, whereas the main results use learned rotations and distillation. Table 9 lacks a matched per-OC control at approximately 1.59 bpw under the full pipeline. With grouped scales, per-OC and RoleQ are already nearly tied after distillation (51.33 vs. 50.89 MMLU). Moreover, Table 8 only compares RoleQ with AdaDim in local reconstruction error. The claimed halving of solver work is relative to a two-way GPTQ search, not AdaDim's RTN-based selection followed by one GPTQ pass. The practical advantage over AdaDim is therefore not established.
Questions for authors
-
Could the authors compare RoleQ with per-OC placement on Qwen3-8B-Base at approximately 1.59 bpw, using identical learned rotations and the same distillation budget?
-
How does RoleQ compare with AdaDim's original axis-selection procedure in final accuracy and measured conversion time?
-
With the Fisher score and total bit budget fixed, does allocating secondary planes to functional units outperform allocation to ordinary fixed blocks?
-
Why choose s2(opt) when s2(l4) matches it after distillation and performs better before distillation?
2 comments
Response to Authors
Thank you for the new experiments and the revised claims. However, they confirm my main concern. Under the full pipeline at 8B, RoleQ and per-OC tie after SD (per-OC is even ahead on MMLU), and RoleQ ties AdaDim end to end, where the axis search costs only 1–4% of GPTQ time. The gains in Table 1 therefore appear to come mainly from rotation, SD and secondary planes rather than from the placement rule. I could not find the table supporting the 14B post-SD claim. Q3 (units vs. fixed blocks at a fixed budget) remains unanswered. I keep my score.
W1 (novelty; s2(opt) vs s2(l4)). Our contributions are the role-based placement rule in a rotated basis, an exact per-vector ternary step within GPTQ, and secondary-plane allocation using recall-conditioned Fisher saliency. The reader refit is adopted from Z-FOLD. The first two are linked: because the rule puts a writer's scale on the axis GPTQ sweeps, each unit gets its own grid, fitted jointly with its codes on the compensated vector, which a per-OC scale does not allow. In Table 13's solver control that fit is worth 1.25× on top of the axis's own 1.23× (1.7B, PTQ). Section 3 identifies the rotation slots and objectives as existing components. Opt directly minimises ternary reconstruction error; l4 is a kurtosis surrogate. We use l4 in s1 because it gives better model quality. In s2, opt and l4 give nearly identical perplexity after SD (20.26 vs 20.27 at 1.7B), and the PTQ difference (70.86 vs 73.66) is within seed variation. Both are data-free at the same cost, so we retain the direct objective where model quality is comparable.
W2 / Q1 (per-OC at 1.59 bpw under the full pipeline). Common response Section 2. The per-OC control under the full recipe with SD at 8B, two seeds, reads 28.53 / 50.33 / 33.40 → 11.48 / 59.94 / 48.91 against the role axis's 22.42 / 50.39 / 33.32 → 11.33 / 60.33 / 47.59. The g128 pair you cite (51.33 vs 50.89) is consistent with a small post-SD effect at 8B: the task differences are small relative to seed variation and perplexity differs by 1-2%. At PTQ, per-OC perplexity is 27% higher; at 14B the advantage persists after SD on all three measures. We revised Section 4.2 to reflect this distinction. The rule also requires no axis search or stored flags, and we saw no measurable runtime overhead in the 8B comparison. Planes naturally refine each unit's fitted grid. Per-OC writer scales span units and do not support that extension; this control therefore concerns one-plane models.
Q2 (AdaDim end to end, with conversion time). We ran AdaDim as published at 1.7B (two seeds, r1 and recipe bases) and 8B (recipe basis). Rotations reduce the outlier effects that make low-bit quantization difficult. At 8B it puts 89% of the matrices on the role axis and reads 23.16 → 11.32 WikiText2, 61.0 zero-shot and 49.3 MMLU after SD against RoleQ-T's 22.20 → 11.27, 60.1 and 44.9 (seed 45), inside the seed ranges on every instrument. The selection takes 1.6 s against 102-124 s of GPTQ kernels per model at 1.7B (1-2%) and 12 s against 325 s at 8B (3.7%), so the conversion-time claim is replaced by this measurement. In the recipe basis at 1.7B the search picks the role axis for 91-92% of the matrices and reads 72.8 / 86.5 PTQ, 20.3 / 20.9 after SD against RoleQ-T's 73.7 / 72.6 and 20.3 / 20.7 (WikiText2); in r1, 94-95% agreement and a tie. Without a rotation it picks the opposite axis on 91% of the matrices and is 2.3× better at PTQ (554 vs 1,246).
Q3 (units vs fixed blocks). With blocks the size of a unit vector, reader blocks coincide with rows. For writers (W_O, W_out), units are columns, whereas row-wise blocks span units. Unit-aligned planes naturally refine individually fitted vector grids. Two measurements support this choice. First, the one-plane solver control separates the axis effect (1.23×) from the exact per-unit fit (a further 1.25×; Table 13, 1.7B PTQ). These are not measurements of the secondary-plane block variant. Second, Fisher scores are more concentrated and stable across units than across equal-sized blocks at 1.7B: the top 30% of W_O units hold 80% of the score, versus 63% for blocks; within W_out layers the medians are 52% and 44%. Split-half rank correlation is 0.97-0.98 for units and 0.89-0.92 for blocks.
Q4 (why s2(opt)). Answered under W1.