- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates an important yet largely overlooked question in post-training quantization: while existing quantization methods typically optimize local reconstruction error or preserve downstream task accuracy and perplexity, do they actually preserve the underlying causal computation mechanisms driving model decisions? Focusing on symbol-binding circuits in formatted multiple-choice question answering, the authors propose Causal-Equivariant Quantization, a structured mixed-precision framework comprising three stages.
Critical strengths
1.Bridging mechanistic interpretability and model compressionto explicitly distinguish between output preservation and mechanism preservation is highly original and well-motivated. It introduces a valuable conceptual angle for evaluating low-bit compression.
2.The proposed Format-Equivariance Error and Causal-Effect Distortion are mathematically clean and effectively capture both behavioral vulnerability under semantics-preserving format shifts and internal circuit degradation. Section 5.6 convincingly demonstrates strong empirical correlations between these metrics and worst-format accuracy.
3.In the controlled stress tests (Table 2 and Section 5.5), the authors transparently report that while CEQ allocation outperforms simple salience heuristics, it is statistically indistinguishable from strong task-gradient (Gradient-2/8) and Fisher (Fisher-2/8) criteria (with Gradient-2/8 achieving a slightly higher mean worst-format accuracy of 83.88% vs. 83.40%). This candid boundary on their empirical claims is commendable.
Critical weaknesses
-
In the controlled experiments, CEQ's causal-regret allocation is statistically indistinguishable from standard task-gradient (Gradient-2/8, ) and Fisher (Fisher-2/8, ) criteria, with Gradient-2/8 achieving a slightly higher mean worst-format accuracy (83.88% vs. 83.40%). Since small activation interventions are closely approximated by first-order Taylor expansions (gradients/Fisher), this raises a critical attribution question for the pretrained LLM results (Tables 3 and 4). An ablation comparing Gradient/Fisher allocation + CEQ-CAL on pretrained LLMs is missing.
-
Computing causal signatures and requires running intervention forward passes for every candidate structured group across 16 (or 25) counterfactual prompt variants. For large models like Qwen2.5-32B, this exhaustive probing appears computationally prohibitive, yet wall-clock overheads and the exact granularity of group in pretrained LLMs are omitted from the main text. Furthermore, the framework relies heavily on explicit option-permutation symmetries () specific to formatted MCQA. It remains unclear how Format-Equivariance Error (FEE) and centered intervention signatures can be generalized to open-ended generation or reasoning tasks where such discrete output symmetries do not exist.
-
The manuscript contains several severe data and formatting errors that hinder evaluation: (a) In Table 4 (MMLU row for CEQ), the standard accuracy (Acc.) is reported as 55.26% while the minimum accuracy across formats (Worst) is reported as 63.40%—by definition, Worst cannot exceed Acc. (and Table 3 reports 63.40% as the standard MMLU accuracy); (b) Table 2 and Line 353 contain highly suspicious leading-digit typos (3.482 for Fisher-2/8 CED instead of 0.482, 3.605 for RTN-2 CED instead of 0.605, and 3.0629 for Magnitude-2/8 FEE instead of 0.0629); (c) Table 5 is severely corrupted in layout with garbled headers ("FITT") and missing method names in the first column; and (d) the mathematical definition of in Eq. 4 and the construction of the controlled dataset are omitted from the main text, alongside broken cross-references (e.g., Line 298 citing Table 2 for dataset splits, and Line 313 citing Eq. 1 for the RTN operator).
Questions for authors
1.To isolate the contribution of counterfactual causal probing on large models (e.g., Qwen2.5-7B in Tables 3 and 4), how does a baseline combining standard task-gradient/Fisher precision allocation (or TaCQ's allocation) with your CEQ-CAL calibration perform compared to full CEQ?
2.Could you explain why tuning a single scalar parameter yields such a dramatic improvement, and clarify the practical benefit of the causal/FEE calibration terms given that KL-only achieves slightly higher worst-format accuracy?
3.Could you please verify and correct the swapped Acc. (55.26%) and Worst (63.40%) numbers for MMLU in Table 4, the suspected typos (3.482, 3.605, 3.0629) in Table 2 and Line 353, and the corrupted layout of Table 5? Additionally, please provide the explicit mathematical definition and dimensionality of the intervention signature in Eq. 4.
Additional Allocation Experiments, Preparation Costs, Calibration Benefits, and Clarifications of Results and Definitions
W1/Q1: Allocation benefit
Controlled Base-3L, Deep-4L, and Wide-3L results show competitiveness with Gradient/Fisher. Tables 3-4 compare methods that use different quantizers, so CEQ’s better results may come from both the quantizer and the causal-regret score.
We added a Qwen2.5-7B-Instruct/MMLU comparison fixing checkpoint, candidate groups, symmetric row-wise 2/8-bit quantizer, 2.12-bit weight budget, calibration examples, and evaluation protocol. All use the same CEQ-Cal objective and clipping search, selecting clipping factors independently with frozen weights.
CEQ exceeds Gradient/Fisher by 1.60/1.20 percentage points before calibration and 1.60/1.30 afterward. Calibration adds 3.80/3.70/3.80 points for Gradient/Fisher/CEQ. CEQ performs better than Gradient/Fisher with the same CEQ-Cal in this comparison.
Manuscript Changes: Sections 5.4/5.7 distinguish allocation and calibration gains; Section 4.1 explains the gradient comparison.
W2: Cost and scope
Each candidate is one query head. We evaluate 784 query heads in Qwen2.5-7B, 2,560 in Qwen2.5-32B, and 1,024 in Llama-3-8B. Shared GQA K/V weights are protected jointly and counted once. Probing uses 128 calibration examples with 16 four-choice variants, or 25 five-choice variants. Preparation time and peak memory usage on an NVIDIA A100 80GB GPU:
Probing is offline and is not repeated during inference.
FEE requires an output correspondence, supplied exactly by answer permutations in MCQA. For reasoning, mapping final answers to a common representation could measure answer consistency, not preservation of reasoning. Centering $dg(t)−∣T∣−1∑t′dg(t′) $requires comparable conditions, not discrete permutations. Meaning-preserving prompts and shared reference-continuation log-likelihood offer a possible formulation for generation.
Changes: Section 5.4 adds group counts and costs. Discussion clarifies alignment, centering, and generalization.
Q2: Why can one clipping scalar help?
The scalar changes clipping thresholds and quantization scales across many weight rows. At low precision, it balances clipping outliers against resolving smaller weights. It can therefore change many predictions without training base-model weights.
In Supplementary Section 6.3, Table 7, KL-only achieves 95.44% worst-format accuracy versus 95.16% for CEQ-Cal. CEQ-Cal changes FEE from .0159 to .0156, CED from .369 to .365, and correlation from .884 to .895. The paired seed-wise results show that four seeds select identical clipping factors, while one accounts for the diagnostic improvement and accuracy decrease. Under the evaluation protocol, CED/correlation here are calibration-set diagnostics, whereas accuracy/FEE use held-out examples. Most accuracy recovery comes from clipping calibration. Additional terms offer modest, seed-dependent diagnostic benefits, without establishing necessity.
Manuscript Changes: Section 5.7 and Supplementary Section 6.3 explain clipping, paired results, and diagnostic scope.
W3/Q3: Corrections and definitions
We verified that the manuscript already contains the correct values: MMLU Acc. 63.40%, Worst 55.26%, Fisher CED �2�, RTN-2 CED �3�, and Magnitude FEE �4�. Table 5 correctly displays all method names and evaluation headers. These entries required no numerical corrections.
For condition (c), the signature entry is $
dg,c(M)=∣Dc∣−1x∈Dc∑[mM(x)−mM∖g(x)],$ where �5� is the correct-answer logit minus the largest incorrect-answer logit. Thus �6�. Controlled probing combines four alphabet averages and four position averages into eight entries. The alphabet term subtracts the mean of the four alphabet effects from each effect.
Each controlled specification samples a target position, four distinct content identifiers, and a label permutation. The target is the symbol at that position. Base-3L uses 60 calibration specifications, four additional position-controlled subsets of 15, and 500 held-out specifications across four alphabets.
Manuscript Changes: Section 4.1 defines the signature and experimental setup explains dataset construction. References identify Supplementary Section 2/Table 2 and Section 4.3/Eq. (1). Results remain unchanged.