- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies reasoning failures caused by low-bit quantization and proposes Rescue Vector, a fixed activation correction constructed from paired BF16-correct and FP4-incorrect responses. The analysis identifies consistent representation differences, and the intervention improves accuracy across several models and benchmarks without retraining or modifying quantized weights.
Critical strengths
-
Interesting and insightful analysis. The observation that quantized responses initially follow comparable reasoning before diverging at intermediate steps is useful, the split-half directional consistency result provides an interesting representation-level perspective on these failures, all these obervations are insightful for us.
-
Simple intervention. The method requires no parameter training and adds only a small stored vector at deployment, this makes it practically appealing as an adaptation of an existing quantized model.
-
Clear experimental separation. Vector construction, development-set layer selection, and final evaluation use separate data, the reported aggregate results are internally consistent, and the ablations examine several important design choices.
Critical weaknesses
-
The comparisons do not establish the benefit under matched adaptation conditions. The author uses labeled target-task data, bf16 forward passes, and task-specific development-set layer selection. The quantization baselines do not receive an equivalent adaptation procedure. I think their comparison is informative about final performance, but does not isolate the value of the proposed correction. Moreover, the gains over the strongest quantization baseline on the three Qwen models are only 0.07–0.30 percentage points in benchmark-average accuracy, without uncertainty estimates.
-
Most benchmarks use multiple-choice or yes/no answers, so evidence for broader open-ended reasoning remains limited. The recovery analysis also does not report how often the intervention turns an originally correct FP4 answer into an incorrect one. Aggregate improvements alone do not reveal this trade-off. The gain on gsm8k is very small.
-
Eq 1 compares representations from independently generated correct and incorrect responses, I think the difference may therefore reflect both quantization effects and differences in response content or correctness. Improved accuracy and directional consistency do not distinguish these explanations. Random and reverse directions also improve Qwen2.5-7B, so more targeted controls would help establish what the learned direction captures.
Questions for authors
-
Can Rescur vector be compared with a lightweight activation-steering baseline using the same labeled examples, BF16 access, and development-set selection budget?
-
Could the authors include control vectors from pairs where both models answer correctly, and differences measured under identical response contexts? These would help separate quantization effects from content and correctness differences.
-
How well does a vector constructed on one task transfer to another without reconstructing it or retuning the injection layer?
-
Could the authors report both incorrect-to-correct and correct-to-incorrect transitions at the default configuration, together with uncertainty estimates for the small advantages over quantization baselines?
7 comments
(3) Performance improvement on GSM8K. We further examine the GSM8K results across all five models and compare Rescue Vector with existing quantization methods. As reported in Table 1, Rescue Vector consistently improves over FP4 across all five models, with gains ranging from 0.38 to 2.19 percentage points. Compared with AWQ, HQQ, and Quanto, Rescue Vector achieves the highest accuracy on Qwen2.5-7B-Instruct (90.60%) and Qwen3-8B (91.81%), and matches HQQ on Qwen3-4B (87.95%). On the two Llama models, however, AWQ and HQQ achieve higher accuracy. These results indicate that the relative advantage of Rescue Vector over existing quantization methods varies across models, while its improvements over the corresponding FP4 baselines remain consistent across all five models.
W3: Representation Differences and Vector Direction
Thank you for raising this concern. We conduct controlled experiments to examine the respective roles of quantization-related representation changes, response content, and answer correctness, and further evaluate vector-direction robustness across models and random seeds.
(1) Controlling for answer correctness and response content.
We consider two controls. First, we construct a Both-Correct vector using examples where both BF16 and FP4 produce correct answers, extracting hidden states from their independently generated responses. Second, we construct a vector from the original BF16-correct and FP4-incorrect pairs under identical response contexts. Specifically, using teacher forcing, both models process the same correct response generated by BF16, and their hidden states are extracted at identical token positions.
The results are summarized below.
| Vector Construction | Response Context | Accuracy (%) | Δ FP4 (pp) |
|---|---|---|---|
| FP4 baseline | – | 73.88 | – |
| Rescue Vector | Different | 76.12 | 2.24 |
| Both-Correct | Different | 74.75 | 0.87 |
| Failure Pairs | Identical | 76.02 | 2.14 |
The Both-Correct control improves accuracy by 0.87 percentage points, but remains 1.37 points below Rescue Vector. This indicates that selecting BF16-correct and FP4-incorrect pairs provides more effective correction directions than using pairs where both models answer correctly. More importantly, under identical response contexts, the resulting vector still improves FP4 accuracy by 2.14 percentage points, only 0.10 percentage points below the original Rescue Vector. This result shows that the correction effect persists when differences in generated response content are controlled, supporting the contribution of quantization-related representation changes to the effectiveness of Rescue Vector.
(2) Examining Random and Reverse directions.
We further examine the direction-related results in Table 2. On Qwen2.5-7B-Instruct, Random and Reverse in the original experiment improve accuracy over FP4 by 0.68 and 0.80 percentage points, respectively, compared with 2.30 points for Rescue Vector. However, these improvements do not generalize across models: both Random and Reverse reduce accuracy on Qwen3-4B and Qwen3-8B, with Reverse falling 0.62 and 3.46 percentage points below the respective FP4 baselines.
To further investigate the Reverse result on Qwen2.5-7B-Instruct, we repeat the experiment with three random seeds (65, 25, and 121). The results are summarized below.
| Seed | Reverse (%) | Rescue Vector (%) |
|---|---|---|
| 65 | 82.25 | 85.27 |
| 25 | 81.72 | 84.98 |
| 121 | 83.03 | 86.73 |
| Avg. | 82.33 | 85.66 |
The FP4 baseline is 83.37%. Reverse falls below this baseline under all three seeds, with an average accuracy of 82.33%. In contrast, Rescue Vector consistently outperforms FP4 across all three seeds, achieving an average accuracy of 85.66%. These results indicate that the improvement from Reverse in the original experiment is not consistently reproduced, whereas Rescue Vector provides more stable gains.
Q1: Matched Activation-Steering Comparison
Please refer to our response to W1. (1) Matched Adaptation Comparison with Activation Steering Methods for the detailed experimental settings and results.
Q2: Both-Correct and Identical-Context Controls
We address both requested controls in our response to W3 above. Specifically, the Both-Correct vector improves PIQA accuracy from 73.88% to 74.75%, while the vector constructed under identical response contexts achieves 76.02%, compared with 76.12% for the original Rescue Vector. These results show that the correction effect remains largely intact when response content is controlled, while selecting quantization-induced failure pairs yields a stronger correction than the Both-Correct construction.
Q3: Cross-Task Transfer
Thank you for the suggestion. To evaluate cross-task transfer without target-task adaptation, we use the Rescue Vector constructed on GSM8K with Qwen2.5-7B-Instruct and directly apply it to ARC-Challenge and OpenBookQA. Specifically, we retain the injection layer selected on GSM8K (Layer 7) for both target tasks, without reconstructing the vector or reselecting the injection layer.
The results are summarized below.
| Target Task | FP4 Baseline (%) | GSM8K-derived Vector (%) | Improvement |
|---|---|---|---|
| ARC-Challenge | 88.05 | 90.87 | 2.82 |
| OpenBookQA | 82.8 | 85.6 | 2.8 |
The GSM8K-derived vector improves accuracy by 2.82 and 2.80 percentage points on ARC-Challenge and OpenBookQA, respectively. These results demonstrate that Rescue Vector can effectively transfer across reasoning tasks without task-specific vector reconstruction or injection-layer retuning, providing further evidence of its cross-task generalization ability.
Q4: Prediction Transitions and Uncertainty Estimates
Please refer to our responses to W2. (2) Sample-level Prediction Transitions and W1. (2) Comparison with Existing Quantization Methods and Uncertainty Estimates for the corresponding experimental results and analyses.
References
[1] Steering Llama 2 via Contrastive Activation Addition.
[2] Representation Engineering: A Top-Down Approach to AI Transparency.
W1: Matched Adaptation and Comparison with Quantization Baselines
We thank the reviewer for raising concerns about the fairness of the adaptation comparisons and the uncertainty of the observed performance differences. We therefore add matched activation-steering baselines and report paired bootstrap confidence intervals.
(1) Matched Adaptation Comparison with Activation Steering Methods.
We compare Rescue Vector with two activation-steering methods: Contrastive Activation Addition (CAA) [1] and RepE-PCA [2]. CAA averages representation differences between positive and negative responses, whereas RepE-PCA extracts the first principal component from per-example representation differences. Both methods use the same labeled examples, BF16 model activations, and development-set selection budget as Rescue Vector. All steering vectors are applied to the same FP4 models for evaluation.
We further extend the comparison to five models across five reasoning benchmarks. The following table reports the average accuracy across the five benchmarks for each model.
Rescue Vector achieves the highest five-benchmark average accuracy across all five models, outperforming CAA and RepE-PCA by 1.61 and 1.94 pp, respectively, in overall average accuracy. These results show that Rescue Vector provides consistent performance gains over both activation-steering baselines under matched adaptation conditions across all five evaluated models.
(2) Comparison with Existing Quantization Methods and Uncertainty Estimates.
We compare Rescue Vector with AWQ, HQQ, and Quanto as alternative quantization methods, and with CAA and RepE-PCA as activation-steering baselines under matched adaptation conditions. Unlike AWQ, HQQ, and Quanto, Rescue Vector preserves FP4 weights and corrects intermediate representations by adding a vector derived from BF16–FP4 hidden-state differences.
For each Qwen model, we calculate paired bootstrap confidence intervals for benchmark-average accuracy across the five tasks against the strongest quantization baseline:
We also report paired bootstrap CIs for CAA and RepE-PCA across all five models:
The 95% confidence intervals exclude zero for all five models in both activation-steering comparisons, indicating statistically significant improvements over CAA and RepE-PCA under matched adaptation conditions. We will include these expanded comparisons and corresponding uncertainty estimates in the revised manuscript.
W2: Open-ended Reasoning and Prediction Transitions
We thank the reviewer for these suggestions. We add MATH-500 results and sample-level prediction-transition analysis under the default configuration.
(1) Open-ended reasoning.
Rescue Vector improves FP4 accuracy by 2.0 and 1.4 percentage points on the two models, recovering 76.9% and 77.8% of the quantization-induced performance losses, respectively. These results support its effectiveness beyond multiple-choice tasks.
(2) Sample-level prediction transitions.
We analyze prediction transitions on Qwen2.5-7B-Instruct across five benchmarks, counting both incorrect-to-correct and correct-to-incorrect changes.
Rescue Vector yields positive net gains on every task, correcting 322 FP4 errors while introducing 211 errors, for a net gain of 111 correct predictions.