acceptodds
Under review as a conference paper at ICLR 2027

Rescue Vector: Bringing Quantized Reasoning Back on Track

Abstract

Low-bit quantization substantially reduces the storage and inference costs of large language models, but can cause pronounced degradation on reasoning tasks. Existing studies have mainly characterized this degradation through task-level performance or sought to mitigate it through improved quantization methods, leaving how quantization changes the reasoning process and internal representations less understood. In this work, we compare paired responses generated by the same model in its unquantized and quantized forms. We find that the two models often follow comparable reasoning initially, but the quantized model can deviate at critical intermediate steps and consequently produce incorrect answers. We further examine their hidden representations and find that the differences between the unquantized and quantized models exhibit a consistent directional pattern across failure cases. Motivated by this finding, we propose **Rescue Vector**, which aggregates these hidden-state differences to correct the intermediate representations of the quantized model. Rescue Vector can be applied directly to a quantized model, without retraining, modifying its quantized weights, or changing the original quantization procedure. Experiments across diverse language models and reasoning benchmarks show that Rescue Vector improves average accuracy under FP4 quantization from 78.76% to 81.16%, recovering 77.2% of the performance lost after quantization. On Llama-3.2-3B-Instruct, the recovery reaches 93.2%. Further analysis shows that, in rescued cases, the corrected hidden states move closer to those of the unquantized model. Together, these findings provide a new perspective on reasoning degradation under quantization from the perspective of internal representations and suggest a practical way to mitigate it.

Primary area:
interpretability and explainable AI
Keywords:
large language models, low-bit quantization, reasoning, representation analysis, representation intervention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LeWsOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reasoning failures caused by low-bit quantization and proposes Rescue Vector, a fixed activation correction constructed from paired BF16-correct and FP4-incorrect responses. The analysis identifies consistent representation differences, and the intervention improves accuracy across several models and benchmarks without retraining or modifying quantized weights.

Critical strengths
  • Interesting and insightful analysis. The observation that quantized responses initially follow comparable reasoning before diverging at intermediate steps is useful, the split-half directional consistency result provides an interesting representation-level perspective on these failures, all these obervations are insightful for us.

  • Simple intervention. The method requires no parameter training and adds only a small stored vector at deployment, this makes it practically appealing as an adaptation of an existing quantized model.

  • Clear experimental separation. Vector construction, development-set layer selection, and final evaluation use separate data, the reported aggregate results are internally consistent, and the ablations examine several important design choices.

Critical weaknesses
  1. The comparisons do not establish the benefit under matched adaptation conditions. The author uses labeled target-task data, bf16 forward passes, and task-specific development-set layer selection. The quantization baselines do not receive an equivalent adaptation procedure. I think their comparison is informative about final performance, but does not isolate the value of the proposed correction. Moreover, the gains over the strongest quantization baseline on the three Qwen models are only 0.07–0.30 percentage points in benchmark-average accuracy, without uncertainty estimates.

  2. Most benchmarks use multiple-choice or yes/no answers, so evidence for broader open-ended reasoning remains limited. The recovery analysis also does not report how often the intervention turns an originally correct FP4 answer into an incorrect one. Aggregate improvements alone do not reveal this trade-off. The gain on gsm8k is very small.

  3. Eq 1 compares representations from independently generated correct and incorrect responses, I think the difference may therefore reflect both quantization effects and differences in response content or correctness. Improved accuracy and directional consistency do not distinguish these explanations. Random and reverse directions also improve Qwen2.5-7B, so more targeted controls would help establish what the learned direction captures.

Questions for authors
  • Can Rescur vector be compared with a lightweight activation-steering baseline using the same labeled examples, BF16 access, and development-set selection budget?

  • Could the authors include control vectors from pairs where both models answer correctly, and differences measured under identical response contexts? These would help separate quantization effects from content and correctness differences.

  • How well does a vector constructed on one task transfer to another without reconstructing it or retuning the injection layer?

  • Could the authors report both incorrect-to-correct and correct-to-incorrect transitions at the default configuration, together with uncertainty estimates for the small advantages over quantization baselines?

7 comments

W1: Matched Adaptation and Comparison with Quantization Baselines

We thank the reviewer for raising concerns about the fairness of the adaptation comparisons and the uncertainty of the observed performance differences. We therefore add matched activation-steering baselines and report paired bootstrap confidence intervals.

(1) Matched Adaptation Comparison with Activation Steering Methods.

We compare Rescue Vector with two activation-steering methods: Contrastive Activation Addition (CAA) [1] and RepE-PCA [2]. CAA averages representation differences between positive and negative responses, whereas RepE-PCA extracts the first principal component from per-example representation differences. Both methods use the same labeled examples, BF16 model activations, and development-set selection budget as Rescue Vector. All steering vectors are applied to the same FP4 models for evaluation.

We further extend the comparison to five models across five reasoning benchmarks. The following table reports the average accuracy across the five benchmarks for each model.

ModelFP4 BaselineCAA [1]RepE-PCA [2]Rescue Vector
Qwen2.5-7B-Instruct83.3784.3784.4585.67
Qwen3-4B81.3581.1581.0182.56
Qwen3-8B85.4285.0485.0586.41
Llama3.1-8B-Instruct76.2577.7576.7579.38
Llama3.2-3B-Instruct67.4269.4768.8471.79
Overall Avg.78.7679.5679.2281.16

Rescue Vector achieves the highest five-benchmark average accuracy across all five models, outperforming CAA and RepE-PCA by 1.61 and 1.94 pp, respectively, in overall average accuracy. These results show that Rescue Vector provides consistent performance gains over both activation-steering baselines under matched adaptation conditions across all five evaluated models.

(2) Comparison with Existing Quantization Methods and Uncertainty Estimates.

We compare Rescue Vector with AWQ, HQQ, and Quanto as alternative quantization methods, and with CAA and RepE-PCA as activation-steering baselines under matched adaptation conditions. Unlike AWQ, HQQ, and Quanto, Rescue Vector preserves FP4 weights and corrects intermediate representations by adding a vector derived from BF16–FP4 hidden-state differences.

For each Qwen model, we calculate paired bootstrap confidence intervals for benchmark-average accuracy across the five tasks against the strongest quantization baseline:

ModelComparisonΔ Avg. Accuracy95% Paired CI
Qwen2.5-7B-InstructRescue − HQQ+0.30[−0.68, +1.23]
Qwen3-4BRescue − HQQ+0.07[−0.87, +1.01]
Qwen3-8BRescue − AWQ+0.10[−0.82, +1.01]

We also report paired bootstrap CIs for CAA and RepE-PCA across all five models:

ModelRescue − CAA (95% CI)Rescue − RepE-PCA (95% CI)
Qwen2.5-7B-Instruct+1.30 [0.19, +2.41]+1.22 [0.26, +2.18]
Qwen3-4B+1.41 [0.39, +2.43]+1.55 [0.20, +2.91]
Qwen3-8B+1.37 [0.08, +2.66]+1.36 [0.29, +2.45]
Llama3.1-8B-Instruct+1.63 [0.07, +3.19]+2.63 [+0.91, +4.34]
Llama3.2-3B-Instruct+2.32 [+0.45, +4.19]+2.95 [+1.08, +4.81]

The 95% confidence intervals exclude zero for all five models in both activation-steering comparisons, indicating statistically significant improvements over CAA and RepE-PCA under matched adaptation conditions. We will include these expanded comparisons and corresponding uncertainty estimates in the revised manuscript.

W2: Open-ended Reasoning and Prediction Transitions

We thank the reviewer for these suggestions. We add MATH-500 results and sample-level prediction-transition analysis under the default configuration.

(1) Open-ended reasoning.

Rescue Vector improves FP4 accuracy by 2.0 and 1.4 percentage points on the two models, recovering 76.9% and 77.8% of the quantization-induced performance losses, respectively. These results support its effectiveness beyond multiple-choice tasks.

ModelBF16 (%)FP4 (%)Rescue Vector (%)
Qwen2.5-7B-Instruct75.272.674.6
Llama-3.2-3B-Instruct41.239.440.8

(2) Sample-level prediction transitions.

We analyze prediction transitions on Qwen2.5-7B-Instruct across five benchmarks, counting both incorrect-to-correct and correct-to-incorrect changes.

DatasetIncorrect → CorrectCorrect → IncorrectNet Gain
GSM8K473413
StrategyQA58499
OpenBookQA391821
ARC-Challenge632043
PIQA1159025
Total322211111

Rescue Vector yields positive net gains on every task, correcting 322 FP4 errors while introducing 211 errors, for a net gain of 111 correct predictions.

(3) Performance improvement on GSM8K. We further examine the GSM8K results across all five models and compare Rescue Vector with existing quantization methods. As reported in Table 1, Rescue Vector consistently improves over FP4 across all five models, with gains ranging from 0.38 to 2.19 percentage points. Compared with AWQ, HQQ, and Quanto, Rescue Vector achieves the highest accuracy on Qwen2.5-7B-Instruct (90.60%) and Qwen3-8B (91.81%), and matches HQQ on Qwen3-4B (87.95%). On the two Llama models, however, AWQ and HQQ achieve higher accuracy. These results indicate that the relative advantage of Rescue Vector over existing quantization methods varies across models, while its improvements over the corresponding FP4 baselines remain consistent across all five models.

W3: Representation Differences and Vector Direction

Thank you for raising this concern. We conduct controlled experiments to examine the respective roles of quantization-related representation changes, response content, and answer correctness, and further evaluate vector-direction robustness across models and random seeds.

(1) Controlling for answer correctness and response content.

We consider two controls. First, we construct a Both-Correct vector using examples where both BF16 and FP4 produce correct answers, extracting hidden states from their independently generated responses. Second, we construct a vector from the original BF16-correct and FP4-incorrect pairs under identical response contexts. Specifically, using teacher forcing, both models process the same correct response generated by BF16, and their hidden states are extracted at identical token positions.

The results are summarized below.

Vector ConstructionResponse ContextAccuracy (%)Δ FP4 (pp)
FP4 baseline–73.88–
Rescue VectorDifferent76.122.24
Both-CorrectDifferent74.750.87
Failure PairsIdentical76.022.14

The Both-Correct control improves accuracy by 0.87 percentage points, but remains 1.37 points below Rescue Vector. This indicates that selecting BF16-correct and FP4-incorrect pairs provides more effective correction directions than using pairs where both models answer correctly. More importantly, under identical response contexts, the resulting vector still improves FP4 accuracy by 2.14 percentage points, only 0.10 percentage points below the original Rescue Vector. This result shows that the correction effect persists when differences in generated response content are controlled, supporting the contribution of quantization-related representation changes to the effectiveness of Rescue Vector.

(2) Examining Random and Reverse directions.

We further examine the direction-related results in Table 2. On Qwen2.5-7B-Instruct, Random and Reverse in the original experiment improve accuracy over FP4 by 0.68 and 0.80 percentage points, respectively, compared with 2.30 points for Rescue Vector. However, these improvements do not generalize across models: both Random and Reverse reduce accuracy on Qwen3-4B and Qwen3-8B, with Reverse falling 0.62 and 3.46 percentage points below the respective FP4 baselines.

To further investigate the Reverse result on Qwen2.5-7B-Instruct, we repeat the experiment with three random seeds (65, 25, and 121). The results are summarized below.

SeedReverse (%)Rescue Vector (%)
6582.2585.27
2581.7284.98
12183.0386.73
Avg.82.3385.66

The FP4 baseline is 83.37%. Reverse falls below this baseline under all three seeds, with an average accuracy of 82.33%. In contrast, Rescue Vector consistently outperforms FP4 across all three seeds, achieving an average accuracy of 85.66%. These results indicate that the improvement from Reverse in the original experiment is not consistently reproduced, whereas Rescue Vector provides more stable gains.

Q1: Matched Activation-Steering Comparison

Please refer to our response to W1. (1) Matched Adaptation Comparison with Activation Steering Methods for the detailed experimental settings and results.

Q2: Both-Correct and Identical-Context Controls

We address both requested controls in our response to W3 above. Specifically, the Both-Correct vector improves PIQA accuracy from 73.88% to 74.75%, while the vector constructed under identical response contexts achieves 76.02%, compared with 76.12% for the original Rescue Vector. These results show that the correction effect remains largely intact when response content is controlled, while selecting quantization-induced failure pairs yields a stronger correction than the Both-Correct construction.

Q3: Cross-Task Transfer

Thank you for the suggestion. To evaluate cross-task transfer without target-task adaptation, we use the Rescue Vector constructed on GSM8K with Qwen2.5-7B-Instruct and directly apply it to ARC-Challenge and OpenBookQA. Specifically, we retain the injection layer selected on GSM8K (Layer 7) for both target tasks, without reconstructing the vector or reselecting the injection layer.

The results are summarized below.

Target TaskFP4 Baseline (%)GSM8K-derived Vector (%)Improvement
ARC-Challenge88.0590.872.82
OpenBookQA82.885.62.8

The GSM8K-derived vector improves accuracy by 2.82 and 2.80 percentage points on ARC-Challenge and OpenBookQA, respectively. These results demonstrate that Rescue Vector can effectively transfer across reasoning tasks without task-specific vector reconstruction or injection-layer retuning, providing further evidence of its cross-task generalization ability.

Q4: Prediction Transitions and Uncertainty Estimates

Please refer to our responses to W2. (2) Sample-level Prediction Transitions and W1. (2) Comparison with Existing Quantization Methods and Uncertainty Estimates for the corresponding experimental results and analyses.

References

[1] Steering Llama 2 via Contrastive Activation Addition.

[2] Representation Engineering: A Top-Down Approach to AI Transparency.

Reviewer_V4RcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates reasoning degradation caused by low-bit quantization of large language models. The authors compare responses from unquantized and quantized models, categorize failures into understanding, reasoning, and execution errors, and identify directional consistency in their hidden-state differences. Based on this observation, the authors propose Rescue Vector. For examples where the unquantized model answers correctly and the quantized model fails, the method averages differences in FFN representations over their respective answer spans. During inference, the resulting fixed vector is added to the FFN output of a selected layer in the quantized model, without retraining or modifying its quantized weights. Experiments cover five models and five reasoning benchmarks. The reported overall average accuracy improves from 78.76% under FP4 quantization to 81.16% with Rescue Vector. The paper also includes ablations on vector direction, construction sample size, injection layer, and intervention strength, alongside limited transfer experiments to AWQ and HQQ.

Critical strengths
  1. The method is simple and delivers consistent improvements with minimal additional vector storage.Rescue Vector requires only an additive intervention at a selected layer, and the unquantized model is unnecessary after construction. All five FP4 models improve in benchmark-average accuracy. For Llama-3.2-3B, accuracy increases from 67.42% to 71.79%, approaching the BF16 result of 72.11%. These results suggest practical value as a lightweight intervention for existing quantized models.

  2. The ablations examine several consequential design choices.Random-direction and reversed-direction controls suggest that the gains depend on the constructed direction rather than arbitrary perturbations. Sample-size and intervention-strength experiments provide additional evidence of robustness in the tested settings. The appendix also demonstrates transfer of FP4-derived vectors to AWQ and HQQ, providing preliminary support for applicability beyond a single quantizer.

Critical weaknesses
  1. The representation analysis does not adequately separate quantization effects from differences in generated content and answer correctness.Equation (1) compares answer-span representations from independently generated responses, specifically selecting examples where BF16 is correct and FP4 is incorrect. These differences therefore combine changes in quantization, generated context, and answer semantics. The directional consistency in Figure 2 does not by itself establish a direction specific to quantization-induced reasoning degradation. Likewise, increased proximity to BF16 representations does not establish the proposed causal mechanism. Comparisons under identical continuation text and controls using examples where both models answer correctly are missing. This limits the paper’s central explanatory contribution.

  2. The evaluation does not sufficiently establish the necessity of paired-failure construction or its advantage over task-specific calibration.The main results use a separately constructed vector and a development-selected layer for each model–benchmark setting. However, the paper does not compare against vectors constructed from unfiltered BF16–FP4 pairs, pairs where both models are correct, or general correct-versus-incorrect response contrasts. It is therefore unclear how much of the benefit reflects quantization-specific correction versus task steering.

3.Some advantages over stronger quantization baselines are small. There is only 0.07 percentage points over HQQ for Qwen3-4B and 0.10 points over AWQ for Qwen3-8B. The main results do not quantify variability across construction and development subsets, and inference latency is not reported. The cross-quantizer experiments help, but cover only one model and three tasks and still require selecting a layer for each target setting.

Questions for authors
  1. Under teacher forcing with identical questions and continuation text, do BF16–FP4 representation differences retain the directional consistency shown in Figure 2? Do vectors constructed under this controlled setting still improve accuracy? These results would materially affect my assessment of the proposed quantization-correction mechanism.

  2. With matched construction budgets and layer-selection procedures, how does Rescue Vector compare against vectors constructed from unfiltered paired examples, examples where both models are correct, and general correct-versus-incorrect response contrasts? Can these comparisons establish the specific value of selecting quantization-induced failures?

3 comments

W1: Controlling for Response Content and Answer Correctness

(1) Controlling for Differences in Generated Content.

We thank the reviewer for pointing out that independently generated responses may introduce differences in content and answer semantics. To control for these factors, we conduct identical-context experiments using the original failure pairs. Under teacher forcing, both models process the same question and BF16-generated correct response, and we extract hidden states at matched token positions to construct new Rescue Vectors.

The results are summarized below:

ModelDatasetFP4 BaselineIdentical-Context VectorRescue Vector
Llama-3.2-3B-InstructPIQA73.8876.0276.12
Qwen2.5-7B-InstructGSM8K89.6190.3790.60

The identical-context vectors improve FP4 accuracy by 2.14 pp on PIQA and 0.76 pp on GSM8K, performing close to the original Rescue Vectors. These results show that the correction effect persists even when differences in generated content are controlled.

(2) Both-Correct Control.

We thank the reviewer for suggesting the Both-Correct control. We construct control vectors from examples that both BF16 and FP4 answer correctly and compare them with Rescue Vectors constructed from BF16-correct and FP4-incorrect pairs. Results on PIQA with Llama-3.2-3B-Instruct are shown below:

Vector ConstructionAccuracyGain
FP4 Baseline73.88--
Both-Correct74.750.87
Rescue Vector76.122.24

We further evaluate this control on GSM8K using three models:

ModelFP4 BaselineBoth-CorrectRescue Vector
Llama-3.2-3B-Instruct64.7565.2066.94
Qwen2.5-7B-Instruct89.6189.9690.60
Qwen3-4B87.5786.5887.95

Rescue Vector consistently outperforms the Both-Correct control across all four evaluation settings, with gains of 0.64–1.74 percentage points over the control vectors. These results indicate that failure pairs provide more targeted information for constructing effective correction vectors.

(3) Directional Consistency and Representation Correction.

We thank the reviewer for this comment. Figure 2 examines whether the BF16–FP4 representation differences share a consistent direction across failure cases. Figure 2(a) compares the mean directions of two independent subsets, while Figure 2(b) verifies their alignment with individual cases in the held-out subset. These results motivate Eq. (2), which aggregates the case-wise differences into Rescue Vector as a shared correction direction, subsequently injected into the FP4 model via Eq. (3).

We further validate this direction through the intervention comparisons in Section 5.4 (Table 2), where Rescue Vector outperforms Random and Reverse across all three Qwen models. Figure 4(b) additionally shows that the rescued representations become closer to their BF16 counterparts. Together, these results support the effectiveness of the estimated correction direction.

W2: Necessity of Paired-Failure Construction

We thank the reviewer for suggesting additional controls to assess the contribution of paired-failure construction. We conduct experiments on GSM8K using three models, with matched vector construction budgets and the same development-set layer-selection procedure across all methods.

We compare Rescue Vector with three alternative constructions: (1) Unfiltered Pairs, which use BF16–FP4 pairs without filtering by answer correctness; (2) Both-Correct Pairs, which use samples where both models answer correctly; and (3) Correct-vs-Wrong Steering, which constructs vectors from general correct and incorrect response contrasts.

The results are summarized below.

ModelFP4UnfilteredBoth-CorrectCorrect-vs-WrongRescue Vector
Llama-3.2-3B-Instruct64.7563.3165.2063.9166.94
Qwen2.5-7B-Instruct89.6188.0189.9689.0990.60
Qwen3-4B87.5786.6686.5887.4187.95

Rescue Vector achieves the highest accuracy across all three models, consistently outperforming the alternative vector construction methods. In particular, Unfiltered and Correct-vs-Wrong vectors fail to improve over the FP4 baseline, while Both-Correct vectors provide smaller gains and even reduce accuracy on Qwen3-4B.

Since all methods use matched construction budgets and the same development-set layer-selection procedure, these results demonstrate the advantage of selecting BF16-correct and FP4-incorrect pairs for vector construction. They further suggest that the improvements go beyond general task steering and development-set calibration.

W3: Robustness and Practical Applicability

(1) Comparison with Strong Quantization Baselines and Activation Steering Methods.

We compare Rescue Vector with AWQ, HQQ, and Quanto as quantization baselines. Unlike these methods, Rescue Vector preserves the quantized weights and applies a lightweight post-hoc correction without retraining or rerunning quantization. We further compare Rescue Vector with two activation-steering methods, CAA [1] and RepE-PCA [2], under matched adaptation conditions, using the same labeled examples, BF16 activations, selection budget, and FP4 evaluation models.

We extend the activation-steering comparisons to five models across five reasoning benchmarks. The following table reports the average accuracy across the five benchmarks for each model.

ModelFP4CAA [1]RepE-PCA [2]Rescue Vector
Qwen2.5-7B83.3784.3784.4585.67
Qwen3-4B81.3581.1581.0182.56
Qwen3-8B85.4285.0485.0586.41
Llama3.1-8B76.2577.7576.7579.38
Llama3.2-3B67.4269.4768.8471.79
Overall Avg.78.7679.5679.2281.16

Rescue Vector achieves the highest five-benchmark average accuracy across all five models, outperforming CAA by 1.61 pp and RepE-PCA by 1.94 pp in overall average accuracy. These results demonstrate that Rescue Vector consistently outperforms both activation-steering baselines across different model families.

(2) Robustness to Construction and Development Subsets.

To evaluate sensitivity to construction and development subsets, we repeat the complete pipeline with three random seeds on GSM8K using Qwen2.5-7B-Instruct and evaluate all runs on the same test set. Appendix B.2 (Table 14) additionally reports robustness to construction-set size and random sampling.

Random SeedAccuracy (%)
12312390.67
340790.83
33390.75
Mean ± Std90.75 ± 0.08

The mean accuracy is 90.75 ± 0.08%, indicating low performance variation across different construction and development subsets.

(3) Inference Latency and Computational Overhead.

We thank the reviewer for highlighting the importance of inference efficiency. We conducted additional inference latency experiments on Qwen2.5-7B-Instruct, comparing the FP4 model with and without Rescue Vector. The results are summarized below.

MethodInference Latency (s)Overhead
FP47.404--
FP4 + Rescue Vector7.4440.54%

The results show that Rescue Vector introduces only 0.54% additional inference latency, indicating minimal overhead in the evaluated setting.

(4) Cross-Quantizer Transfer and Layer Selection.

As shown in Appendix B.3 (Table 15), we directly apply Rescue Vectors constructed from BF16–FP4 representation differences to AWQ and HQQ, without reconstructing the vectors for the target quantizers. Accuracy improvements are observed across all three evaluated tasks, with gains of 3.20 and 1.40 percentage points on OpenBookQA for AWQ and HQQ, respectively.

Regarding injection-layer selection, Appendix B.1 reports performance across different injection layers, demonstrating the importance of selecting an appropriate injection location. We select the injection layer using only 100 development examples, rather than the full test set, and keep it fixed during subsequent inference. Although target-specific layer selection remains necessary, cross-quantizer transfer requires neither new paired failure cases nor vector reconstruction. Thus, existing Rescue Vectors can be reused across different quantization methods with limited additional adaptation costs.

Q1: Identical-Continuation Validation

We thank the reviewer for this question. The identical-context experimental setup and corresponding accuracy results are reported in our response to W1(1). Controlling for Differences in Generated Content. Beyond these accuracy results, we further examine whether the BF16–FP4 representation differences retain directional consistency under identical continuations. The split-half cosine similarities are 0.960, 0.945, and 0.950 at layers 7, 14, and 21, respectively. These results demonstrate that directional consistency is preserved when both models process identical continuations.

Q2: Necessity of Paired-Failure Construction

We thank the reviewer for this question. Please refer to our response to W2: Necessity of Paired-Failure Construction.

References

[1] Steering Llama 2 via Contrastive Activation Addition.

[2] Representation Engineering: A Top-Down Approach to AI Transparency.

AuthorsOpenReview ↗

Updated Manuscript and Additional Clarifications

We sincerely thank you again for your thoughtful and constructive feedback. We have carefully revised the manuscript to incorporate additional identical-context experiments, vector construction controls, and robustness analyses, with the corresponding results presented in the main text and appendix.

We greatly appreciate your suggestions, which have helped us strengthen the paper. If you have an opportunity to review the updated manuscript, we would be very grateful for any further thoughts or suggestions you may have. We would also be happy to provide additional clarifications if needed.

Reviewer_o3wJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper observes that the quantized model can deviate at critical steps and there is a consistent directional difference between the unquantized and quantized model.

This paper proposes Rescue Vector to estimate a correct direction between unquantized and quantized models for recovering the gap without retraining or modifying the quantized weights.

Extensive experiments are conducted to show Rescue Vector consistently improves FP4-quantized models.

Critical strengths

1, The observation for quantized models deviating at critical intermediate steps by comparing paired responses and showing a consistent directional pattern across failure cases is interesting.

2, Experimental results across Qwen and Llama models and five reasoning benchmarks demonstrate the effectiveness of Rescue Vector to reduce the gap between quantized and unquantized models.

3, Ablation studies are provided to support the Rescue Vector is helping move closer to the original unquantized reasoning models.

Critical weaknesses

1, lacking the experiments for magnitude consistency of hidden-stage differences. Fig. 2 only shows the directional consistency of hidden-stage differences. If magnitude varies very much, a fixed \mu may not be able to recover.

2, More explanation for Rescue Vector. Line 268-269 mention that Rescue Vector is a fixed vector. When there is a successful case going through the network, the Rescue Vector should be smaller compared to feeding a failure case to recover. The Rescue Vector makes more sense for a flexible vector depending on input, which is contradictory to a fixed vector.

3, Confusion for Vector Direction in Table 2. Reverse direction only performs a little bit worse than Rescue Vector. Why this is the case?

4, Explanation why \mu = 1.0 gives the best performance. \mu is for controlling the magnitude strength. With a successful case, \mu should be smaller. A reasonably quantized model, especially for Queen 2.5-7B-instruct should have a high pass rate. \mu should be smaller for this case.

5, Missing experimental results for non-reasoning benchmarks. How Rescue Vectors generalize to non-reasoning tasks for understanding if Rescue Vector is overfitting or not.

Questions for authors

See Weaknesses. I will change my score based on rebuttal.

3 comments

W1: Magnitude Consistency of Hidden-State Differences and the Effectiveness of a Fixed μ

We thank the reviewer for the valuable suggestion. We provide additional analysis of magnitude consistency and the effectiveness of a fixed injection strength μ.

(1) Magnitude Consistency Analysis.

Under identical text contexts, we use teacher forcing to provide BF16 and FP4 models with the same questions and continuation tokens, extracting hidden states at matched answer-token positions. We compute the hidden-state differences for held-out failure cases on GSM8K and project each sample's difference onto the corresponding shared Rescue Vector. The normalized projection coefficient measures the relative magnitude along the shared correction direction. At their respective selected injection layers, the median projection coefficients for Qwen2.5-7B-Instruct, Qwen3-4B, and Llama-3.2-3B-Instruct are 0.871, 0.933, and 1.081, respectively, with corresponding 10th–90th percentile ranges of [0.727, 1.082], [0.809, 1.073], and [0.778, 1.245]. These results show that the projection magnitudes are relatively concentrated across samples, with the middle 80% falling within approximately 0.7–1.25.

(2) Effectiveness of a Fixed μ.

Table 4 in the paper reports an ablation study of different injection strengths. On Qwen2.5-7B-Instruct, all tested values of μ from 0.25 to 2.0 outperform the FP4 baseline (83.37%), achieving accuracies ranging from 84.08% to 85.67%, with the best performance of 85.67% at μ = 1.0. These experiments use a single global μ for all samples in each evaluation, without adjusting the injection strength according to individual representation differences. The consistent improvements across a broad range of μ values suggest that the shared correction direction provides the essential correction signal, while μ controls its overall injection strength. Therefore, sample-specific magnitude adjustment is not necessary to achieve effective recovery in this setting.

W2: On the Use of a Fixed Rescue Vector

We thank the reviewer for the valuable suggestion. Input-adaptive intervention is an interesting direction for future exploration. In this work, we focus on a fixed-vector intervention that does not require an additional mechanism to determine the intervention strength for each input. We further explain its effectiveness from the following perspectives.

(1) Correction Mechanism.

As described in Eq. (3) and Lines 262–264, Rescue Vector modifies the intermediate hidden states at the selected layer, and the corrected representations are subsequently passed through the remaining Transformer layers for further reasoning. Although the same correction vector is injected into different inputs, its interaction with input-specific hidden states and subsequent Transformer layers can lead to different effects on their final predictions. Furthermore, Figure 4(b) shows that, for successfully recovered cases, the corrected hidden states become closer to their BF16 counterparts, providing further evidence of the representation correction achieved by Rescue Vector.

(2) Empirical Analysis of Prediction Transitions.

To quantify this trade-off, we analyze prediction transitions on Qwen2.5-7B-Instruct across all five benchmarks. Rescue Vector corrects 322 initially incorrect predictions (Incorrect → Correct), while 211 initially correct predictions become incorrect (Correct → Incorrect), yielding a net gain of 111 correct predictions. Importantly, all five benchmarks exhibit positive net gains, as summarized below.

DatasetIncorrect → CorrectCorrect → IncorrectNet Gain
GSM8K473413
StrategyQA58499
OpenBookQA391821
ARC-Challenge632043
PIQA1159025
Total322211111

These results show that the fixed intervention corrects more initially incorrect predictions than it degrades initially correct ones, resulting in positive net accuracy gains across all five benchmarks. Furthermore, Table 4 shows that Rescue Vector consistently improves average accuracy over the FP4 baseline across a range of fixed injection strengths, suggesting that its effectiveness does not require precise input-specific adjustment of the injection strength.

We acknowledge that adaptive intervention may further reduce undesirable prediction changes. Exploring input-dependent correction strengths or selective intervention mechanisms is therefore a promising direction for future work.

W3: Confusion Regarding the Vector Direction in Table 2

We thank the reviewer for highlighting the unexpected performance of the reversed direction on Qwen2.5-7B-Instruct. We further examine this observation through cross-model comparisons and additional experiments with different random seeds.

(1) Cross-model comparison. As shown in Table 2, Reverse slightly improves Qwen2.5-7B-Instruct over the FP4 baseline (84.17% vs. 83.37%). However, this improvement does not generalize to the other models. Reverse reduces accuracy by 0.62 and 3.46 percentage points on Qwen3-4B and Qwen3-8B, respectively. Across the three models, Reverse achieves an average accuracy of 82.29%, below the FP4 baseline (83.38%) and substantially lower than Rescue Vector (84.88%). These results indicate that the reversed direction does not provide a consistent performance benefit.

(2) Robustness across random seeds. To further examine the isolated improvement on Qwen2.5-7B-Instruct, we conduct additional experiments using three random seeds (65, 25, and 121). As shown below, Reverse consistently underperforms the FP4 baseline across all three seeds, achieving an average accuracy of 82.33%. In contrast, Rescue Vector consistently outperforms both Reverse and the FP4 baseline, with an average accuracy of 85.66%.

SeedReverseRescue Vector
6582.2585.27
2581.7284.98
12183.0386.73
Average82.3385.66

These findings suggest that the slight improvement from Reverse in the original Qwen2.5-7B-Instruct experiment is sensitive to the experimental configuration and does not represent a consistently beneficial correction direction. More importantly, Rescue Vector maintains a clear advantage over Reverse across both models and repeated experiments, providing stronger empirical support for the importance of the correction direction.

W4: Selection of the Injection Strength

We thank the reviewer for the suggestion regarding the choice of injection strength. Rescue Vector provides a shared correction direction for intermediate representations, while μ controls the strength of the intervention. The modified representations are then processed by the remaining Transformer layers to produce the final prediction.

To examine the effect of injection strength, we evaluate μ values from 0.25 to 2.0 on Qwen2.5-7B-Instruct, with complete results reported in Appendix Table 16. Smaller values (μ = 0.25, 0.5, and 0.75) also improve performance, achieving average accuracies of 85.01%, 84.98%, and 85.12%, respectively. However, μ = 1.0 achieves the highest average accuracy of 85.67%, while larger values (μ = 1.5 and 2.0) reduce accuracy to 84.63% and 84.08%. Although μ = 1.0 is not necessarily optimal for every individual benchmark, it achieves the highest average accuracy across the five benchmarks. Since we aim to use a single global injection strength rather than tune μ separately for each benchmark, we adopt μ = 1.0 as the default setting.

We appreciate the reviewer's suggestion that input-adaptive intervention may better accommodate individual inputs. In this work, we focus on a fixed-vector intervention with a single global μ, without requiring input-specific strength adjustment. As shown in W2: On the Use of a Fixed Rescue Vector, this fixed intervention achieves positive net accuracy gains across all five benchmarks. Exploring input-adaptive intervention remains an interesting direction for future work.

W5: Evaluation on Non-Reasoning Benchmarks

We thank the reviewer for this valuable suggestion. To evaluate the applicability of Rescue Vector beyond reasoning tasks, we conduct additional experiments on SST-2, a sentiment classification benchmark, using Qwen3-4B and Llama-3.2-3B-Instruct. Following the same vector construction and intervention procedure as in our paper, Rescue Vector improves FP4 accuracy from 70.53% to 86.47% (+15.94 pp) on Qwen3-4B and from 73.28% to 84.75% (+11.47 pp) on Llama-3.2-3B-Instruct. These results provide further evidence that Rescue Vector can mitigate quantization-induced performance degradation beyond reasoning tasks, suggesting that its effectiveness is not limited to reasoning benchmarks.

ModelFP4BF16Rescue Vector
Qwen3-4B70.5390.1486.47
Llama-3.2-3B-Instruct73.2883.4984.75
AuthorsOpenReview ↗

Further Clarifications in the Revised Manuscript

We sincerely thank you again for your constructive feedback. We have now incorporated the additional analyses on magnitude consistency, fixed-vector intervention, and directional robustness into the revised manuscript, along with the extended evaluation on non-reasoning tasks.

We hope these revisions provide a clearer explanation of our method and its empirical behavior. We would greatly appreciate any further feedback you may have on the updated manuscript, particularly if there are remaining concerns that we could address.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank all reviewers for their constructive feedback. In response, we conducted additional experiments and analyses, added new baselines and controlled comparisons, and further examined the robustness and generalization of Rescue Vector. We summarize the main concerns and responses below.

1. Representation Analysis and Quantization-Induced Effects (V4Rc W1/Q1; LeWs W3/Q2)

We conducted identical-context experiments under teacher forcing, where BF16 and FP4 models process the same questions and continuation tokens. The resulting representation differences retain directional consistency, with split-half cosine similarities of 0.960/0.945/0.950 at layers 7/14/21. The resulting vectors improve accuracy by 2.14/0.76 pp on PIQA/GSM8K, respectively, approaching the original Rescue Vector results. We also introduced Both-Correct controls, showing that Rescue Vector consistently outperforms these controls across four evaluation settings.

2. Necessity of Paired-Failure Construction and Matched Baselines (V4Rc W2/Q2; LeWs W1/Q1)

We compared Rescue Vector against Unfiltered Pairs, Both-Correct Pairs, and Correct-vs-Wrong Steering under matched construction budgets and layer-selection procedures on three models. Rescue Vector achieves the highest accuracy across all three models, demonstrating the advantage of selecting BF16-correct and FP4-incorrect pairs. We further compared Rescue Vector with two activation-steering baselines, CAA and RepE-PCA, across five models and five reasoning benchmarks under matched adaptation conditions. Rescue Vector achieves the highest five-task average accuracy across all five models, outperforming CAA and RepE-PCA by 1.61 and 1.94 pp, respectively. Paired bootstrap 95% confidence intervals exclude zero for both comparisons across all five models, confirming statistically significant improvements over both activation-steering baselines.

3. Fixed-Vector Intervention, Magnitude Consistency, and Directional Robustness (o3wJ W1/W2/W3/W4; LeWs W2/W3/Q4)

We analyzed the magnitude consistency of hidden-state differences under identical contexts. The middle 80% of normalized projection coefficients fall approximately within 0.7–1.25 across three models. We further evaluated fixed injection strengths from μ = 0.25 to 2.0, showing that μ = 1.0 achieves the highest average accuracy on Qwen2.5-7B-Instruct. Prediction-transition analysis shows 322 Incorrect-to-Correct changes versus 211 Correct-to-Incorrect changes, yielding a net gain of 111 correct predictions across five benchmarks. Additional cross-model and multi-seed experiments show that Reverse does not provide consistent improvements, whereas Rescue Vector maintains stable gains over the FP4 baseline.

4. Generalization Across Tasks and Quantization Methods (o3wJ W5; V4Rc W3; LeWs W2/Q3)

We extended evaluation beyond multiple-choice reasoning to open-ended reasoning on MATH-500 and non-reasoning sentiment classification on SST-2, observing consistent improvements over FP4 baselines. We also demonstrated cross-task transfer by directly applying a GSM8K-derived vector to ARC-Challenge and OpenBookQA, improving accuracy by 2.82/2.80 pp without reconstructing the vector or retuning the injection layer. Furthermore, we evaluated transfer to AWQ and HQQ, observing improvements across all three evaluated tasks without reconstructing vectors for the target quantizers.

5. Robustness, Statistical Uncertainty, and Inference Efficiency (V4Rc W3; LeWs W1/Q4)

We reported paired bootstrap confidence intervals for comparisons with stronger quantization baselines, as well as matched activation-steering baselines. The results show that Rescue Vector significantly outperforms CAA and RepE-PCA on OpenBookQA, while the confidence intervals for comparisons with stronger quantization baselines include zero. We additionally evaluated robustness across three random seeds, achieving 90.75 ± 0.08% accuracy on GSM8K. Finally, inference latency experiments show only 0.54% additional overhead when applying Rescue Vector to the FP4 model.

Overall, these additional experiments strengthen the empirical evidence for Rescue Vector, address the reviewers' concerns regarding its construction and intervention mechanisms, and further demonstrate its generalization and practical applicability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.