acceptodds
Under review as a conference paper at ICLR 2027

Cross-Scale Model Resilience for LLM Inference Under Hardware Faults

Abstract

Modern large-language models (LLMs) deployed on computing systems at scale are increasingly exposed to transient faults. These faults can silently corrupt persistent model state (i.e., parameters), affecting subsequent inference and potentially causing consequential decisions and actions. Existing resilience mechanisms detect and correct corruptions in bits, activations, and parameters. However, bounding the numerical errors from such corruptions does not guarantee semantically correct outputs. In this work, we present cross-scale model resilience (CSMR), a mechanism that detects and mitigates fault-induced errors at the output level. CSMR pairs a target model with a lightweight auxiliary model and measures their token- and sequence-level semantic discrepancy, triggering correction only when the discrepancy exceeds a calibrated threshold. To minimize overhead, we integrate CSMR into speculative decoding, using the auxiliary as the drafter and embedding discrepancy checking into the acceptance stage. Upon detection, CSMR selectively reloads corrupted parameters. Across multiple LLMs and benchmarks under diverse fault models, CSMR restores accuracy from 0% to 95.6–99.6%, with less than 0.4% latency overhead in fault-free executions.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
large language models, fault resilience

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UYM4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes shifting the criteria for LLM inference accuracy from numerical correctness to semantic-level correctness. It calculates the per-token discrepancy scores between the target and draft models (taking the maximum of the normalized NLL and JS divergence scores) over a sliding window. If this value exceeds a predefined threshold, SDC is suspected. Upon detection, GPU-native hashing is used to identify corrupted weight tensors, which are selectively reloaded, and the response is regenerated. By integrating detection into the verification phase of speculative decoding, the authors minimize overhead. The paper reports that accuracy recovers to 96.9–99.4% under critical fault conditions, while latency overhead remains as low as 0.1–0.4% under fault-free and benign fault conditions.

Critical strengths
  1. The motivational assumptions are clear. Unfaulted target and draft models should produce reasonably aligned output distributions, and the draft model already present in a speculative decoding can be repurposed for fault detection. In environments where speculative decoding is already utilized, SDC detection can be performed with minimal additional overhead, which serves as a key strength.

  2. Because faults are detected based on differences in output distributions rather than direct weight integrity checks, the mechanism can detect not only weight memory corruption but also activation corruption and computational faults to some extent. Providing this expanded fault coverage without dedicated additional protection mechanisms is a clear advantage.

Critical weaknesses
  1. CSMR mainly consists of a discrepancy-based trigger and a hash-based selective recovery module. The primary protected target appears to be weight memory errors. However, during selective recovery, the system hashes all weight tensors on the GPU after detection to identify corrupted ones. A natural alternative would be to perform this hash check periodically without relying on a discrepancy-based trigger. For instance, by distributing checks across iterations (e.g., hashing one layer per iteration) or adjusting the inspection interval to trade off overhead against detection latency. To justify the necessity of a discrepancy-based trigger, a direct comparison against this periodic baseline is required.

  2. This paper claims that CSMR is compatible with any draft–verify scheme that uses a smaller draft model without modification. However, modern speculative decoding methods such as EAGLE-3 and DSpark feed the target model's hidden states into the draft model to generate tokens, rendering CSMR's detection effectiveness under these methods uncertain. Under these architectures, hidden features corrupted by weight errors in the target model may be propagated to the draft model. Consequently, the prediction errors of the two models may become correlated, potentially preventing the distribution discrepancy from exceeding the detection threshold even when the target model produces incorrect outputs. Note that the experiment in Section 4.7 (both models setting) injects independent faults into both models separately, which does not simulate the scenario where target corruption propagates to the draft model through hidden states.

Questions for authors
  1. CSMR's selective recovery hashes all weight tensors post-detection to identify corrupted ones. How does the discrepancy-based trigger perform compared to a baseline that performs the same hash checks periodically without the trigger (e.g., checking one layer per iteration or tuning the interval to match CSMR's 0.1–0.4% latency overhead)? While periodic hashing can catch all persistent weight corruptions regardless of whether they are benign or critical, it leaves an exposure window where corrupted outputs might be served between checks. Please provide a comparison of both approaches in terms of the trade-off between overhead and exposure window size.

  2. Does the true positive rate (TPR) hold up under draft models that rely on target hidden states, such as EAGLE-3?

  3. In Section 4.5, the draft model's memory footprint is calculated as additional overhead. However, if the target scenario assumes an environment already running speculative decoding—where SDC detection is essentially provided "for free"—the draft model memory should be excluded from the incremental overhead attributable to CSMR. Conversely, if applied to an environment without pre-existing speculative decoding, the draft model memory is indeed an overhead, but the latency acceleration gained from speculative decoding must also be factored in, making it inaccurate to present it purely as an overhead. The former interpretation appears more appropriate. Clarifying the intended deployment scenario and presenting overheads against a consistent baseline would improve clarity. The current text leaves room for ambiguity on this point.

2 comments
AuthorsOpenReview ↗

Rebuttal Response (Thank You for the Feedback)

We thank the reviewer for recognizing CSMR's key strengths (its low-overhead integration of SDC detection into speculative decoding and its potential to detect diverse types of hardware faults beyond weight corruption). Below, we address the concerns and questions.

(W1/Q1) A natural alternative would be to perform this hash check periodically without relying on a discrepancy-based trigger. To justify the necessity of a discrepancy-based trigger, a direct comparison against this periodic baseline is required. How does the discrepancy-based trigger perform compared to a baseline that performs the same hash checks periodically without the trigger (e.g., checking one layer per iteration or tuning the interval to match CSMR's 0.1–0.4% latency overhead)? Please provide a comparison of both approaches in terms of the trade-off between overhead and exposure window size.

We appreciate the opportunity to address this concern. We implement and evaluate a periodic hashing baseline that hashes all weights every XX decoding steps using CSMR's hashing methodology. For a fair comparison, all hashes stay on the GPU and are compared without synchronization, mirroring CSMR's detection.

(1) To match CSMR's overhead, periodic hashing requires a 43–100$\times$ larger exposure window. Following our setup in Section 4.4, we measure the fault-free latency of periodic hashing on 100 prompts for X∈{1,2,4,8,16}X \in \{ 1, 2, 4, 8, 16 \}. Because hashing's overhead scales roughly with 1/X1/X, we determine the interval required to match CSMR's overhead by computing X∗=O1/OCSMRX^* = O_1 / O_\mathrm{CSMR}, where O1O_1 is the overhead of hashing at every decoding step and OCSMRO_\mathrm{CSMR} is CSMR's overhead. As X∗X^* is measured in speculative decoding steps, we convert it to tokens using the measured average number of tokens accepted per decoding step (3.7–4.4). We report the results averaged across 5 trials below.

Check Interval (X)Llama-3-8BFalcon3-10BQwen3-32B
CSMR+0.36%+0.35%+0.15%
1+41.25%+41.51%+34.43%
2+20.25%+20.39%+17.04%
4+9.81%+9.88%+8.31%
8+4.58%+4.58%+3.98%
16+1.97%+2.09%+1.82%
X∗X^* (tokens)4305131,001

At short intervals (1–8), periodic hashing increases latency by 3.98–41.51%, compared to 0.15–0.36% for CSMR. To match CSMR's overhead, periodic hashing would need an interval of 430–1,001 tokens and it grows with model size. This is equivalent to 3–4 complete average responses, so corrupted responses would routinely be generated in full before any check occurs.

(2) Checking one layer per step also yields higher overhead and a larger exposure window than CSMR. Because hashing's cost scales linearly with the number of parameters hashed, we estimate from our X=1X=1 results that checking one layer per step incurs 0.54–1.29% overhead with an exposure window of 120–279 tokens (one full sweep over all layers). In contrast, CSMR operates at 0.15–0.36% overhead with a 10-token detection window.

Therefore, while periodic hashing eventually catches all persistent corruptions, it faces a worse trade-off between the exposure window and overhead. It also detects benign faults, which are costly to recover from and CSMR is intentionally designed to avoid. We will include this comparison in our final version.

(W2/Q2) Modern speculative decoding methods such as EAGLE-3 and DSpark feed the target model's hidden states into the draft model to generate tokens, rendering CSMR's detection effectiveness under these methods uncertain. Does the true positive rate (TPR) hold up under draft models that rely on target hidden states, such as EAGLE-3?

To address this concern, we evaluate CSMR when integrated into EAGLE3 and Dflash2. For EAGLE3, we use a public checkpoint [1] of a draft model for Qwen3-32B, while for Dflash2 we use the official draft model for Qwen3-4B. The EAGLE3 draft model for Qwen3-32B has only a subset of the target models vocabulary, so we restrict decoding to only generate tokens in both models’ vocabularies. For DFlash2, we use only the JS divergence because the draft model assigns near-zero NLL to most clean tokens, so the NLL's interquartile range is too small for the robust normalization to be stable. For both model pairs, we re-calibrate CSMR’s hyperparameters and sample 100 benign and critical faults on GSM8K for the target models. We report the results below.

Target ModelSpeculative Decoding MethodTPRFPR (Fault-Free)FPR (Benign)
Qwen3-32BEAGLE394.6%0%0%
Qwen3-4BDFlash298.0%1.0%1.0%

CSMR remains effective when paired with modern speculative decoding methods. Across methods, TPR is 94.6–98.0%, comparable to the 95.6–99.6% in our main evaluation, with 1% FPR under benign faults and fault-free execution. We will include these results in our revision.

[1] https://huggingface.co/RedHatAI/Qwen3-32B-speculator.eagle3

AuthorsOpenReview ↗

Rebuttal Response (Thank You for the Feedback) - Cont'd

(Q3) Clarifying the intended deployment scenario and presenting overheads against a consistent baseline would improve clarity.

We clarify that CSMR is intended for deployments that already use speculative decoding. We discuss the draft model’s memory footprint in Section 4.5 for deployments that would need to enable speculative decoding. We will clarify this distinction in the final version and state explicitly that the draft model’s memory footprint is excluded from CSMR’s incremental overhead.


We hope our responses address the reviewer's questions and concerns. We would be happy to provide further clarification if needed.

Reviewer_qNzoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that some hardware faults are benign during LLM inference because they do not affect the generated output. Therefore, rather than correcting every detected hardware fault, the system should monitor output behavior and trigger recovery only when faults cause meaningful divergence. The authors adapt speculative decoding to detect potentially cascading errors in the target model while limiting the impact of draft-model faults. They also introduce selective weight recovery to reduce the cost of reloading corrupted parameters. The empirical study provides encouraging evidence for the effectiveness and efficiency of the proposed method.

Critical strengths
  1. The paper presents a clever design that integrates output-level fault detection with speculative decoding and selective weight recovery.

  2. The empirical evaluation is thorough, covering multiple LLM families, memory-fault models, and scenarios involving faults in the draft model.

  3. The paper provides useful analyses of runtime and memory overhead, including comparisons with existing fault-protection mechanisms.

Critical weaknesses
  1. Restricted decoding setting. All experiments assume greedy decoding. It remains unclear whether the proposed discrepancy-based detection mechanism remains effective under stochastic sampling, which is widely used in LLM inference and RL rollouts.

  2. Limited improvement in fault-recovery effectiveness. CSMR achieves recovery accuracy comparable to, but not consistently better than, Structural Coding and LM-Fix (Figure 4). Therefore, its primary contribution lies in efficiency rather than improved resilience.

  3. Unclear deployment-wide efficiency gains. Under critical faults, CSMR has latency comparable to Structural Coding and LM-Fix. The improvement under fault-free execution is modest in absolute terms, although the savings under benign faults are substantial (Figure 5). Since the overall benefit depends on the frequency and persistence of different fault types in production, an analysis using realistic fault distributions would better establish the practical significance of the efficiency gains.

Questions for authors

1. Generalization beyond greedy decoding

How would CSMR perform under stochastic decoding with nonzero temperature? Since sampling can naturally produce different outputs without hardware faults, would this increase the false-positive rate or reduce detection effectiveness? What happens at higher temperatures, such as those used in RL rollouts?

2. Expected runtime improvement under realistic fault distributions

Could the authors estimate the expected runtime savings over Structural Coding and LM-Fix under realistic production fault distributions? Since the savings differ substantially among fault-free, benign-fault, and critical-fault conditions, a deployment-level analysis accounting for fault frequency and persistence would help quantify the practical benefits of CSMR.

3. Choice of discrepancy metric

Why is Jensen–Shannon divergence preferred over KL divergence in Equation (2)? Although JS divergence is symmetric and bounded, could the authors provide an ablation comparing JS with forward and reverse KL in terms of detection TPR, FPR, and computational overhead?

4. Minor questions

  1. The references to "scikit-learn developers" (line 246), "SGLang Team," and "Appleby" (line 265) appear incomplete. Could the authors provide formal citations?

  2. Figure 7 shows relatively low TPR for certain MMLU settings. Could this be related to generation length? Since the detector aggregates discrepancies over a 10-token window, shorter responses may provide fewer opportunities to detect cascading errors. Could the authors analyze TPR as a function of output length and window size?

5 comments
AuthorsOpenReview ↗

Rebuttal Response (Thank You for the Feedback)

We thank the reviewer for recognizing CSMR's design, particularly its integration of output-level fault detection, speculative decoding, and selective weight recovery, as well as the thoroughness of our evaluation. Below, we address the concerns and questions.

(W1/Q1) All experiments assume greedy decoding. It remains unclear whether the proposed discrepancy-based detection mechanism remains effective under stochastic sampling. How would CSMR perform under stochastic decoding with nonzero temperature? Since sampling can naturally produce different outputs without hardware faults, would this increase the false-positive rate or reduce detection effectiveness? What happens at higher temperatures, such as those used in RL rollouts?

We thank the reviewer for raising this question. We extend our evaluation to two stochastic decoding settings on Falcon3 and GSM8K. We test T=1.0T=1.0 and a higher-temperature setting with T=1.2T=1.2 and top-p=0.95p=0.95. Both temperatures have been used in RL rollouts.

SettingCalibration SettingTPRFPR (Fault-Free)FPR (Benign Fault)
T=1.0T=1.0Greedy calibration reused99.5%2.2%1.7%
Recalibrated99.3%0.3%0.0%
T=1.2T=1.2, top-p=0.95p=0.95Greedy calibration reused99.5%1.3%1.3%
Recalibrated99.5%0.0%0.0%

CSMR remains effective under stochastic decoding. Reusing the hyperparameters calibrated under greedy decoding yields a TPR of 99.5% in both sampling settings, with FPRs of 1.3–2.2% across the fault-free and benign-fault conditions. Recalibrating CSMR for each sampling setting reduces FPRs to 0.0–0.3% while maintaining TPRs of 99.3–99.5%. We will include these results in the final version.

(W2) CSMR achieves recovery accuracy comparable to, but not consistently better than, Structural Coding and LM-Fix (Figure 4). Therefore, its primary contribution lies in efficiency rather than improved resilience.

We agree that CSMR’s recovery accuracy is comparable to that of Structural Coding and LM-Fix. CSMR’s primary contribution is a semantics-aware approach to LLM fault resilience that improves runtime efficiency while maintaining recovery accuracy similar to exhaustive approaches. As shown in Section 3.1, the effects of most faults are masked; CSMR uses this observation to recover selectively from faults that cause meaningful semantic corruption.

More generally, these results highlight a trade-off between runtime overhead and fault detection. When guaranteed detection is required, exhaustive protection mechanisms are preferable despite their higher latency. CSMR offers a different point in this trade-off, substantially reducing latency during fault-free inference and in the presence of benign faults while achieving a TPR of 95.6–99.6% on critical faults. CSMR also motivates future research exploring how the semantic effects of faults can guide the design of more efficient protection mechanisms. We will include this discussion in our revision.

(W3/Q2) Unclear deployment-wide efficiency gains. Under critical faults, CSMR has latency comparable to Structural Coding and LM-Fix. Could the authors estimate the expected runtime savings over Structural Coding and LM-Fix under realistic production fault distributions?

Estimating deployment-wide runtime savings requires knowing how often inference requests encounter corrupted weights. Prior studies report that double-bit memory errors are rare on individual GPUs [1,2]. We therefore expect most requests to run fault-free, where CSMR has its largest latency advantage over prior work.

Additionally, we quantify the expected runtime when a double-bit fault occurs. Appendix B reports that across all of our sampled double-bit faults, 88.6% are benign and 11.4% are critical. Weighting the corresponding latencies in Figure 5 by these proportions gives an expected latency of 3.43s / 3.95s / 20.49s for CSMR, 3.79s / 4.24s / 20.80s for Structural Coding, and 4.24s / 5.03s / 22.21s for LM-Fix (Llama / Falcon / Qwen). On average across models, CSMR reduces this expected latency by 5.9% vs. Structural Coding and 16.1% vs. LM-Fix. The improvement primarily comes from CSMR not detecting benign faults, demonstrating the advantage of our semantics-aware approach. We will include these results in our final version.

[1] Oles et al., “Understanding GPU Memory Corruption at Extreme Scale: The Summit Case Study,” ICS (2024).

[2] Cui et al., “Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUs,” SC (2025).

AuthorsOpenReview ↗

Rebuttal Response (Thank You for the Feedback) - Cont'd

(Q3) Why is Jensen–Shannon divergence preferred over KL divergence in Equation (2)? Could the authors provide an ablation comparing JS with forward and reverse KL in terms of detection TPR, FPR, and computational overhead?

We primarily chose Jensen–Shannon (JS) divergence because it is symmetric. Swapping the target and draft model’s distributions leaves JS divergence unchanged, whereas forward and reverse KL can yield different values for the same pair. This lets us quantify fault-induced discrepancies without choosing a KL direction. We will clarify this rationale in the final version.

We also evaluate whether this choice affects detection performance. We replace JS with forward or reverse KL in Equation (2) and measure TPR and FPR under fault-free and benign-fault conditions. The table below reports results on GSM8K for all three models. All rates are percentages.

ModelMetricTPRFPR (Fault-Free)FPR (Benign Fault)
Llama3max(NLL, JS) (Ours)97.70.00.0
max(NLL, KL forward)98.30.00.0
max(NLL, KL reverse)97.70.00.0
Falcon3max(NLL, JS) (Ours)97.50.40.6
max(NLL, KL forward)97.72.32.5
max(NLL, KL reverse)97.52.32.5
Qwen3max(NLL, JS) (Ours)98.40.00.2
max(NLL, KL forward)99.00.00.2
max(NLL, KL reverse)98.80.00.2

JS achieves comparable TPR across models and lowers FPR on Falcon3. TPRs range from 97.5–98.4% with JS and 97.5–99.0% across the two KL variants. On Falcon3, JS yields fault-free and benign-fault FPRs of 0.4% and 0.6%, respectively, compared with 2.3% and 2.5% for both KL variants. The FPRs are identical across all three metrics on Llama3 and Qwen3.

We also measure the time per forward pass with each metric integrated into CSMR. The table reports latency in milliseconds across three context lengths, averaged over 50 trials; ±\pm reports the standard deviation.

ModelContext Lengthmax(NLL, JS) (Ours)max(NLL, KL forward)max(NLL, KL reverse)
Llama325634.614 ± 0.03034.536 ± 0.02034.534 ± 0.021
51235.529 ± 0.01935.451 ± 0.01635.449 ± 0.017
102437.315 ± 0.02537.237 ± 0.02937.235 ± 0.026
Falcon325645.362 ± 0.01845.280 ± 0.02045.277 ± 0.021
51246.592 ± 0.01946.510 ± 0.01646.507 ± 0.016
102448.948 ± 0.01848.866 ± 0.01648.864 ± 0.017
Qwen3256136.280 ± 0.084136.206 ± 0.084136.204 ± 0.080
512138.684 ± 0.085138.611 ± 0.086138.609 ± 0.079
1024143.260 ± 0.086143.186 ± 0.078143.187 ± 0.082

Across all models and context lengths, JS adds 0.073–0.085 ms per forward pass relative to either KL variant, an increase of less than 0.24%.

Therefore, compared to either KL direction, JS divergence offers comparable TPR with improved FPR on certain models, while adding negligible latency. We will include these results in our final version.

(Q4) The references to "scikit-learn developers" (line 246), "SGLang Team," and "Appleby" (line 265) appear incomplete. Could the authors provide formal citations?

We appreciate the reviewer noticing some missing information in these references. We will fix them in our final version.

AuthorsOpenReview ↗

Rebuttal Response (Thank You for the Feedback)

(Q5) Figure 7 shows relatively low TPR for certain MMLU settings. Could this be related to generation length? Since the detector aggregates discrepancies over a 10-token window, shorter responses may provide fewer opportunities to detect cascading errors. Could the authors analyze TPR as a function of output length and window size?

We thank the reviewer for suggesting this insightful analysis. We first measure TPR as a function of output length by inspecting our experimental outputs from Sections 4.3 (faults in the target model only) and 4.7 (faults in the draft model only). We report the results for Llama3-8B on MMLU below.

Response Length (tokens)Target Model: # FaultsTarget Model: TPR (%)Draft Model: # FaultsDraft Model: TPR (%)
1--165394.31968.4
17--641392.33666.7
65--2561546.728866.3
257--102437099.72352.2

CSMR's detection rate does not depend on high output lengths. With faults in the target model, even responses shorter than the 10-token window are detected 94.3% of the time. The only large drop in detection rate is at 65–256 tokens, likely because faults resulted in coherent but incorrect responses; however, they account for just 15 total instances. With faults in the draft model only, TPR is roughly constant (~66%) at every length. The lower draft-only TPR on MMLU therefore does not come from short responses. We hypothesize it is instead because the clean target model corrects every token, so the faults do not cascade through the context and cause sufficient semantic divergence.

Regarding window size, our sensitivity analysis (Appendix G) shows that on GSM8K, TPR plateaus for w∈[0,25]w \in [0, 25], with w=10w = 10 offering the best TPR–FPR tradeoff across all evaluated models. We extend this analysis to MMLU and find that the same result holds: on Llama3, smaller windows reduce TPR (44.8% at w=2 vs. 97.1% at w=10), while larger windows give no gain and add false positives. We will include these results in our final version.


We hope our responses address the reviewer's questions and concerns. We would be happy to provide further clarification if needed.

Reviewer_rxsLOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the problem of detecting two-bit fault errors in memory when serving LLMs. The paper proposes CSMR, which utilizes a small model to check the output distribution of the target model. If the distribution's difference is too large based on some pre-collected offline statistics, the fault is detected. Then the tensors are hashed to identify the erroneous one and reloaded. The paper proposes to use the draft model of speculative decoding for the small detector model of CSMR, in which case little overhead will be added for this detection.

Critical strengths
  1. The paper studies an often-ignored problem of two (or more) -bit faults during GPU serving. The paper proposes a method that has very small overhead if applied together with a system that already has speculative decoding.
Critical weaknesses
  1. The speculative decoding setting is not realistic. The modern spec-decoding methods typically do not involve a separate LLM of the sizes as tested in the experiments, but utilize relatively small extra network structures to take in latents from the base model and generate the draft. To make the experiment more realistic, the paper should test on spec-decoding methods such as egale-3 and Dflash2. One extra problem they raise is that their drafters depend on the target’s hidden states, so target corruption may affect both sides of CSMR’s comparison. Dflash2 also raises a question about whether a DLM-based small model can effectively perform the detection.

  2. The error setting is not solid enough, or basically the importance of the target problem is not established well. It is probably worth providing the rate of the two- (or more-) bit faults per error on modern GPUs. Also, it should be discussed in detail whether NVidia already provides methods/hardware to perform similar or related functionalities. There has been some discussion and description in the current paper, but I think, according to the search, it seems that two-bit faults can be automatically detected.

  3. The latency results do not adequately isolate speculative decoding from CSMR, and there is no detailed profiling of draft computation, target verification, discrepancy computation, full-weight hashing, and regeneration.

  4. The paper states that discrepancy scores are computed at accepted speculative token positions. However, if the base model becomes corrupted, its predictions may systematically disagree with the draft model, causing repeated rejections. Are discrepancy scores also computed at rejected positions? If only accepted positions are monitored, could a severe fault paradoxically prevent the detector from receiving sufficient observations?

Questions for authors

Please address the weaknesses.

5 comments
AuthorsOpenReview ↗

Rebuttal Response (Thank You for the Feedback)

We thank the reviewer for recognizing the importance of addressing multi-bit memory faults in LLM inference and CSMR's low-overhead integration with speculative decoding. Below, we address the concerns and provide additional experimental evidence and clarifications.

(W1) The modern spec-decoding methods typically do not involve a separate LLM of the sizes as tested in the experiments, but utilize relatively small extra network structures to take in latents from the base model and generate the draft. To make the experiment more realistic, the paper should test on spec-decoding methods such as egale-3 and Dflash2.

To address this concern, we evaluate CSMR when integrated into EAGLE3 and Dflash2. For EAGLE3, we use a public checkpoint [1] of a draft model for Qwen3-32B, while for Dflash2 we use the official draft model for Qwen3-4B. The EAGLE3 draft model for Qwen3-32B has only a subset of the target models vocabulary, so we restrict decoding to only generate tokens in both models’ vocabularies. For DFlash2, we use only the JS divergence because the draft model assigns near-zero NLL to most clean tokens, so the NLL's interquartile range is too small for the robust normalization to be stable. For both model pairs, we re-calibrate CSMR’s hyperparameters and sample 100 benign and critical faults on GSM8K for the target models. We report the results below.

Target ModelSpeculative Decoding MethodTPRFPR (Fault-Free)FPR (Benign)
Qwen3-32BEAGLE394.6%0%0%
Qwen3-4BDFlash298.0%1.0%1.0%

CSMR remains effective when paired with modern speculative decoding methods. Across methods, TPR is 94.6–98.0%, comparable to the 95.6–99.6% in our main evaluation, with 1% FPR under benign faults and fault-free execution. We will include these results in our revision.

[1] https://huggingface.co/RedHatAI/Qwen3-32B-speculator.eagle3

(W2) The error setting is not solid enough, or basically the importance of the target problem is not established well. It is probably worth providing the rate of the two- (or more-) bit faults per error on modern GPUs. Also, it should be discussed in detail whether NVidia already provides methods/hardware to perform similar or related functionalities. There has been some discussion and description in the current paper, but I think, according to the search, it seems that two-bit faults can be automatically detected.

(1) Rate of two- or more bit faults per error on modern GPUs.

To our knowledge, no prior studies quantified the fraction of all memory errors that involve two- or more bit faults. However, prior studies have quantified the rate of hardware-detected double-bit memory errors on large GPU systems and found them to be significant. Oles et al. [1] report a system-wide mean time of 70.7 hours between such errors on the Summit supercomputer’s V100 fleet. More recently, Cui et al. [2] report 17 double-bit memory errors on Delta’s H100 fleet, with a system-wide mean interval of 206 hours. These findings establish that double-bit memory errors occur in production, including on recent GPUs. We will discuss these measurements in the final version.

AuthorsOpenReview ↗

Rebuttal Response (Thank You for the Feedback) - Cont'd

(2) Does NVIDIA already provide methods/hardware to perform similar or related functionalities to CSMR? It seems that two-bit faults can be automatically detected.

We agree that NVIDIA’s SECDED ECC automatically detects two-bit errors within a single protected codeword. When an uncorrectable HBM error is detected and contained, NVIDIA’s driver terminates the affected application [3]. CSMR complements ECC in cases where inference instead continues with corrupted weights. Prior work has observed silent corruption in GPU applications with ECC enabled [4], and Zheng et al. [5] documented logic-level corruption that standard ECC did not catch in production LLM training. If corruption persists in model weights, CSMR can detect its effect on generation and trigger recovery (even if it affects more than 2 bits, as we show in Section 4.6). CSMR can also provide this protection when system-level ECC is disabled, which is a realistic deployment setting; for example, Runpod documents serving Llama with vLLM on RTX 4090 GPUs [6], a consumer GPU model for which system-level ECC is reported to be off by default [7].

In our experiments, we inject faults directly into the model weights to represent persistent corruption at the application level. This lets us measure the protection CSMR adds when ECC misses the corruption or is disabled. We will clarify how CSMR complements hardware protection mechanisms such as ECC in the final version.

[1] Oles et al., “Understanding GPU Memory Corruption at Extreme Scale: The Summit Case Study,” ICS (2024).

[2] Cui et al., “Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUs,” SC (2025).

[3] https://docs.nvidia.com/deploy/a100-gpu-mem-error-mgmt/latest/response-to-uncorrectable-contained-ecc-errors.html

[4] Fernandes dos Santos et al., “Experimental Findings on the Sources of Detected Unrecoverable Errors in GPUs,” TNS (2022).

[5] Zheng et al., “SDCs in the Wild: Characterizing and Diagnosing SDC-defective GPUs in Production LLM Training,” OSDI (2026).

[6] https://www.runpod.io/models/meta-llama-llama-3-2-3b-instruct

[7] https://support.chaos.com/hc/en-us/articles/24886637113617-ECC-State-for-Nvidia-GPUs-Error-Correcting-Code

AuthorsOpenReview ↗

Rebuttal Response (Thank You for the Feedback) - Cont'd

(W3) The latency results do not adequately isolate speculative decoding from CSMR, and there is no detailed profiling of draft computation, target verification, discrepancy computation, full-weight hashing, and regeneration.

To better isolate CSMR’s runtime overhead, we run a detailed profiling of each step of inference specified by the reviewer. First, we re-run our fault-free latency experiments, this time measuring the time of speculative decoding (including draft computation and target verification) and CSMR’s discrepancy check. The table below reports the milliseconds per generated token, averaged over the same 100 prompts used in Section 4.4.

ModelTarget onlySpeculative Decoding (SD)CSMR on top of SDSD + CSMR Speed-Up
Llama333.0425.97+ 0.191.27$\times$
Falcon343.2128.43+ 0.171.51$\times$
Qwen3131.9074.83+ 0.191.76$\times$

CSMR adds just 0.17–0.19 ms over speculative decoding, remaining roughly constant as model size increases. Relative to inference without speculative decoding, speculative decoding with CSMR active results in a 1.27–1.76$\times$ speedup.

Second, we measure the time of recovery under detected faults. For each critical fault used in Section 4.4 that CSMR detects, we measure the time spent (a) generating tokens prior to detection, (b) hashing every weight, (c) reloading affected tensors, and (d) regenerating the complete, uncorrupted output. The table below shows the time in seconds of each step, averaged over the same prompts.

ModelInitial GenerationHashingReloadingRegenerationTotal (Fault-Free)
Llama30.720.070.103.494.38 (3.49)
Falcon30.830.090.103.944.96 (3.94)
Qwen32.600.220.2118.8821.90 (18.88)

CSMR’s hashing and reloading of affected weight tensors makes up just 0.17–0.43 seconds of the total recovery, with most going to the initial generation prior to detection (0.72–2.60) and the regeneration of the response (3.49–18.88). Compared to fault-free inference with CSMR active, the total time to regenerate a response under detected faults adds just $\sim$1–3 seconds. We will include these results in the final version.

(W4) The paper states that discrepancy scores are computed at accepted speculative token positions. However, if the base model becomes corrupted, its predictions may systematically disagree with the draft model, causing repeated rejections. Are discrepancy scores also computed at rejected positions? If only accepted positions are monitored, could a severe fault paradoxically prevent the detector from receiving sufficient observations?

We clarify that by “accepted token positions,” we refer to the positions of all tokens emitted by a particular speculative decoding step. When a step rejects a draft token, it also generates the base model’s token at the same position, and CSMR computes a discrepancy score there. Thus, even if the verifier rejects the first draft token in every step of a generation, CSMR still obtains one discrepancy score per step. We will clarify this terminology in the final version.


We hope our responses address the reviewer's questions and concerns. We would be happy to provide further clarification if needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.