- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes shifting the criteria for LLM inference accuracy from numerical correctness to semantic-level correctness. It calculates the per-token discrepancy scores between the target and draft models (taking the maximum of the normalized NLL and JS divergence scores) over a sliding window. If this value exceeds a predefined threshold, SDC is suspected. Upon detection, GPU-native hashing is used to identify corrupted weight tensors, which are selectively reloaded, and the response is regenerated. By integrating detection into the verification phase of speculative decoding, the authors minimize overhead. The paper reports that accuracy recovers to 96.9–99.4% under critical fault conditions, while latency overhead remains as low as 0.1–0.4% under fault-free and benign fault conditions.
Critical strengths
-
The motivational assumptions are clear. Unfaulted target and draft models should produce reasonably aligned output distributions, and the draft model already present in a speculative decoding can be repurposed for fault detection. In environments where speculative decoding is already utilized, SDC detection can be performed with minimal additional overhead, which serves as a key strength.
-
Because faults are detected based on differences in output distributions rather than direct weight integrity checks, the mechanism can detect not only weight memory corruption but also activation corruption and computational faults to some extent. Providing this expanded fault coverage without dedicated additional protection mechanisms is a clear advantage.
Critical weaknesses
-
CSMR mainly consists of a discrepancy-based trigger and a hash-based selective recovery module. The primary protected target appears to be weight memory errors. However, during selective recovery, the system hashes all weight tensors on the GPU after detection to identify corrupted ones. A natural alternative would be to perform this hash check periodically without relying on a discrepancy-based trigger. For instance, by distributing checks across iterations (e.g., hashing one layer per iteration) or adjusting the inspection interval to trade off overhead against detection latency. To justify the necessity of a discrepancy-based trigger, a direct comparison against this periodic baseline is required.
-
This paper claims that CSMR is compatible with any draft–verify scheme that uses a smaller draft model without modification. However, modern speculative decoding methods such as EAGLE-3 and DSpark feed the target model's hidden states into the draft model to generate tokens, rendering CSMR's detection effectiveness under these methods uncertain. Under these architectures, hidden features corrupted by weight errors in the target model may be propagated to the draft model. Consequently, the prediction errors of the two models may become correlated, potentially preventing the distribution discrepancy from exceeding the detection threshold even when the target model produces incorrect outputs. Note that the experiment in Section 4.7 (both models setting) injects independent faults into both models separately, which does not simulate the scenario where target corruption propagates to the draft model through hidden states.
Questions for authors
-
CSMR's selective recovery hashes all weight tensors post-detection to identify corrupted ones. How does the discrepancy-based trigger perform compared to a baseline that performs the same hash checks periodically without the trigger (e.g., checking one layer per iteration or tuning the interval to match CSMR's 0.1–0.4% latency overhead)? While periodic hashing can catch all persistent weight corruptions regardless of whether they are benign or critical, it leaves an exposure window where corrupted outputs might be served between checks. Please provide a comparison of both approaches in terms of the trade-off between overhead and exposure window size.
-
Does the true positive rate (TPR) hold up under draft models that rely on target hidden states, such as EAGLE-3?
-
In Section 4.5, the draft model's memory footprint is calculated as additional overhead. However, if the target scenario assumes an environment already running speculative decoding—where SDC detection is essentially provided "for free"—the draft model memory should be excluded from the incremental overhead attributable to CSMR. Conversely, if applied to an environment without pre-existing speculative decoding, the draft model memory is indeed an overhead, but the latency acceleration gained from speculative decoding must also be factored in, making it inaccurate to present it purely as an overhead. The former interpretation appears more appropriate. Clarifying the intended deployment scenario and presenting overheads against a consistent baseline would improve clarity. The current text leaves room for ambiguity on this point.
2 comments
Rebuttal Response (Thank You for the Feedback) - Cont'd
(Q3) Clarifying the intended deployment scenario and presenting overheads against a consistent baseline would improve clarity.
We clarify that CSMR is intended for deployments that already use speculative decoding. We discuss the draft model’s memory footprint in Section 4.5 for deployments that would need to enable speculative decoding. We will clarify this distinction in the final version and state explicitly that the draft model’s memory footprint is excluded from CSMR’s incremental overhead.
We hope our responses address the reviewer's questions and concerns. We would be happy to provide further clarification if needed.
Rebuttal Response (Thank You for the Feedback)
We thank the reviewer for recognizing CSMR's key strengths (its low-overhead integration of SDC detection into speculative decoding and its potential to detect diverse types of hardware faults beyond weight corruption). Below, we address the concerns and questions.
We appreciate the opportunity to address this concern. We implement and evaluate a periodic hashing baseline that hashes all weights every X decoding steps using CSMR's hashing methodology. For a fair comparison, all hashes stay on the GPU and are compared without synchronization, mirroring CSMR's detection.
(1) To match CSMR's overhead, periodic hashing requires a 43–100$\times$ larger exposure window. Following our setup in Section 4.4, we measure the fault-free latency of periodic hashing on 100 prompts for X∈{1,2,4,8,16}. Because hashing's overhead scales roughly with 1/X, we determine the interval required to match CSMR's overhead by computing X∗=O1/OCSMR, where O1 is the overhead of hashing at every decoding step and OCSMR is CSMR's overhead. As X∗ is measured in speculative decoding steps, we convert it to tokens using the measured average number of tokens accepted per decoding step (3.7–4.4). We report the results averaged across 5 trials below.
At short intervals (1–8), periodic hashing increases latency by 3.98–41.51%, compared to 0.15–0.36% for CSMR. To match CSMR's overhead, periodic hashing would need an interval of 430–1,001 tokens and it grows with model size. This is equivalent to 3–4 complete average responses, so corrupted responses would routinely be generated in full before any check occurs.
(2) Checking one layer per step also yields higher overhead and a larger exposure window than CSMR. Because hashing's cost scales linearly with the number of parameters hashed, we estimate from our X=1 results that checking one layer per step incurs 0.54–1.29% overhead with an exposure window of 120–279 tokens (one full sweep over all layers). In contrast, CSMR operates at 0.15–0.36% overhead with a 10-token detection window.
Therefore, while periodic hashing eventually catches all persistent corruptions, it faces a worse trade-off between the exposure window and overhead. It also detects benign faults, which are costly to recover from and CSMR is intentionally designed to avoid. We will include this comparison in our final version.
To address this concern, we evaluate CSMR when integrated into EAGLE3 and Dflash2. For EAGLE3, we use a public checkpoint [1] of a draft model for Qwen3-32B, while for Dflash2 we use the official draft model for Qwen3-4B. The EAGLE3 draft model for Qwen3-32B has only a subset of the target models vocabulary, so we restrict decoding to only generate tokens in both models’ vocabularies. For DFlash2, we use only the JS divergence because the draft model assigns near-zero NLL to most clean tokens, so the NLL's interquartile range is too small for the robust normalization to be stable. For both model pairs, we re-calibrate CSMR’s hyperparameters and sample 100 benign and critical faults on GSM8K for the target models. We report the results below.
CSMR remains effective when paired with modern speculative decoding methods. Across methods, TPR is 94.6–98.0%, comparable to the 95.6–99.6% in our main evaluation, with 1% FPR under benign faults and fault-free execution. We will include these results in our revision.
[1] https://huggingface.co/RedHatAI/Qwen3-32B-speculator.eagle3