- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SIEVE, a system to detect the rare GPU soft errors that change what an LLM's answer means, which numerical thresholds handle poorly. It predicts each monitored layer's features from the layer before, then feeds the prediction errors to an XGBoost classifier. Across three vision-language models and three VQA benchmarks, it reaches 93.78% F1 with 6.65% added latency, beating Ranger and Dr.DNA by about 10–11 F1 points.
Critical strengths
- The motivation is good: numerical deviation and semantic deviation may not be strongly correlated, and previous methods that rely only on numerical deviation fail.
- The method consistently outperforms the baselines.
Critical weaknesses
- The detector’s training data comes from fault injection, and the fault distribution after deployment may differ from the injected one.
- The semantic labels come from a small 7B model, which could be unreliable.
- The dataset is skewed: most executions contain errors. This may not be realistic and could inflate precision, recall, and F1.
Questions for authors
- Would larger models be more fault-tolerant, for example with a much lower fraction of Significant SDCs?
- Would the algorithm’s performance drop on larger models, either because Significant SDCs are rarer or because the XGBoost input grows larger and noisier?
- How would the detector hold up if the deployment fault distribution differs from the injected one?
- Is there a specific reason not to use a SOTA LLM as the judge to generate semantic labels?
- What is the probability of a dual-bit fault in the experimental setup, and is it based on real statistics?
5 comments
Response to Question 1
Given our time and GPU constraints, we compared InternVL3-8B and InternVL3-14B, as shown in Table R4. Each model received 15,000 valid injections. For 8B, the proportions of SDCs and Significant SDCs were 5.25% and 3.89% (583 / 14,970), respectively. The corresponding proportions for 14B were 4.51% and 3.45% (517 / 14,973). For this task and injection setup, the proportion of Significant SDCs for 14B was only 0.44% lower.
Table R4. Mixed-injection results for InternVL3-8B and InternVL3-14B on LingoQA.
| Model | SDC proportion | Significant SDC proportion | Precision | Recall | F1 | False positive rate |
|---|---|---|---|---|---|---|
| InternVL3-8B | 5.25% (786 / 14,970) | 3.89% (583 / 14,970) | 70.65% | 82.28% | 76.02% | 1.25% |
| InternVL3-14B | 4.51% (676 / 14,973) | 3.45% (517 / 14,973) | 75.58% | 81.25% | 78.31% | 0.97% |
Response to Question 2
The language decoder has 48 layers in InternVL3-14B and 28 layers in the 8B variant. For both models, we project the activations to 64 dimensions, monitor three layer pairs, and extract 12 summary statistics per pair. The XGBoost input therefore has 36 dimensions in both cases. With a fixed number of layer pairs and the same summary statistics, the XGBoost input dimension stays fixed even if the activation projection dimension increases.
As shown in Table R5, on the mixed-injection test subset for LingoQA, 14B achieved an F1 of 78.31%, 2.29% higher than the 8B score of 76.02%. Precision increased from 70.65% to 75.58%, the false positive rate decreased from 1.25% to 0.97%, and recall decreased slightly from 82.28% to 81.25%. With the same detector input dimension, SIEVE achieved comparable recall and higher F1 on the 14B model than on 8B in this comparison.
Response to Question 5
We did not set a fault rate in the experiment. Faults are rare in practice, and running the experiment for several days at a real-world fault rate would yield too few fault samples. We therefore inject one random double-bit flip per inference run in the fault-injection group, while the fault-free group receives no injection. After 55,000 valid injections, Significant SDCs accounted for 3.65% of the samples.
Thank you for the response. I have two follow-up questions.
Weakness 1 / Question 3: What if double-bit faults occur systematically rather than randomly, for example repeatedly in the same memory region or bit position? Would this affect the detector’s performance, given that the XGBoost classifier is trained on randomly injected faults?
Question 1: In general, are larger models more robust to soft errors, so that SDCs affecting the answer’s meaning occur less often? For example, would a model with 500B or 1T parameters be robust enough that such detection becomes unnecessary? Some discussion of how SIEVE’s value scales with model size would help clarify its practical relevance.
Thank you for recognizing the motivation of our work and for raising these questions. We have addressed them below and hope the results help resolve your concerns.
Response to Weakness 1 / Question 3
Data distribution is a general concern, and the space of possible random faults is particularly large in LLMs. Under our double-bit fault model, we randomly select the injection location and the bits to flip. For a 16-bit tensor, a single element has (216)=120 distinct bit pairs. A tensor with 10,000 elements therefore has 1,200,000 possible injections based on the element location and bit pair alone. We consider the results of our large-scale random fault injection experiments statistically informative and representative of detection performance across different fault distributions.
We also ran experiments that change the distribution of fault injection locations. We trained separate detectors using faults injected into the Attention or MLP modules, froze each detector and its threshold, and evaluated it on the test subset for the other module. As shown in Table R3, when the training fault source is switched from Attention to MLP, mean F1 on the Attention test subset is 88.90% and the false positive rate is 0.206%. On the MLP test subset, both detectors identify 16 out of 19 Significant SDCs. Compared with the detector trained on MLP faults, the detector trained on Attention faults achieves a higher mean F1 and a lower false positive rate. SIEVE thus retains a high detection rate for Significant SDCs under the two evaluated shifts in fault location.
Table R3. Cross-module detection under random double-bit injection. Results are for the prefill test subsets and are averaged equally across the nine tasks.
Response to Weakness 2 / Question 4
The judge applies a fixed rubric to two responses to the same input and determines whether the core answer has changed semantically. Given the requirements and scale of this annotation task, we selected Prometheus-7B v2.0. At the time, it was a state-of-the-art judge specifically trained to apply custom scoring rubrics. Its 7B size makes large-scale local annotation practical and lets us use a fixed model version and scoring prompt throughout the process. We also manually spot-checked a subset of the labels to verify that the judge's decisions followed the rubric.
Response to Weakness 3
As shown in Table A1 in the appendix, 90.90% of valid executions were subjected to fault injection, and 83.59% of those injections did not lead to an SDC. Significant SDCs accounted for 3.65% of the original test set (2,720 / 74,586). Although most executions received an injected fault, only a small fraction resulted in an SDC. SIEVE thus achieved high recall at a low prevalence of Significant SDCs without overstating recall.