acceptodds
Under review as a conference paper at ICLR 2027

When Space Flips a Bit: Recovering Transformer Inference from Radiation-Induced Faults

Abstract

As transformer models move into space, radiation-induced faults threaten inference reliability by silently corrupting hidden states. Existing defenses offer limited fault coverage, risk suppressing legitimate rare computations, or require hardware modifications that are impractical for commercial off-the-shelf accelerators. We introduce RIFT (Recovery via Inference-time Fault Tolerance), a five-stage pipeline that detects and corrects non-repeating transient corruption without retraining, architectural changes, or a clean reference model. RIFT screens hidden states using spectral, temporal, and cross-layer residuals, then re-executes flagged layers to distinguish corruption from legitimate novelty. It recovers corrupted states through uncertainty-aware spectral correction and gated consistency fusion, and adjusts output distributions according to estimated fault severity. Across language, vision-language, and vision-language-action transformers subjected to single-event upsets, multi-bit upsets, single-event transients, and bus corruption, RIFT approaches the task quality of triple modular redundancy. Under single-event upsets, it achieves WikiText-2 perplexity of 6.21 versus 6.15 and LIBERO task success of 76.8% versus 78.1%, with 5.5% clean-run overhead on LLaMA-3-8B and 95.7–97.1% novelty preservation across the three evaluated models. We further evaluate RIFT on a compact decoder running aboard a satellite in low Earth orbit. Across 400 controlled fault injections, RIFT achieves 91.5% fault detection and 91.7% recovered-token agreement with fault-free execution, with runtime overheads of 8.4% during clean execution and 12.6% under injected faults. This orbital experiment demonstrates recovery from controlled faults on hardware operating in space, while recovery from naturally occurring radiation-induced faults remains unverified. Together, these results support inference-time fault recovery that preserves legitimate model behavior across transformer workloads and extends to orbital computing.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Inference-time fault tolerance, radiation-induced faults, silent data corruption, transformer models, space-based artificial intelligence, hidden-state recovery, vision-language-action models.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BQTmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RIFT, a software-based technique to protect Transformer inference from radiation-induced transient hardware faults in space environments. RIFT is a 5-stage pipeline applied to pre-trained models without requiring retraining or architectural changes.

In Stage 1, hidden states of each layer are screened via spectral, temporal, and cross-layer residual signals, and Stage 2 re-executes flagged layers to assess replay consistency through a thresholded functional coherence gate. If the replay discrepancy exceeds the threshold, the pipeline proceeds to Stages 3–4, which recover the hidden state through uncertainty-aware spectral shrinkage, replay blending, and gated temporal and cross-layer fusion, while Stage 5 adjusts the output distribution's temperature based on the estimated fault severity.

In software fault injection experiments across four fault types (SEU, MBU, SET, and bus corruption) on LLaMA-3-8B, Qwen2.5-VL-7B, and OpenVLA-7B, RIFT demonstrates higher output quality than existing baselines and achieves performance close to TMR. On LLaMA-3-8B, the runtime overhead is reported as 5.5% in a fault-free environment and 8.2% under SEU faults. The authors also conduct controlled fault injection experiments with a small model on a satellite-mounted GPU in Low Earth Orbit (LEO).

Critical strengths

The evaluation scope is comprehensive. The authors evaluate RIFT across three model categories (LLM, VLM, and VLA) and four fault types (SEU, MBU, SET, and bus corruption), comparing it against six baselines as well as TMR. They also present results across model scales ranging from 3B to 72B and on additional models. Furthermore, the paper includes additional experiments on sustained fault bursts, compound faults, nonuniform fault placement, adaptation under domain shifts, and statistical stability over long contexts.

Critical weaknesses

The recovery mechanism in Stages 3–4 is not sufficiently justified. When the original execution and replay disagree, the discrepancy alone does not reveal which execution was corrupted. RIFT instead performs approximate recovery by blending the original layer output, the re-executed output, the previous corrected hidden state, and a linear cross-layer prediction. Therefore, the potentially corrupted original output is mixed back into the final state. However, a simpler alternative could be considered: Selective TMR, which triggers a third execution only when the replay discrepancy exceeds the coherence threshold and selects an output through majority voting based on numerical agreement. The paper reports that the re-execution rate during clean execution is approximately 5%. If, for instance, only 10% of these cases result in a discrepancy exceeding the threshold (p_corrupt = 0.5%), the additional layer-forward computation required for the third execution would be only about 0.5% overall. In this case, the total additional layer-forward compute needed for both the second and third executions would be approximately 5.5% of the original layer-forward computation.

Furthermore, under the paper's assumptions of approximately deterministic clean execution and independent transient faults, majority voting across three runs could identify a correct output when at least two executions are fault-free and agree within numerical tolerance. While the paper provides ablation studies removing individual components of Stages 3–4, it lacks a comparison against Selective TMR. Moreover, the p_corrupt value in the main experimental setup is not reported. Consequently, it is difficult to assess whether RIFT's recovery (Blend) in Stages 3–4 provides meaningful advantages over simple selective re-execution in terms of recovery accuracy or execution cost.

Questions for authors
  1. What are the empirically measured values of p_flag and p_corrupt under fault-free and fault-injected conditions? Specifically, what proportion of re-executed layer-steps proceed to Stages 3–4? How does RIFT compare against Selective TMR in terms of recovery accuracy and end-to-end latency? This baseline would use exactly the same Stage 1 and Stage 2 as RIFT, but replace Stages 3–4 with a third execution and majority voting whenever the Stage 2 coherence gate detects a discrepancy.

  2. Is it possible to evaluate the recovery quality strictly on cases where Stages 3–4 are actually triggered? Instead of using the re-executed result directly, RIFT recovers the state by blending the potentially corrupted original output, the re-executed output, the previous hidden state, and the cross-layer prediction. This approximate recovery may leave residual errors compared to a fault-free hidden state. If actual recovery occurs in only a small fraction of all layer-steps, these recovery errors might be masked by aggregate task metrics. It would be helpful to separately report the hidden-state reconstruction error or subsequent token agreement specifically for cases where recovery was performed. Additionally, how does RIFT compare against simpler baselines that use the re-executed result as-is, or rely solely on the previous token state or inter-layer prediction, rather than blending multiple states?

Reviewer_Z1THOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses silent data corruption in transformer inference on commercial off-the-shelf accelerators in space. RIFT is a wrapper that needs no retraining. Spectral, temporal and cross-layer residuals screen hidden states. Flagged layers are re-executed, and a thresholded gate fires when the replay disagrees with the original, which separates transient faults from legitimate novelty. Confirmed states are shrunk in the covariance eigenbasis, blended with the replay, and fused with temporal and cross-layer predictions. Output temperature then rises with estimated severity. Faults are injected in software as SEU, MBU, SET and bus corruption on LLaMA-3-8B, Qwen2.5-VL-7B and OpenVLA-7B. Under these faults the authors claim:

  • quality close to triple modular redundancy (perplexity 6.21 vs 6.15);
  • 5.5% clean-run overhead;
  • 95.7–97.1% novelty preservation;
  • recovery of 400 injected faults on a Jetson Orin NX in low Earth orbit.
Critical strengths
  1. The paper frames recovery as preserving rare but correct computation, with a dedicated novelty-preservation benchmark (Table 2, A4.3). This is a useful advance over range restriction.
  2. Replay gating is simple, model-agnostic and needs no retraining.
  3. The evaluation is broad: three modalities, four fault types, models from 3B to 72B (Table 7, A9), bursts, compound faults and sensitivity sweeps (Figs. 2, 6–8).
  4. Limitations are stated candidly (sec. 5, A1, A4.4), and A7.3 directly probes the gate's failure boundary.
  5. On-orbit execution with telemetry is rare in this literature.
Critical weaknesses
  1. In the main experiments, faults hit only hidden states and never recur on replay. This is exactly what the gate detects by construction. Persistent upsets in weights and the KV cache, which have the largest memory cross-section, are excluded. A7.3 shows FDR drops to 72.2% at ρ_persist=0.25. Please evaluate weight and KV-cache SEUs, or combine RIFT with memory checks and report joint coverage.

  2. A key baseline is missing: screening plus replay, then adopting the replay output or a third-run vote at flagged layers. With independent faults this is near-exact and cheap. Without it, the value of Stages 3–4, which only yield an approximate state, is unclear. Also, Stage 5 lowers perplexity without changing greedy outputs, so Table 1 perplexity partly reflects calibration.

  3. About 5% of clean layer-steps are flagged (A3), so nearly every sequence should contain a flag. Yet the "screening only" and "no gate" variants report NPR of 82.7% and 71.4% (Table 5). Gate activations on clean runs (NPR below 100%) also contradict the deterministic-replay assumption. On orbit, a "clean gate firing" of 6.2% seems to conflict with the sequence-level NPR of 93.8%.

  4. Updating U_l,k and the full spectrum (Eq. 10), plus RLS on a d×d B_l, at every step seems incompatible with 5.5% overhead. Rank-64 factors also cannot supply k=1024 directions (A5.5). Four baselines are reimplemented and run outside the fault models they were designed for; for example, SAVE targets memory flips.

  5. The orbit study uses a 0.31M-parameter model, software injection and no ground replica. It shows deployment feasibility, not radiation robustness, which the title and abstract overstate. The abstract also names the provider despite the stated anonymisation.

  6. Theorem 1 restates the threshold rule, and Theorem 2 is standard Wiener shrinkage. FDR should also be broken down by bit position and SDC impact.

  7. Related work is missing: SWIFT (Reis et al., CGO'05); GPU instruction replication (Mahmoud et al., SC'18); activation-based error detection (Schorn et al., SAFECOMP'18); GPU beam tests of DNNs (dos Santos et al., IEEE TR'19).

Questions for authors
  1. How is sequence-level NPR computed for the "screening only" and "no gate" variants? What is the mean number of flags per sequence?
  2. What causes gate activations on clean runs? What is the measured d_fp?
  3. What is the perplexity when flagged states simply adopt ȟ, or when a third-run vote is used?
  4. How, and how often, are U_l,k, the eigenvalues and the RLS inverse updated? What is the timing breakdown of the 5.5%?
  5. What are the greedy-output metrics for Table 1 without Stage 5?
  6. How were the baselines adapted to hidden-state faults?
  7. Are there ground-replica results for the Jetson experiment?
Reviewer_3ejAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an inference-time software framework called RIFT that is designed to detect and recover from instantaneous bit errors from transient faults, such as SEU, MBU, SET, bus corruption in the hidden states of Transformer-based models without retraining or redesigning the model’s architecture. The authors argue that existing methods like TMR are computationally expensive, incurring nearly 200% overhead while other methods like activation clipping mistakenly eliminate legitimate semantic novelty. To solve these problems, the paper proposes a functional coherence gate mechanism that only when a state is flagged, as statistically abnormal is the layer re-executed. If the replay result does not match the initial run, a hardware error is confirmed to trigger spectral shrinkage-based recovery and consistent fusion. The evaluation is made on three model types under four software-injected error modes with model size range from 3B to 72B parameters. The reported results showed a perplexity of 6.21 on WikiText-2 compared to 6.15 for TMR under SEU mode, with an overhead of 5.5% on clean runs and 8.2% when errors occurred on LLaMA-3-8B. The study performed a real-world test with a low-orbit satellite using a 0.31M decoder model. Additionally, natural radiation errors were not evaluated.

Critical strengths
  1. The core idea is algorithmically logical, the authors separated the statistical screening step from the functional validation step by repeating the layer computation, therefore, a hidden state is only corrected if the subsequent run yields different results, not simply because it is statistically abnormal. Additionally, the method can be directly dropped into pretrained models. The five stages are designed through a single threshold gate, state consistency is maintained while recovery increases progressively with the deviation between the original and re-runs. The authors also presents quite comprehensively the assumptions and limitations of the method.

  2. The authors evaluates three model types LLM, VLM and VLA under four failure modes SEU, MBU, SET and bus corruption at three severity levels, compared to six baselines and TMR. The article also has extensed results from 3B to 72B models, ablation for each stage, and measuring semantic preservation in addition to aggregate metrics. Implementation of the pipeline on the Jetson Orin NX in low orbit demonstrates the method's ability to run on real hardware.

Critical weaknesses
  1. The proposed method is a complex pipeline, but the authors have not demonstrated its superiority over simpler baselines. The paper assumes that errors are instantaneous, meaning that the first run of the layer has a radiation error leading to incorrect results. When the filter detects a suspected error, the system must re-execute the same layer on the same clean input. If this run is free of radiation errors, the output is correct. The paper lacks evidence demonstrating the advantage of adding Stage 3 (Spectral shrinkage) and Stage 4 (Consistency Fusion) steps over directly using the replay results for subsequent inference.

  2. The theory of screening relies on k = 1024 eigenvectors, but the compression implementation uses only a covariance matrix of rank r = 64 as in Appendix A5.5. A matrix of rank 64 cannot generate 1024 directions with independent variances, leaving the remaining 960 directions undefined and the actual cost unverifiable at 5.5% overhead. Furthermore, at an error level of BER = 1e-4, the probability of a layer-step having a bit flip is 99.86%. To achieve a detection rate of 98.4% as shown in Table 2, the system would have to rerun almost 100% of the steps, meaning the compute time would double. The actual overhead reported at only 8.2% in Table 9 is mathematically impossible. Additionally, the study only injects errors into the output activation and ignores errors in Weights and KV-cache, which account for most of the actual exposure.

Questions for authors
  1. In the RIFT process, every time a state is flagged, the system incurs the full computational cost of re-executing that layer. If the replay is clean, why doesn't the system directly use the replay result to continue the inference? The authors should provide comparative evidence to determine whether adding Stage 3 (Spectral shrinkage) and Stage 4 (Consistency Fusion) deliver superior added value compared to a simpler baseline.

  2. According to the cost calculation formula as in Equation 15 in the paper, replay alone increases the computational load of the forward pass layer by nearly 98%. Could the authors please explain why the actual runtime overhead measured in this configuration is only 8.2%?

  3. According to the proposed method, when radiation flips bits in the weight memory or KV cache, re-running the layer will reread those corrupted bits and produce the exact same erroneous result as the first run, causing the inspection gate to mistakenly believe this is a valid novel state and include it in the background statistical updates. So, does RIFT have a mechanism to identify these recurring errors and prevent statistical poisoning? Also, if the KV cache is corrupted at a decoding step, which record will the system retain, and how will this impact the accuracy of all subsequent token generation steps?

  4. For tasks using greedy decoding such as OpenVLA on LIBERO and Qwen2.5-VL on MMBench, dividing logits by a positive temperature preserves the argmax, so could the authors confirm how Stage 5 contributes changes to the actual output of VLM/VLA?

  5. Regarding the discrepancy between the 1024-dimensional screening spectrum and the 64-rank approximation, could the authors clarify how the system identified, constructed, and calculated the remaining 960 basis directions, and how the computational cost and update cycle of all these directions are specified in the pipeline to maintain a very low overhead of 5.5% on clean runs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.