acceptodds
Under review as a conference paper at ICLR 2027

Blackout vs. Freeze: Analyzing Physical Failure Modes of VLAs under Camera Faults

Abstract

Unreliable visual inputs can harm task performance and cause potential physical safety risks for vision-language-action (VLA) models. We analyze how and GR00T models act under input faults such as image blackouts and freezing. We find that blackout and freezing produce distinct physical failure modes even when task-success rates are similarly low: freezing causes more extreme joint behavior, whereas blackout after gripper closure can cause more object drops, most markedly without proprioception. Selective intervention studies reveal that proprioception (current robot state) partly compensates for the removed robot depictions and reduces non-target contact. However, it cannot sufficiently restore task success when wrist-view object information is removed, even when aided by the remaining scene view. We then evaluate two mitigation approaches: camera-blackout training and training-free replacement of faulty visual embeddings. Both improve task success in selected conditions, but can increase unintended contact or disturbance to surrounding objects. Real-robot trials further show that successful execution under camera faults can still involve unintended physical interactions. These findings motivate designing VLA policies that use the robot and object information still available under camera faults to limit hazardous motion.

Primary area:
interpretability and explainable AI
Keywords:
Vision-Language-Action Models, Safety, Model Interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AmuzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the robustness of Vision-Language-Action (VLA) models under spatial object perturbations, where policies may continue reaching toward an object's original position after it has been displaced. The authors use activation patching to examine how object-position information propagates through visual tokens and attention heads, and propose a training-free attention-head amplification method to strengthen the contribution of displacement-sensitive heads to action prediction. The method identifies relevant attention heads through counterfactual interventions and amplifies their input-dependent deviations without modifying model parameters.

Critical strengths
  1. The paper investigates the internal mechanisms underlying spatial robustness failures in VLA models. Through activation patching and attention-head interventions, the authors examine whether object-position information is absent from model representations or insufficiently utilized during action prediction. This analysis provides empirical evidence about how spatial information influences VLA behavior.
  2. The paper evaluates the proposed intervention across multiple VLA architectures. The experiments cover five VLA policies with different action-generation mechanisms and include paired closed-loop evaluations, attention-head controls, and transfer experiments on LIBERO-Pro. These comparisons help characterize the effectiveness and limitations of the proposed method.
Critical weaknesses
  1. The proposed method primarily adapts established activation patching and attention-head intervention techniques to VLA models. Although the application to spatial perturbations provides a different setting, the core amplification mechanism is relatively straightforward. The paper should better clarify its methodological novelty compared with existing activation steering and inference-time attention manipulation methods.
  2. The experiments focus on target-object displacement in simulated robotic manipulation tasks. It remains unclear whether the identified attention-head mechanisms generalize to other perturbations, such as viewpoint changes, distractors, object appearance variations, or more complex manipulation scenarios. Additional evaluations would strengthen the generalizability claims.
  3. Attention-head amplification improves closed-loop success in only three of the five evaluated policies, while providing no consistent gains for OpenVLA and GR00T N1.7. These results suggest that the method depends heavily on architecture-specific information routing. A more systematic characterization of the conditions under which amplification succeeds or fails would strengthen its practical applicability.
  4. The proposed intervention requires selecting specific attention heads, amplification factors, and, in some cases, displacement subspaces. The paper acknowledges that effective amplification ranges can be narrow and vary across architectures. However, it remains unclear how reliably these configurations can be identified for unseen tasks, checkpoints, or deployment environments without extensive validation.
  5. Although the method improves success rates in several settings, the gains remain relatively modest in some cases. For example, OpenVLA-OFT improves from 12.3% to 17.9% at a 10 cm displacement, leaving most perturbed episodes unsuccessful. Moreover, not all held-out improvements are statistically significant. Further experiments are needed to demonstrate whether the improvements are sufficiently reliable for practical robotic deployment.
Questions for authors

Please see the above comments.

Reviewer_BuTsOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper fine-tunes π0.5 and GR00T N1.7 on LIBERO with and without proprioception, then applies blackout or freezing to the scene camera or the wrist camera, from episode start or after a finger-width closure trigger. Task success is reported on the Spatial, Object, and Goal suites. Physical outcomes are reported mainly on Spatial: non-target contact, object disturbance, sustained force, joint-velocity exceedance, joint-limit proximity, and target drops. Separate experiments remove robot parts or objects from one camera view and evaluate blackout training and a training-free mean-embedding replacement. The real-robot evaluation is a WidowX block-in-bowl task with π0.5, ten trials per condition, and faults from episode start.

Wrist faults and freezing reduce success more than scene faults and blackout. In simulation the physical outcomes differ as well. Freezing produces more extreme joint motion, more non-target contact, and more object disturbance. Higher drop rates after closure are concentrated in the policies trained without proprioception. Proprioception restores success when robot geometry is hidden and does not restore it when objects are hidden from the wrist view. Both mitigations raise success in some conditions and can increase contact or disturbance. On the real robot, wrist faults still lead to task failure, and the joint-limit pattern seen in simulation is absent.

Critical strengths
  1. Blackout and freezing lead to different physical behavior at low task success. With proprioception on LIBERO-Spatial, wrist blackout and wrist freezing from episode start leave π0.5 at 21.3% and 0.0% success. Non-target contact is 1.4% of steps under blackout and 19.5% under freezing, and object disturbance is 5.3% of episodes under blackout and 42.7% under freezing (Table 2). For GR00T, wrist joint-velocity exceedance rises from 3.3% to 82.0% of episodes. The command record supplies a mechanism. Under freezing, commands initially match the paired clean execution and diverge as the held frame becomes outdated. Under blackout, translation commands change when the fault begins (Section 2.2, Appendix Table 9).

  2. The proprioception and mitigation experiments restrict two conclusions that the success rates alone would suggest. Removing objects from the wrist view leaves success at 0–2.7% even though the scene camera still shows those objects, with or without proprioception. Removing the arm and palm from the scene view leaves π0.5 at 18.7% without proprioception and 63.3% with it (Table 10). Scene-blackout training raises scene-freezing success from 2.7% to 53.0%. Mean-embedding replacement raises π0.5 scene-blackout success from 22.7% to 64.7% and raises object disturbance from 6.7% to 13.3%. An increase in task success is therefore not, by itself, a reduction in physical side effects.

Critical weaknesses
  1. The real-robot experiments do not establish the physical failure modes named in the title. The evaluation uses one WidowX task, π0.5 only, ten trials per condition, and faults from episode start, so drops after gripper closure are not measured on the robot. Scene blackout succeeds in 9 of 10 real trials; the same fault leaves the LIBERO π0.5 policy near 23% success on Spatial. Section 2.4 reports that joint-limit margins do not follow the freezing pattern from simulation. The constraint behind that discrepancy is in the appendix: the controller clips command step size and joint position (Appendix A.3), so the simulated joint-exceedance rates are not motions this arm executes.

  2. With proprioception, the higher drop rate under wrist blackout is a small difference in point estimates. After gripper closure, blackout versus freezing gives 4.0% versus 2.0% for π0.5 and 6.7% versus 4.0% for GR00T. The task-cluster intervals in Appendix Table 8 are +2.0 [0.0, +4.7] and +2.7 [−2.0, +7.3]. The large differences are in the policies without proprioception: 28.7% and 42.7% under wrist blackout, against 6.7% under wrist freezing. The abstract already places the strong effect in that condition. Section 2.3 still presents the with-proprioception percentages as a difference and does not give the intervals.

Questions for authors
  1. Before clipping, do the WidowX joint commands show the elevated joint velocities reported for freezing in simulation? If they do, the simulated joint behavior is present in the commands and is then removed by the controller. If they do not, this policy does not produce that pattern on the real arm.

  2. Do you regard the higher drop rate under blackout, relative to freezing, as established for the policies trained with proprioception? The intervals in Table 8 for those policies touch or include zero. The differences for the policies without proprioception are large.

4 comments
AuthorsOpenReview ↗

[1/2] Official Rebuttal

Thank you for your constructive review and for highlighting our two central findings: (1) that blackout and freezing produce distinct physical behaviors, and (2) that higher task success does not by itself reduce physical side effects. Below, we present 80 new post-closure WidowX trials and pre-clipping command analyses showing that drops, non-target contact, and freezing-induced joint-limit proximity all appear on the physical robot, while clarifying the scope of the simulation JVE and drop claims in the revised manuscript.

[W1] Drops after gripper closure on the robot.

We ran 80 additional WidowX trials with the fault starting after gripper closure (π0.5, ten trials per cell; Table R5).

Table R5. WidowX task success and drops in ten post-closure trials per condition.

Fault after closureSuccess (+P)Success (−P)Drop (+P)Drop (−P)
Wrist blackout4/106/100/103/10
Wrist freezing7/104/101/102/10
Scene blackout8/104/100/100/10
Scene freezing9/102/100/100/10

Under wrist faults, drops occur in 5 of 20 trials without proprioception and in 1 of 20 with it (0 of 20 under clean inputs), matching the simulation finding that post-closure drops under wrist faults are concentrated in policies without proprioception. On the same WidowX block-in-bowl task, we also observed non-target contacts (Section 2.4) and found that the arm approached its joint limits under post-closure scene freezing, as detailed below. The full results are reported in Appendix A.3 of the revision.

AuthorsOpenReview ↗

[2/2] Official Rebuttal

[W1, Q1] Joint-limit margins on WidowX and elevated joint velocities before clipping

Before clipping, do the WidowX joint commands show the elevated joint velocities reported for freezing in simulation? (Q1).

Before clipping, the peak commanded joint velocity is 0.72–0.82 rad/s in every condition, including clean inputs (differences ≤ 0.043 rad/s). The policy therefore does not command elevated velocities on the real arm; they are not clipped out by the controller. This difference comes from the action parameterization: the WidowX policy outputs joint offsets (clarified in Appendix A.3), whereas the simulated policies output end-effector displacements that the controller converts to joint motion, and JVE arises from this conversion at full extension (see our response to Reviewer 9Dxh, W1 item 2 - JVE). When we retrain the simulated π0.5 policy with joint-offset actions, JVE disappears in all 16 fault cells (from 6.7–92.7% under freezing). This suggests that the action parameterization and controller jointly influence vulnerability to camera faults. Switching to joint-offset actions removes JVE, but joint-limit proximity remains higher under freezing than under blackout in seven of eight comparisons (see our response to Reviewer 9Dxh, W1 item 2). We have clarified in Sections 2.3 and 2.4 that the JVE result applies to end-effector actions.

Joint-limit margins do not follow the freezing pattern from simulation.

With faults from episode start, the robot does not show the freezing pattern, where joint-limit proximity is more frequent under freezing than under blackout. Without proprioception, the arm comes within 0.05 rad of a joint limit under both faults, but freezing does not produce more near-limit events than blackout for either camera. However, in our new trials where faults are introduced after gripper closure, the robot shows the pattern observed in simulation. Specifically, without proprioception, the arm comes within 0.05 rad of a joint limit in 9 of 10 trials under scene freezing, compared with 1 of 10 under scene blackout and 1 of 10 under clean inputs. We report these additional post-closure trials in Section 2.4 of the revised manuscript and provide the full trial counts in Appendix E.4, Table 16.

[W1] Scene blackout: 9 of 10 on the robot vs. 23% in LIBERO-Spatial.

The difference may come from the fact that the two tasks depend on the scene camera to different degrees: the WidowX task has a single block and a single bowl, whereas LIBERO-Spatial requires choosing between two identical bowls by their position relative to surrounding objects, which only the scene view shows. Although the absolute success rates differ substantially, the relative ordering of the two scene faults is consistent across both settings: scene blackout is less harmful than scene freezing (9/10 vs. 5/10 on the robot, clean 10/10; 23.3% vs. 2.7% in simulation, clean 95.3%). We have noted this task-dependency difference in Section 2.4.

[W2, Q2] Do you regard the higher drop rate under blackout, relative to freezing, as established for the policies trained with proprioception?

No, we do not regard it as established for the policies trained with proprioception: the confidence intervals for the blackout–freezing differences in Table 8 touch or include zero. We have revised Section 2.3 to state the wrist-blackout drop result for the policies trained without proprioception and to include the Table 8 intervals in the main text.

We hope these additional trials and analyses address your concerns, and we remain happy to clarify any further questions.

Dear Reviewer BuTs,

Your feedback prompted us to run 80 additional post-closure WidowX trials and examine why the joint-velocity results differed between simulation and the real robot. We also clarified that the evidence for a higher drop rate under wrist blackout than under wrist freezing applies only to policies trained without proprioception.

Before the discussion period closes on November 18, we’d appreciate hearing whether these additions address your concerns or whether there’s anything else we can clarify.

Thank you for your time and consideration.

Sincerely,
The Authors

Thank you for the additional trials and for answering the questions. These results address the points I raised. I am leaving the initial score. The revision matches the evidence on which this score was based. The new robot trials support the narrower claim and do not go beyond it.

Reviewer_9DxhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper compares camera blackout and frozen images in pi05 and GR00T, varying the affected camera, fault onset, and availability of proprioception. It measures physical outcomes alongside task success, investigates dependence on robot and object imagery, and evaluates blackout training and mean-embedding replacement as mitigations.

Critical strengths

The main strength of the paper is that it reveals the brittleness of the paper under freezing and blackout observations. Models like GR00T can behave erratically with wrist view frozen. Experiments show different camera faults lead to different suboptimal model behaviors.

Critical weaknesses

My main concern of the paper is that the study does not explain the failure modes sufficiently despite the good characterization. The experiments demonstrate the different outcomes under blackout and freezing, but do not clearly separate missing information, distribution shift, or even controller dynamics as the possible causes. More studies on the exact cause of the failure modes might lead to more interesting and convincing ways of mitigating the failures seen in the experiments.

This is also related to the relatively limited technical contribution of the paper: the two mitigation approaches proposed are relatively simple tricks that do not resolve the issue to the large extent.

Questions for authors

What explains the large success losses from wrist-embedding replacement at grasp?

5 comments
AuthorsOpenReview ↗

[1/4] Official Rebuttal

Thank you for your time and effort in reviewing our paper. We sincerely appreciate your recognition that our study reveals the brittleness of VLA policies under camera blackout and freezing, and your acknowledgment that our experiments characterize how different camera faults lead to distinct failure behaviors.

In response to your comments, we have added (i) intervention and trajectory analyses that separate the causes of each physical failure mode (object drops, joint-limit proximity, joint-velocity exceedance, and unintended contact), (ii) a mitigation combining blackout training with freeze detection that recovers most of the freezing loss, and (iii) an analysis of the success loss from wrist-embedding replacement at grasp. We have revised our manuscript accordingly, with updates highlighted in blue.

[W1] The experiments do not clearly separate the possible causes.

"…do not clearly separate missing information, distribution shift, or even controller dynamics as the possible causes."

We address this concern with wrist-image interventions for object drops, command and trajectory analyses for joint behavior and contact, and an action-parameterization experiment for JVE. Throughout this response, +P and −P denote policies trained with and without proprioception, respectively.

  1. Drops under wrist blackout are caused mainly by missing object information in the wrist view.

Immediately after gripper closure, we either erased the held target from the live wrist image or kept only the held target visible on an otherwise black wrist image (Table R1).

With the held target erased from the live image, Drop was 42.0% for π0.5 −P and 68.0% for GR00T −P. With only the held target visible on a black background, Drop was 22.0% and 8.7%, respectively, compared with 2.0% and 0.7% under clean inputs.

Additionally, to assess visual distribution shift, we used the 5-nearest-neighbor (5-NN) distance to clean post-closure wrist embeddings as a measure of how atypical each input was in the policy’s feature space. We normalized distances separately for each policy so that the fully black input has a value of 1.00. Smaller values indicate greater proximity to clean wrist embeddings.

Table R1. Wrist-feature distance, post-closure Drop, and task success (SR) under wrist-image interventions for policies trained without proprioception.

Wrist image after closureπ0.5 −P 5-NN distanceπ0.5 −P Drop (%)π0.5 −P SR (%)GR00T −P 5-NN distanceGR00T −P Drop (%)GR00T −P SR (%)
Clean0.242.094.70.230.795.3
Black, held target only0.4822.044.70.518.748.0
Live, held target erased0.5842.01.30.4068.04.0
Fully black1.0030.716.71.0047.314.7

Across the wrist-image interventions, inputs farther from clean wrist embeddings generally had lower task success. Drop, however, did not consistently increase with this measure of distribution shift. For π0.5 −P, the live image with only the held target erased was closer to clean than full blackout (5-NN distance of 0.58 vs. 1.00), yet its Drop rate was higher (42.0% vs. 30.7%). Restoring only the held target on a black background reduced Drop in both policies, from 30.7% to 22.0% for π0.5 −P and from 47.3% to 8.7% for GR00T −P. These results suggest that task success is sensitive to visual distribution shift, while the target-removal and restoration experiments show that losing visual cues from the held target increases the risk of a drop.

Under wrist blackout, −P policies bring the object over the plate in 70–71% of trials (97% clean) but lower it to the plate in only 21–22% (93% clean), opening the gripper 8–12 cm above the plate (Table R2). With only the held target visible, transport over the plate returns to near-clean levels, but descent recovers only partially. With all object pixels visible, both the lowering rate and the downward command (dz) approach clean levels.

Table R2. Placement trajectories after gripper closure for the two policies without proprioception.

Wrist image after closure (π0.5 −P / GR00T −P)Brought over plate (%)Lowered to plate (%)Descent command dz over plate
Clean97.3 / 96.793.3 / 93.3−0.60 / −0.61
Black71.1 / 70.020.8 / 22.0−0.29 / −0.09
Black, held target only90.6 / 96.047.7 / 76.0−0.26 / −0.23
Black, all object pixels kept93.3 / 98.783.2 / 94.7−0.52 / −0.54

The held target provides a key visual cue for maintaining the grasp after closure. Restoring its pixels alone reduces drops under wrist blackout (Table R1), and restoring the other object pixels as well brings the rate of lowering the target to the plate close to clean levels (Table R2).

AuthorsOpenReview ↗

[2/4] Official Rebuttal

W1 continued: Joint behavior and unintended contact under freezing

In Section 2.3 (Table 2), we found that freezing produced more extreme joint behavior than blackout, including more frequent approaches to joint limits (JLP) and joint-speed exceedances (JVE). We also observed more non-target contact (NTC), object disturbance (OD), and sustained force (F5) under wrist freezing than under wrist blackout for π0.5. In Section 2.2, we found that commands under freezing initially align with those in paired clean runs but diverge as the frozen frame becomes outdated.

We hypothesize that, as the arm moves, the unchanged image from the affected (frozen) camera leads the policy to keep issuing motion commands in a similar direction. We first test whether commands accumulate more strongly in one direction under freezing than under either clean inputs or blackout. We then examine how this persistent motion relates to the observed joint behavior (point 2) and unintended contact (point 3).

To quantify whether the commands accumulate in one direction, we compute the accumulation ratio across steps 50–150. Letting aₜ denote the executed translation command (x, y, z) at step t:

accumulation ratio = ‖Σₜ aₜ‖ / Σₜ ‖aₜ‖, t = 50,…,150

The ratio is 1 when every command points the same way and 0 when the commands cancel out. Table R3 reports medians over 150 episodes per condition.

Table R3. Command accumulation over steps 50–150 (median across 150 episodes per condition).

PolicyCleanScene blackoutWrist blackoutScene freezingWrist freezing
π0.5 +P0.480.300.190.760.70
π0.5 −P0.500.330.300.970.88
GR00T +P0.510.460.400.810.92
GR00T −P0.510.490.550.960.95

2. Extreme joint behavior under freezing

  1. JLP is caused by the accumulation of outward motion commands.

With a frozen image, the policy repeatedly commands outward motion (radial cosine 0.29–0.73). These commands add up in one direction: the accumulation ratio is 0.70–0.97 under freezing, vs. 0.48–0.51 in clean runs and 0.19–0.55 under blackout. As a result, the arm keeps extending until its joints approach their limits, which raises JLP under freezing (58.7–89.3% of episodes, vs. 1.3–2.7% in clean runs).

  1. JVE arises from the controller’s conversion of end-effector actions into joint motion near full extension.

This persistent outward motion drives the arm to full extension (≥99% of reach; clean episodes stay ≤96%). The policies output end-effector displacements, which LIBERO’s operational-space controller (robosuite OSC_POSE) converts into joint motion. An end-effector velocity ẋ requires joint velocities q̇ satisfying ẋ = J(q)q̇, so along the least-actuated direction (where the arm is hardest to move),

‖q̇‖ = ‖ẋ‖ / σₘᵢₙ(J(q))

where σₘᵢₙ is the smallest singular value of the Jacobian. At full extension, σₘᵢₙ falls from 0.2435 to 0.0123, so the same end-effector speed along this direction requires about 20 times the joint speed and exceeds the joint-velocity limit; 99.5% of JVE events occur at this pose.

If JVE arises from the controller’s conversion of end-effector displacements into joint motion near full extension, we would expect JVE to disappear when the policy predicts joint-position offsets instead of end-effector displacements. To test this prediction, we retrained π0.5 with this action representation and used a joint-position controller to execute the predicted offsets. With the original setup, JVE occurred in 6.7–92.7% of episodes under freezing. With joint-offset actions, we observed no JVE in any of the fault conditions. Joint-limit proximity, however, remained more frequent under freezing than under blackout in seven of eight comparisons. These results support our explanation that the end-effector-to-joint conversion contributes to JVE.

AuthorsOpenReview ↗

[3/4] Official Rebuttal

W1 continued

3. Contact under freezing (NTC, OD, F5) is caused by the same accumulation: the approach motion of the frozen frame continues.

As shown in Point 2, the policy keeps issuing commands in the same direction under freezing. The repeated descent command brings the gripper down to the table, where it presses against the surface instead of grasping and generates sustained force (F5) (the robot contacts the table in 10.4–20.9% of steps under freezing, vs. 0.1–0.3% in clean runs). The gripper then continues moving at table height, contacting and pushing surrounding objects (NTC, OD), and once fully extended, slides along its reach boundary and sweeps further objects in its path.

Under blackout, the commands do not accumulate and the robot rarely touches the table (0.0–2.6% of steps), so these contacts are less frequent.

In summary, (1) object drops (Drop) under wrist blackout are mainly caused by missing object information in the wrist view; (2) under freezing, persistent outward commands drive the arm toward its joint limits (JLP), while joint-velocity exceedance (JVE) additionally stems from the controller’s inverse-Jacobian conversion at full extension; and (3) the same command accumulation drives the gripper down against the table to produce sustained force (F5) and moves it along the surface, contacting and displacing nearby objects (NTC, OD). We report these results in Sections 3.1 and 3.2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.