acceptodds
Under review as a conference paper at ICLR 2027

Causal Transfer of Scene Information in Robot Policies

Abstract

An internal edit can move a robot toward a donor’s target without identifying what scene information it transferred. We study whether a controlled scene factor can be transported through a policy’s internal state into predictable physical behavior. Using target location, we generate counterfactual observations, replace key–value caches at selected recipient layers with donor caches, and execute all branches from a common simulated physical state. Our primary frozen confirmation compares the true cache difference with four fixed channel-sign transformations that preserve each selected layer’s K/V perturbation norm within 2% while changing its direction. Across two manipulation tasks, FastWAM and π0.5, 128 new model rollouts yield 117 reachable and 114 visually eligible states; all four task–model cells meet the joint criterion for bidirectional relative endpoint effects exceeding the equal-norm controls, with control-adjusted means of 51–72 mm. A separate non-target spatial comparison supports three cells, with the fourth limited by a pre-specified control-visibility requirement. In bounded two-axis development pilots (four model-visited states per policy family), own-axis effects exceeded cross-axis responses for every tested state–axis pair; these pilots are descriptive rather than independent confirmation. Independent offline reconstruction reproduces the primary effects over 2,340 branches. Together, these results show that, under the tested cache intervention and short simulated horizon, the direction of target-conditioned internal differences—not only their magnitude—can be transferred to produce predictable relative object-motion biases.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, vision-language-action models, activation patching, causal interventions, robotic manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bsMGOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether target-location information can be causally transferred through internal representations of vision-language-action (VLA) policies to produce predictable changes in robot behavior. The proposed protocol transplants selected key-value caches between counterfactual observations while keeping the physical execution state fixed. Experiments across two VLA models and two manipulation tasks demonstrate bidirectional relative endpoint biases that exceed the mean absolute endpoint effects of four predefined equal-norm controls.

Critical strengths

The experimental design carefully separates internal interventions from changes in the physical environment, providing a controlled setting for examining causal effects. The use of bidirectional predictions and equal-norm controls helps distinguish target-conditioned directional effects from generic perturbation effects. The evaluation across two VLA models and two tasks, together with a frozen experimental protocol and code-independent offline reconstruction, strengthens the reliability of the empirical findings.

Critical weaknesses

The research question and the experimental design used to address it are clear. However, I would appreciate a clearer explanation of the new scientific understanding gained from establishing these causal effects. In particular, what do these findings reveal about VLA internal representations and action generation that prior probing or activation-intervention studies did not establish? While more rigorous causal evidence is valuable in itself, the implications of this evidence for our understanding of how these policies generate behavior could be articulated more explicitly.

I would also appreciate clarification on how these findings could inform existing methods for analyzing or controlling VLA models. This does not necessarily require proposing a new control algorithm or improving task success. Rather, explaining the additional value these findings offer to existing methods, or providing a simple illustrative application, would help clarify the significance of the work.

The finding that internal interventions can produce predictable directional biases in robot behavior is interesting. It also raises the question of whether these findings could support more flexible interventions, including human-in-the-loop robotic control. For example, could behavior be adjusted finely in response to user-specified changes in target location? Although the paper acknowledges its limited scope, a more explicit discussion of how the demonstrated effects relate to these broader possibilities, and what limits such extensions, would be helpful.

The evaluation across two models and two tasks is a strength. Nevertheless, the restricted experimental environments and intervention scope leave open how broadly the findings generalize. Building on the paper’s discussion of its controls, a more direct comparison with closely related activation-steering and interpretability studies would help clarify the specific evidential advantage of the proposed protocol.

Questions for authors

Please see the weaknesses.

2 comments
AuthorsOpenReview ↗

Cross-scene cache transfer and 10 mm corrections

What does this establish about action generation? A probe could decode the pad’s location even if the policy ignored it. Here, moving only the pad in the donor image predicted which way a cache swap would shift the held object’s x endpoint. In all 114 eligible states across both policies and tasks, the two swaps produced opposite shifts relative to their unedited recipients. Each exceeded the mean absolute effect of four equal-norm edits (§5.2). The target-induced cache difference was sufficient to bias action generation.

Häon et al. steer direction and speed, and Hong et al. use contrastive activations for feedback control. Grant et al. reproduce a source trajectory by injecting π0.5 action-expert activations, including on a physical robot. We tested π0.5’s multimodal prefix against two references in the same simulated recipient scene. One moved the pad in the recipient image. The other replayed the source’s actions. We compared the resulting planar object displacements at +24, each relative to its center baseline.

We matched 14 disjoint pairs on stapler, pad color and arm and used one instruction per pair. Five met the opposite-side pad criterion. Testing both recipients in four directions gave 40 comparisons, 34 with distinguishable references. With a same-scene donor, 33/34 edits landed at least 5 mm closer to the local visual response than to replay. With a cache difference from the other scene, 15 favored the local response, 16 favored replay and 3 were ambiguous. Only 3/34 showed a local advantage at least 5 mm greater than both the zero-displacement margin and the median of four equal-norm controls. A steering method can use these two references to check whether its edit follows a new scene. In these five pairs, visual cues and prepared movements both remain plausible.

Could an operator request a 10 mm correction? We calibrated an x shift relative to the unedited +24 endpoint on 12 simulated FastWAM/stapler states. In 13 held-out states, a nominal 10 mm donor offset gave 7/13 hits each way (x error and unintended y shift each within 5 mm). Calibrated offsets of 16.7 mm left and 17.1 mm right gave 10/13 each way. Also requiring height within 5 mm gave 9/13 leftward and 7/13 rightward hits. One leftward request gave 42 mm left and 16 mm sideways. We would reobserve the object before another correction.

Thank you to the authors for their rebuttal. However, as the response was submitted on the last day, I have not had sufficient time to fully understand and assess it. So I will defer further discussion and evaluation to the discussion phase with the AC.

Reviewer_78ubOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a controlled protocol for testing whether internal differences induced by changes in target position can cause predictable changes in robot behavior. The authors transplant selected donor key–value caches into recipient computations while executing all branches from the same simulated physical state. Bidirectional swaps and four layerwise equal-norm controls help assess whether the behavioral effects depend on the direction of the transferred difference beyond its magnitude. Across two manipulation tasks and two policy families, all four task–model combinations meet the prespecified criteria, with donor-consistent relative endpoint effects exceeding the mean absolute control effects. Small two-axis development pilots also suggest coordinate-selective responses. The findings support target-conditioned behavioral transfer under the tested short-horizon cache interventions.

Critical strengths

1.By linking a controlled target-position change to an explicit directional prediction, the paper offers a useful protocol for studying internal interventions in robot policies. Common-state execution, bidirectional swaps, and equal-norm controls make the behavioral effects of cache transplantation easier to interpret, providing stronger evidence for the role of scene-conditioned internal differences in robot control. 2.I also find the validation convincing within the tested scope. Both directional effects exceed the mean absolute control effects in all 114 eligible states across the four task–model combinations. Together with the frozen protocol and outcome-independent state selection, this gives confidence that the finding is consistent across the evaluated states and is not driven by a few favorable examples.

Critical weaknesses

1.The main confirmation tests target-position changes along one axis in two tasks at a single post-grasp stage. The two-axis evidence comes from only four development states per policy family. This leaves limited evidence for how broadly the protocol can be applied to other task-relevant scene factors and manipulation stages.

2.Section 5.2 reports that both directions pass the donor-endpoint proximity check in only 11/29 FastWAM/stapler states, compared with much higher rates in the other settings. Could the authors clarify why the pass rate is lower in this setting?

Questions for authors

1.The main confirmation tests target-position changes along one axis in two tasks at a single post-grasp stage. The two-axis evidence comes from only four development states per policy family. This leaves limited evidence for how broadly the protocol can be applied to other task-relevant scene factors and manipulation stages.

2.Section 5.2 reports that both directions pass the donor-endpoint proximity check in only 11/29 FastWAM/stapler states, compared with much higher rates in the other settings. Could the authors clarify why the pass rate is lower in this setting?

1 comment
AuthorsOpenReview ↗

Two-axis results, pre-grasp transfer, and the 11/29 endpoint check

How far does the result generalize? We expanded the post-grasp two-axis pilot in §5.4 from four development states per policy to 64 new scenes shared by the two policies. Of 128 rollouts, 101 qualified. In all four task–policy combinations, the donor-aligned shift exceeded the equal-norm controls on both axes at the group level. Three combinations also passed the stricter test of whether an x or y edit preferentially moved the object along that axis. The prespecified overall criterion required all four. For π0.5 on display stand, one y edit led to a fall with substantial x motion. With that endpoint included, the mean x-selectivity margin was −1.0 mm (adjusted 99.6875% CI [−65.0, 21.6]).

To test an earlier stage, we kept the FastWAM policy and stapler task fixed and intervened before the first reach. We moved the stapler itself 50 mm left or right in the donor view, swapped the same cache layers, and kept the physical scene fixed. After a four-seed pilot (three eligible), we tested all 28 remaining seeds. At action 24, the reaching wrist’s x endpoint shifted by 64.8 mm on average in each donor direction relative to its unedited recipient. Both shifts exceeded their matched controls in every state (mean absolute control shifts of 3.6 mm right and 3.0 mm left). This extends the evidence from carrying an object toward a pad to reaching for the object itself. Both tests change location. Whether the protocol extends to other scene properties, such as object identity, remains open.

Why 11/29 for FastWAM/stapler? We measured the held stapler’s x position after 24 actions from the same post-grasp state, with the physical pad fixed. In all 29 states, swapping the cache shifted the stapler in the predicted direction relative to its unedited branch. Section 5.2 also asks how far it shifted. To count, the edited stapler had to finish closer to where the unedited policy placed it when shown the donor pad position than when shown the original pad position. Both swap directions had to pass.

Take one rightward swap. With no edit, showing the pad in its original position gave a stapler x position of 180.6 mm. Showing the pad on the right gave 223.9 mm. Halfway between those readings was 202.3 mm. The unedited left view gave 134.7 mm. Putting the right-view cache into that left view gave 200.1 mm. The 65.4 mm rightward shift still fell about 2 mm short of the midpoint.

In both FastWAM tasks, edits shifted the object by about 55 mm on average in each direction. The shift needed to reach the midpoint averaged 51–53 mm for stapler versus about 49 mm for display stand. Of the 29 stapler states, 17 passed on one side only. In 16 of those, the side that missed required the larger shift. Eleven passed both ways, compared with 26/29 on display stand. The relative contributions of the policy and contact dynamics to these distances remain open.

Reviewer_MePvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows a rigorously controlled test whether a robot policy’s memory carries usable where-is-the-target information. With the physical world (the perception) stays fixed, only the policy’s internal belief about the target is changed. By replacing selected key-value cache layers with those from a donor computation, the authors demonstrate that this internal edit produces a predictable, donor-consistent bias in the manipulated object's physical motion. Furthermore, because this behavioral shift exceeded the effects of magnitude-matched layerwise equal-norm controls, the results confirm that the specific direction of the target-conditioned difference causally drives the robot's actions.

Critical strengths

Rigorous design: Moving the target exclusively during input generation while keeping the physical execution scene perfectly fixed, the design nicely eliminates the possibility that the robot simply reacts to a moved physical object.

Experimental design: The study demonstrates good experimental methodology including bidirectional predictions, Bonferroni-corrected bootstrap intervals, and state-wise consistency.

A relatively strong performance: Internal interventions yield meaningful physical results, producing control-adjusted shifts of 51.2 to 72.4 mm against a 100 mm baseline.

Critical weaknesses

The evaluation takes a 24-action budget to measure relative shifts, meaning it does not demonstrate long-horizon task success.

Relocating the rendered input target inherently alters low-level features like edges or occlusion, which prevents the test from isolating an abstract symbolic coordinate representation.

Questions for authors

Do these tests maintain stability with cache interventions over a full, long-horizon trajectory once it surpasses the 24-action window?

Could the authors add a control that matches the target’s low-level visual change without changing its task-relevant position?

1 comment
AuthorsOpenReview ↗

Persistence after replanning and fixed-target visual controls

Does one edit survive replanning? A directional shift often survived to episode end. In all 29 FastWAM/stapler states, we edited one post-grasp prediction and resumed normal replanning until success or the 400-action limit. Comparing each branch at its own endpoint, 23/29 rightward and 25/29 leftward pairs retained the donor’s sign (median x shifts 11.4 and 17.6 mm). Placement success showed no consistent gain. Rightward edits succeeded in 18 cases versus 12 for their matched baselines. The corresponding leftward counts were 14 versus 19.

At +32, finishing each branch’s original prediction gave mean right/left shifts of 60.9/60.1 mm. From the same branch history through +24, replanning instead gave 33.0/33.9 mm. Under normal replanning, 26/28 pairs observed at +48 in each direction retained the donor’s sign.

For π0.5, we followed the first two eligible states per task. Both display-stand centers succeeded before +50. In the two stapler states, we followed the four original equal-norm controls per direction after seeing the donor outcomes. At +50, all eight donor shifts exceeded their mean absolute controls. After one unedited replan, early successes left five of eight complete comparisons at +100. Their margins were 14.3–48.7 mm.

What if the pad stays put? In four FastWAM/stapler states, moving the pad in the donor view shifted the held stapler by 29.7 mm on average. An exposure change matched for RGB magnitude in each camera gave a mean absolute shift of 0.85 mm.

We then changed only the tabletop texture beside the pad. The pad, its contour, gripper and stapler kept their original pixels. Both texture polarities had to match the translation in each active camera on RGB change (within 10%), edge change (within a factor of two) and partial spatial overlap. Two of eight pilot comparisons qualified, both rightward. Before viewing texture outcomes, we fixed the same screen for the remaining 25 states. Ten of 50 comparisons qualified, five per direction in ten different states. At +24, the translated pad exceeded the mean absolute texture effect in all ten. The means were 31.9 versus 1.4 mm leftward and 34.5 versus 2.9 mm rightward. The moving pad’s own edge and the surfaces it reveals remain possible cues.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.