acceptodds
Under review as a conference paper at ICLR 2027

Steering Policies with Emergent Registers

Abstract

Vision-language-action (VLA) policies acquire rich manipulation skills, yet deployment may require using these skills in ways not seen during training, such as manipulating new objects, correcting an ongoing reach, or avoiding a designated region. Test-time steering can provide guidance to the policy for these issues without retraining. Prior steering methods largely modify the action-generation process, such as its latent noise or flow dynamics. We instead ask whether steering can act through visual representations within the VLM backbone. Across multiple VLAs, we find that action tokens concentrate attention on visually uninformative image patches, resembling registers in Vision Transformers. We call these action registers. Surprisingly, masking attention to action registers changes predicted trajectories far more than masking attention to task-relevant image tokens. We therefore introduce RegiSteer, which applies test-time residuals to action registers to steer frozen VLAs toward desired Cartesian directions while leaving detailed manipulation to the policy. RegiSteer improves manipulation across generalist and specialist policies without weight updates, remains robust to noisy spatial guidance, and supports spatial safety constraints.

Primary area:
applications to robotics, autonomy, planning
Keywords:
steering, VLA, manipulation, robotics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wjnMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies “action registers”: visually uninformative image tokens that receive high action attention and strongly influence VLA outputs. It proposes RegiSteer, which optimizes test-time residuals on these tokens to guide a frozen policy toward 3D waypoints while retaining learned manipulation behavior.

Experiments cover three generalist policies on 120 RoboLab tasks and OpenVLA in LIBERO-PRO, showing improved placement and task-dependent success gains. Ablations examine register influence, alternative steering channels, guidance noise, and runtime. Small-scale physical-robot demonstrations use precomputed directional residuals to verify the effectiveness.

Critical strengths
  1. Clear motivation and training-free adaptation. The paper addresses the practical need to redirect pretrained VLAs toward new spatial goals while retaining their manipulation skills. RegiSteer keeps policy weights frozen and optimizes only residuals on selected visual tokens, avoiding additional policy training and reducing adaptation costs. This makes the approach convenient to apply to existing policies.

  2. Evaluation across multiple policy families. Experiments cover three generalist policies on 120 RoboLab tasks and an autoregressive OpenVLA policy under object relocation in LIBERO-PRO. The placement improvements and recovery of OpenVLA success from 0% to 65% provide meaningful evidence that the approach can redirect existing manipulation skills across different action-generation architectures.

Critical weaknesses
  1. Substantial online inference overhead. Online RegiSteer requires two forward generations and one backward pass, increasing measured latency by approximately threefold—for example, from 57 ms to 177 ms per action chunk for π0⋅5\pi_{0\cdot5} on an H200. This weakens its practical appeal for responsive robot control. Comparisons under matched latency or control-frequency budgets are needed to establish whether the performance gains justify this cost.

  2. Limited real-world validation. Physical experiments consist of small-scale demonstrations using precomputed directional residuals, rather than the online gradient-based method evaluated in simulation. The paper lacks systematic real-robot task-success comparisons against strong baselines across diverse tasks. Consequently, it remains unclear whether the simulation gains and claimed preservation of manipulation behavior translate reliably to physical deployment.

Questions for authors
  1. Given the approximately threefold online inference cost, does RegiSteer retain its advantage when competing methods operate under matched latency or control-frequency budgets?
  2. Has the online gradient-based variant been tested on physical robots? What evidence supports its practical effectiveness beyond the precomputed-residual demonstrations?
  3. How does RegiSteer compare with directly optimizing a residual on the final action chunk using the same waypoints and kinematic objective?
  4. Across the 120 RoboLab tasks, what proportions improve, remain unchanged, or deteriorate? Are gains concentrated in particular task types?
  5. What exact procedure determines the fixed steering masks, and were the tasks used for mask selection separate from those used for evaluation?
5 comments
AuthorsOpenReview ↗

Reply for wjnM [1/1]

Thank you for the constructive review. We are glad you find the motivation clear, the training-free approach convenient, and the results across policy families meaningful. All additions below are in the revised PDF, edited and new content in blue.

The general comment lists the new table numbers and explains which tokens we steer.


[W1, Q1]

"Substantial online inference overhead. (...) Comparisons under matched latency or control-frequency budgets are needed"

We added the runtimes of Flow Control, DynaGuide and FRS to Tab 11 (A.1.4). Latency per query:

Methodπ0 (ms)π0.5 (ms)
Base5457
Flow Control6165
RegiSteer163177
DynaGuide177186
FRS188204

To make each method pay for its own latency, we ran π0.5 on a separate set of RoboLab tasks with noisy waypoints and the simulator clock running during inference. Objects placed per episode (Tab 11, bottom):

Clock runningσ = 8 cmσ = 16 cm
Base0.370.37
Flow Control0.600.50
IK-EE0.500.43
RegiSteer1.070.80

RegiSteer places the most objects at both noise levels, so its extra latency doesn't erase its lead, and at σ = 8 cm its lead over IK-EE is significant (p = 0.02, Tab 19). IK-EE tracks the noisy target blindly, while RegiSteer only biases a policy that still sees the current images (Sec 5.2). On the real robot, where the steering directions are known in advance, precomputed residuals cut the extra cost to about 0.1% (Sec 5.3).


[W2, Q2]

"Physical experiments consist of small-scale demonstrations using precomputed directional residuals, rather than the online gradient-based method evaluated in simulation."

We agree that the robot part is a demonstration, and Sec 6 says broader physical evaluation is needed. It shows that the register channel can be steered in a different VLA (MolmoAct2) on a different robot (SO-101). We added the online gradient variant on the SO-101 in A.5. From Tab 14:

VariantMean alignmentMin. alignmentEE displacement per query
Precomputed0.987≥ 0.9650–95 mm
Online gradient0.995≥ 0.9810–25 mm

We also added task success over five trials per task (Tab 14, bottom). For the online variant we set the waypoints ourselves, relative to the end-effector position.

Task (5 trials each)MolmoAct2+ RegiSteer, precomputed+ RegiSteer, online
Pick the leftmost object and drop it into the bin1/54/55/5
White object into the mug, black object into the bin0/53/54/5

Both variants raise success well above MolmoAct2 alone, and the online variant does slightly better (A.5).


[Q3]

"How does RegiSteer compare with directly optimizing a residual on the final action chunk using the same waypoints and kinematic objective?"

We added it to Tab 1 ("Residual δ to actions"). With the same loss it moves the arm strongly (gain 9.12, alignment 0.903) but places only 0.50 objects, against 2.46 for RegiSteer and 2.08 for IK-EE. The policy does not see an action residual, so its next chunk pulls back toward its own plan. IK-EE avoids this by replacing the actions during transport, which also removes the visual feedback. RegiSteer edits the policy's input instead, so each new chunk carries the correction.


[Q4]

"Across the 120 RoboLab tasks, what proportions improve, remain unchanged, or deteriorate?"

Counts over the 120 tasks in which OBJ improves, stays the same or gets worse against the base policy (A.11):

PolicyImprovesUnchangedWorse
π0.5584517
π061545
GR00T634116

Most unchanged tasks are ones where neither the base policy nor RegiSteer places an object. By group, OBJ improves in all 18 groups of Tab 15 and success in 13.


[Q5]

"What exact procedure determines the fixed steering masks, and were the tasks used for mask selection separate from those used for evaluation?"

It is the A.1.1 procedure, now also stated in Sec 4.1, run once per policy and camera view. We average action-to-image attention over one episode of each of six RoboLab tasks, keep the 10% most-attended patches per task (Eq. 6), and keep the locations shared by all six. This gives the Fig 12 sites, which all main results use, and Tab 20 lists their coordinates.

  • The six selection tasks are among the 120, and some also appear in Tab 1, Tab 4, Tab 5 or Fig 7. Leaving them out changes the OBJ gains of Tab 15 by at most 0.4 points (A.1.1).
  • For OpenVLA the procedure runs on three randomly chosen LIBERO tasks, and for MolmoAct2 on the demonstration images of its repository.

Thank you for the helpful questions. We would be glad if you would consider raising your score.

Thank you for the additional experiments. The latency-aware simulation shows that the placement gains persist despite the inference overhead in the tested settings, and the action-residual baseline directly addresses my question about modifying the final action chunk. The new SO-101 trials also demonstrate that the online method can work on a physical robot.

My main remaining concern is how reliably those gains transfer across physical tasks. The new RoboLab breakdown shows improvements across several task categories, but it does not tell me whether the gains mainly come from tasks where correcting the approach position is enough. Could you report success rates over repeated trials on a few more real-robot tasks or scene layouts, comparing cases where an object is simply relocated with cases where its pose changes and the robot needs a different wrist orientation or grasp? A comparison with the base policy, and with IK-EE using the same waypoints if feasible, would help clarify when RegiSteer is useful in practice.

If the added trials show consistent gains across a broader set of physical tasks, I would consider raising my score.

AuthorsOpenReview ↗

A few more real-life robot tasks

Thank you for your feedback, we appreciate it. We are happy that the questions are directly addressed.


[A few more real-robot tasks]

"Could you report success rates over repeated trials on a few more real-robot tasks (...) comparing cases where an object is simply relocated with cases where its pose changes and the robot needs a different wrist orientation or grasp? A comparison with the base policy, and with IK-EE using the same waypoints"

Thank you for the recommendation. As you said, we added a few more real-robot tasks (four SO-101 tasks with five trials each), where the task success would need different wrist orientations from the initial position. IK-EE and RegiSteer use the same waypoint. Please see visuals and numbers in Tab 14, App A.5 (page 28-29).

TaskMolmoAct2+ IK-EE+ RegiSteer
Marker0/50/52/5
Granola bar2/50/54/5
Bottle3/50/54/5
Pill bottle3/55/55/5
Total8/205/2015/20

The pill bottle can be grasped from above at any wrist angle, allowing both IK-EE and RegiSteer to succeed in all five trials. The other three tasks require a change in wrist orientation. In these tasks, IK-EE retains the initial wrist orientation and fails, whereas RegiSteer allows MolmoAct2 to rotate the wrist as it approaches the object.

"If the added trials show consistent gains across a broader set of physical tasks"

RegiSteer matches or beats both methods on every task, and its gain over IK-EE is significant (p = 0.004). It also improves on MolmoAct2 alone in all six physical tasks, including the two from our first reply.


[Scope clarification]

We would also like to clarify the scope of our contribution. Our main contribution is identifying a novel property of VLAs and using it as an effective steering channel, while preserving their learned manipulation capabilities. We do not try to replace inverse kinematics in robotics as a whole.

We include IK-EE as a classical baseline to assess whether the same waypoint guidance can be used effectively through direct end-effector control. Our experiments highlight its limitations when the wrist orientation is unsuitable for grasping the object, or when noisy waypoints require correction from the VLA's learned visual priors (Sec 5.2, A.5). A more elaborate IK-based pipeline could incorporate perception, grasp planning and other sub-components to operate without a VLA (Sec 4.1). Such a pipeline represents a different approach to manipulation, and falls outside the scope of our work, which focuses on steering VLAs. Nevertheless, we still compare with a collision-free planner that hands off to the VLA, and RegiSteer leads it with noisy or perceived waypoints (Tab 5).


Thank you for making our submission stronger. We would appreciate if you would reconsider your score.

Reviewer_ZyYhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduce a VLA steering method. Specifically, they firstly observe that in several VLAs, action tokens usually concentrate on background regions. These are considered as "action registers" by the author, since masking them will significantly decrease the performance compared with normal content tokens. By using this observation, they steer a frozen VLA by residuals for the action register tokens. With this, the successrate on different tasks greatly improves.

Critical strengths
  • The method is highly efficient and training-free. Also, it is easy to generalize. Compared with sampler-specific steering methods, it only needs a single backward pass.
  • The experiments and the emperical study in the paper are clear and extensive. Compared with baseline, the method demonstrate large advantages. Also, the evaluation itself is complete since a large number of tasks and different random seeds are used.
Critical weaknesses
  • The comparison is kind of unfair. Specifically, compared with Astra, the RegiSteer have the ground truth object mask from the simulator, as well as the multi-view geometry. Thus, it gets much more information. Also, compared with IK-EE, the improvements are not significant enough especially on LIBERO. Considering IK-EE itself is a weak baseline, the gain is kind of small. It will be great to see stronger baseline like use traditional planner for the trajectory before stopping steering, and using the same input (object mask, MV geometry, etc.) for Astra.
  • Currently there seems do not have a clear rule to select the action registers. Specifically, the experimental masks are not the output of the procedure of A1.1 (the top-10% intersection across tasks), where it is claimed only for analysis,, and are "distinct from" the recurrence maps (Fig. 12). Sec. 4.1 describes the actual masks only as candidates read off heatmaps. Only π0/π0.5 is clearly explained.
Questions for authors
  • Is there any clean analysis to study whether the action registers are different from attention sinks[1]? Also, it is better to cite corresponding papers. [1] Xiao, Guangxuan, et al. "Efficient streaming language models with attention sinks." ICLR. 2024.
4 comments
AuthorsOpenReview ↗

Reply for ZyYh [1/1]

Thank you for the positive and constructive review. All additions below are in the revised PDF, edited and new content in blue.

The general comment lists the new table numbers and explains which tokens we steer.


[W1]

"compared with Astra, the RegiSteer have the ground truth object mask from the simulator, as well as the multi-view geometry."

We re-ran Astra (high reasoning) with at least the same information as RegiSteer: the simulator object poses, two-view estimates from the instance masks, the robot state and, on request, the exact object and gripper geometry. Objects placed per trial, from Tab 16 (3 trials per task for π0.5 and Astra, 6 for RegiSteer):

ClutterRedItemsCondimentsToolsCannedButterRaisinBoxOBJ / SR (%)
Targets per trial3242111
π0.52,2,20,0,20,1,11,0,00,1,00,0,00,0,028.6 / 9.5
Astra2,2,31,2,23,1,10,0,01,1,01,1,10,0,052.4 / 38.1
Astra + same geometry3,3,21,2,24,3,10,0,01,1,01,1,11,1,066.7 / 57.1
π0.5 + RegiSteer2,3,2,3,2,32,1,1,2,2,04,3,2,3,2,21,2,0,1,0,11,1,1,1,1,11,1,1,1,1,11,1,1,1,0,172.6 / 59.5

With the same information Astra closes most of the gap, and we read the two as comparable in task outcome.

  • RegiSteer still leads by 5.9 OBJ points, and A.9 traces this lead to ToolOrganizationBoth.
  • RegiSteer reaches this with a frozen VLA at 177 ms per query (Tab 11), while Astra needs 38 s per decision (A.9). The simulator clock is paused for both, so these scores do not include the cost of Astra's longer inference.

"compared with IK-EE, the improvements are not significant enough especially on LIBERO."

We added significance tests in A.11 (Tab 19).

  • RoboLab. RegiSteer places more objects than IK-EE in all 9 policy–difficulty cells of Tab 15, significantly for every policy over the 120 tasks (p ≤ 0.018).
  • LIBERO-PRO. The lead (65.0 vs 60.0, Tab 3) is not significant (p = 0.46). Our main point on LIBERO-PRO is that RegiSteer also steers a suite-finetuned autoregressive policy to unseen object locations, which FRS and Flow Control cannot do.

"It will be great to see stronger baseline like use traditional planner"

We added a cuRobo planner through the same waypoints, with exact scene geometry and the same hand-off to π0.5 (Sec 5.2). From Tab 5, objects per episode on the three tasks of Tab 4, and OBJ / SR (%) on the seven Fig 7 tasks:

σ = 8 cmσ = 16 cmSimulator masksSegmentation masks
IK-EE1.070.3751.2 / 33.346.4 / 28.6
Planner (cuRobo)1.170.4057.1 / 40.548.8 / 33.3
RegiSteer1.600.6372.6 / 59.569.0 / 54.8

Under noise the planner behaves like IK-EE (Sec 5.2), and with noisy or perceived waypoints RegiSteer leads it (p = 0.02 and 0.03, Tab 19), without the planner's exact scene geometry.


[W2]

"Currently there seems do not have a clear rule to select the action registers."

The rule is already in the A.1.1 procedure, now also stated in Sec 4.1. Per task, we keep the 10% most-attended patches, then the locations shared by all six tasks. This gives the Fig 12 sites R, which all main steering results use. The submission's "distinct from the fixed experimental masks" was badly worded and read as if the experiments used other masks. The new Tab 20 lists their (row, column) coordinates, so R can be reproduced from the paper alone, and we will release the index files. For OpenVLA we run the same procedure on three randomly chosen LIBERO tasks, and for MolmoAct2 on the demonstration images of its repository. On π0.5 the 35 sites keep 87% of the steering gain of all padding, and 35 random padding tokens only 40% (Tab 9).


[Q1]

"Is there any clean analysis to study whether the action registers are different from attention sinks? Also, it is better to cite corresponding papers."

We added this analysis in A.1.3 and now cite Xiao et al. (2024) alongside Sun et al. (2024). Sinks draw attention from all queries and carry massive activations. Candidate action registers differ from sinks in hidden-state norm and in attention from text and image queries (Tab 10):

Ratio to content tokensAttention sinks (typical, from the literature)Candidate action registers (measured)Like sinks?
Value-vector normsmall0.62–0.87×yes
Hidden-state normmassive0.7–1.4×no
Attention from action querieshigh2.21–7.86×yes
Attention from text and image querieshigh0.9–1.2×no

A residual on π0.5's three sinks reaches only 0.19 alignment, against 0.47 for three random register sites. Registers therefore mainly serve the action tokens.


Thank you again for the constructive feedback. We would be glad if you would consider raising your score.

Dear Reviewer,

Thank you again for your thoughtful feedback and suggestions. As the discussion session is about to end, we would appreciate hearing whether our responses have addressed your concerns. We would be happy to clarify any remaining points.

Thank you for your time and consideration.

Thanks for the additional experiments and the revision to the paper for clarity. It make more sense to see Astra much performing better than the original reported number. Most of my concerns are addressed and I am glad to keep my rating.

Dear Reviewer,

Thank you for participating in the discussion phase and suggesting new experiments to make the submission stronger. We are glad that the concerns are resolved and the rating is positive.

Reviewer_SZYmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies cross-attention from action tokens to image tokens in several VLAs and reports that action queries concentrate attention on visually uninformative image patches at fixed locations: padding borders for the π models, static background corners for GR00T. By analogy with registers in ViTs, the authors call these “action registers,” while noting they are attention outliers rather than norm outliers. Masking action-to-register attention perturbs predicted trajectories more than masking action-to-content attention. Building on this, the paper proposes RegiSteer: at test time, a residual is added to the register tokens at the vision-encoder output, computed by a single backward pass through the unrolled denoising steps and forward kinematics so that the predicted end-effector displacement aligns with a commanded Cartesian direction toward a waypoint.

Critical strengths
  1. The core observation is interesting and reasonably characterized. High-attention locations recur across tasks and across camera views, are not explained by value-vector norms, and are not produced by architecture alone.
  2. Repurposing these tokens as a control interface is a neat idea. Because the intervention is at the encoder output, the backbone and action head are untouched, so the same mechanism applies to flow/diffusion policies and to an autoregressive policy (OpenVLA).
Critical weaknesses
  1. The demonstrated capability is much narrower than the title and abstract suggest. Every experiment reduces to one behaviour: move the end-effector toward a supplied 3D waypoint in free space. Steering is switched off near the object and grasp, release, and orientation are left to the VLA. What is actually being steered is which object or region to go to next. On the paper’s own numbers, a kinematics solver given the same waypoints captures most of this: IK-EE vs RegiSteer is 60.0 vs 65.0 on LIBERO-PRO (Table 2, overlapping standard errors), 2.08 vs 2.46 placements in Table 1, and 2.20 vs 2.20 in Table 3. No experiment shows the register channel doing something IK-EE cannot. The only non-directional objective is the repulsion term, demonstrated on one rollout.

  2. Simpler ways to supply the same guidance are not compared. The waypoint sequence provides two things: sequential decomposition (index j advances one object at a time) and localization from simulator ground-truth masks. The base policies are run with the original compound instructions (e.g. “put all plastic bottles away”) and no decomposed or rephrased prompt is tried, so the share of the gain attributable to sequencing, which language can deliver for free, is unknown. Separately, concurrent work trains VLAs to accept motion and point commands directly (Steerable Policies, Chen et al., arXiv:2602.13193, which also fine-tunes pi-0.5); that line delivers the same directional and positional guidance through language plus subtasks, points, and composition. The paper should position itself against training-time steerability and state that its regime is “no retraining possible,” and it should include a decomposed-prompt baseline on the same frozen policies. Activation-direction steering (Häon et al., 2025), the closest mechanistic relative, is cited but not compared.

  3. The register used in experiments is not the register described in the text. For the pi models the experimental mask appears to be all padding tokens, whereas the recurrence analysis identifies 35 shared sites, and the Fig. 12 caption states the recurrence maps are “distinct from the fixed experimental masks.” Selection is by inspection of heatmaps with no stated criterion. This also makes the masking result close to tautological: tokens selected for carrying ~60% of attention mass will perturb the output more when masked, and count-matching does not remove the per-token attention gap. The control that would separate “registers” from “constant tokens that can be written to without corrupting perception” is missing: steer through the padding tokens of paligemma_diffusion, and compare residuals on the 35 recurrent sites vs all padding.

Questions for authors
  1. Why is only directional steering shown? Is this a choice or a limitation of the mechanism? The loss in chapter 4.2 admits any differentiable function of the action chunk; did you try orientation, gripper state, or speed objectives, and if so what happened? Does the single shared unit-norm residual per view have the capacity for more than one objective, and do precomputed residuals add?

  2. What exactly is the register set R in the experiments for each policy? Is it all padding tokens for pi-0/pi-0.5? Does steering through only the 35 recurrent sites of Table 4 perform the same? Does adding a residual to the padding tokens of paligemma_diffusion steer it?

  3. Can you report a decomposed-prompt baseline (one object per instruction) on the same frozen pi-0.5, and RegiSteer on top of it, so the contribution of spatial guidance beyond sequencing is isolated?

  4. Add significance tests for the RegiSteer vs IK-EE comparisons.

4 comments
AuthorsOpenReview ↗

Reply for SZYm [1/2]

Thank you for the careful review and the concrete suggestions. We are glad you find action registers interesting and their use as a control handle a neat idea. All new experiments below are in the revised PDF, edited and new content marked with blue.

The general comment lists the new table numbers and explains which tokens we steer.


[W1. Scope and IK-EE]

"The demonstrated capability is much narrower than the title and abstract suggest."

We respectfully disagree, as our title and abstract are compliant with our claims. Our main finding is a control knob inside frozen VLAs, the action registers, and it is not limited to waypoints. The Fig 1 recipe works with any differentiable objective on the action chunk, and only the loss changes with the task (Sec 4.2). Waypoints are our main test, but with other losses the same registers turn the wrist, close the gripper and speed up the motion (Tab 18), steer the arm away from a region (Sec 5.2), and carry two objectives at once (A.10.2).

What matters is that the VLA keeps its priors while it is steered, such as how to grasp an object. With RegiSteer, the frozen VLA still produces every action and controls the whole arm, including the wrist. IK-EE cannot do this. Because for IK-EE, VLA is completely disentangled from the arm, and it targets only the 3D end-effector position (Eq. 4–5). Therefore it does not control the wrist. The VLA then has to grasp from a wrist pose it did not choose, which causes failures and lowers success (Fig 6, Tab 5).

Steering frozen VLAs is an active and important area, and it will be more important as general policies grow larger and stronger. Retraining such a policy for every deployment is slow and costly. Steering instead adapts the policy to a new setting, or fixes a failure, quickly at inference time, and not all fixes can be handled by inverse kinematics. All robotics steering papers in Tab 1 are very recent around 2025-2026. RegiSteer also needs no training, unlike Steerable Policies (Chen et al., 2026).


"a kinematics solver given the same waypoints captures most of this" / "No experiment shows the register channel doing something IK-EE cannot."

Given the same waypoints, the cost of disentangling the VLA shows in two common cases:

  • Objects that need different grasps. On the seven Fig 7 tasks, with hammers, bottles, a mug and boxes, RegiSteer leads with the same waypoints (72.6 vs 51.2 OBJ), and also leads a planner that knows the exact scene geometry (57.1, Tab 5). In Fig 2, IK-EE places none of the hammers, and in Fig 22 only RegiSteer turns the mug upright.
  • Imprecise waypoints. IK-EE goes wherever the waypoint points, while the steered VLA still sees the object. With waypoints from Grounded-SAM 2, RegiSteer places 69.0% of the objects against 46.4% (p = 0.004, Tab 19).

Over the 120 RoboLab tasks, RegiSteer places more objects than IK-EE in every cell of Tab 15, significantly for every policy (p ≤ 0.018, Tab 19). IK-EE is therefore enough only when waypoints are exact and all objects take the same grasp. Adding the correction to the output actions steers strongly but places far fewer objects (0.50 vs 2.46, Tab 1), as the policy pulls back toward its own plan.


[W2. Simpler guidance]

"no decomposed or rephrased prompt is tried"

We added this baseline on the seven Fig 7 tasks, with one prompt per object (A.10.1). OBJ / SR (%) from Tab 2:

Compound promptOne prompt per object
π0.528.6 / 9.531.0 / 11.9
π0.5 + RegiSteer72.6 / 59.573.8 / 61.9

Decomposition adds 2.4 OBJ points, while RegiSteer adds 44.0 to the compound prompt and 42.8 on top of the decomposed one. This matches Steerable Policies (Chen et al.), who find that subtasks alone are not a viable interface for directing a low-level VLA.


"position itself against training-time steerability" / "Activation-direction steering (Häon et al., 2025) ... is cited but not compared."

Sec 2 now contrasts the retraining of Steerable Policies with our frozen weights, and Sec 6 states that our task-level evidence mainly steers the end effector. We also added Häon et al. to Tab 1. Concept-neuron steering gives no reliable directional control (gain 0.22, alignment 0.21) and places no objects.

AuthorsOpenReview ↗

Reply for SZYm [2/2]

[W3. Which tokens are the registers]

"For the pi models the experimental mask appears to be all padding tokens ... the Fig. 12 caption states the recurrence maps are 'distinct from the fixed experimental masks.'"

Our main experiments always used R, the Fig 12 sites, whose coordinates are now listed in Tab 20. The Fig 12 caption's "fixed experimental masks" and A.1.1's "original fixed register masks" were badly worded and read as if the experiments used other masks. "Restricted to padding for the π models" referred to the candidate action registers, which are the whole padding, or a one-token frame for GR00T. Only the attention analyses and the count-matched masking of Fig 17 use that larger set. Note that this clarification does not change the main contribution or claims of the paper.


"Selection is by inspection of heatmaps with no stated criterion."

Sec 4.1 now states the A.1.1 criterion. Per task we keep the 10% most-attended patches (Eq. 6), then the locations shared by all six tasks (IoU 0.61–0.90 against ≈0.05 for random sets, Tab 6). The procedure does not restrict the image region, and the π sites fall in the padding only because the action queries attend there.


"This also makes the masking result close to tautological"

The results are not tautological. We follow A.1.1. Furthermore, high attention alone does not make a token influential. Attention sinks receive the most attention from all queries, yet a residual on π0.5's three sinks steers much less than one on three random register sites (alignment 0.19 vs 0.47, Tab 10). We agree that count-matching (Fig 17) does not control attention mass, which is why the steering controls below are our main evidence.


"steer through the padding tokens of paligemma_diffusion, and compare residuals on the 35 recurrent sites vs all padding"

We added both controls in A.1.3 and A.2 (single-chunk probe, 6 directions × 6 arm poses × 3 seeds). Gains from Tab 9:

π0.5, residual onGain (% of all padding)
All padding (196 tokens)100
Register sites (35)87
Random padding (35)40
Random content (35)36

The 35 sites are 18% of the padding but keep 87% of its gain, while the same number of random padding tokens keeps only 40%, barely more than random content. GR00T shows the same pattern (91% vs 57%). In paligemma_diffusion, which has the same padding but no registers, padding steers at only 0.70× the gain of content tokens, against 1.71× in π0.5 (Tab 13). Constant, writable tokens therefore do not explain the effect.


[Q1] Objectives other than direction

Waypoints are our main test because a 3D target per object comes directly from detections (A.4). We leave grasping to the VLA, which already learned it. The loss can be any differentiable function of the action chunk, though, so we added other objectives in A.10.2 (closed loop on π0.5, 18 samples each, Tab 18):

ObjectiveAverage effect of steering
Direction (our loss)gain 1.68, alignment 0.64 (closed loop, so not directly comparable with Tab 1)
Faster motionchunk travels 3.6× farther than without steering
Gripper closing87% of open gripper commands become closed
Wrist rotationwrist turns by 1.7–7.3°

We also repeated the repulsion study (W1) over four seeds and two tasks (Sec 5.2). Steps inside the region drop from 9.7% to 2.8%, and the closest distance grows from 2.7 to 5.3 cm.

On capacity, one residual for direction and gripper keeps 89% of the direction effect and 87% of the gripper effect, and adding separately computed residuals keeps 94% and 87% (A.10.2). Adding the +x and +y residuals moves the arm along the diagonal at least as well as a residual computed for the diagonal itself. A residual reused at a new arm pose drops to 0.46 alignment (0.68 when recomputed).


[Q2] R is the set of Fig 12 sites, 16+16 for π0, 21+14 for π0.5 and 12+8 for GR00T, and Tab 20 lists their (row, column) coordinates, so R can be reproduced from the paper alone. The OpenVLA sites come from the same procedure on three randomly chosen LIBERO tasks, and the MolmoAct2 sites from the demonstration images of its repository. π0's interior cluster (Fig 13) is not in the top 10% of all six tasks, so it is not part of R. We will also release the index files with the code.


[Q3] Yes, see W2. On top of the decomposed prompts, RegiSteer adds 42.8 OBJ and 50.0 SR points (Tab 2).


[Q4] We added these tests in A.11 (Tab 19). Noisy waypoints (p = 0.021), segmentation waypoints (p = 0.004) and RoboLab (p ≤ 0.018 per policy over 120 tasks) are significant. Tab 1 (p = 0.19, a single-task screen) and LIBERO-PRO (p = 0.46) are not.


Thank you again for the concrete suggestions, which made the paper stronger. We hope you would consider raising your score.

Dear Reviewer,

Thank you again for your thoughtful feedback and suggestions. As the discussion session is about to end, we would appreciate hearing whether our responses have addressed your concerns. We would be happy to clarify any remaining points.

Thank you for your time and consideration.

Dear Reviewer,

Since we are nearly ending the discussion phase, we would really like to hear your response on the new experiments and clarifications, and whether they do or do not address your concerns. Thank you for your consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General overview and clarification on which tokens we steer

We thank all reviewers and the AC for their careful reading. We uploaded a revised PDF with edited and added content in blue. Details are in our reply to each reviewer.

Figure and section numbers are unchanged, and the new A.10-A.12 move the AI disclosure to A.13. Main paper table numbers are changed as 2→3, 3→4, and its appendix tables map as 4→6, 5→7, 6→8, 7→11, 8→12, 9→14, 10→15 and 11→16.


[What the new experiments show]

  • Over the 120 RoboLab tasks, RegiSteer places more objects than IK-EE for every policy (p ≤ 0.018, Tab 19).
  • With noisy waypoints or waypoints from Grounded-SAM 2, it leads IK-EE and a cuRobo planner (Tab 4, Tab 5).
  • Prompting does not explain the gain. One prompt per object adds only 2.4 OBJ points, against 44.0 for RegiSteer (Tab 2).
  • Writable padding does not explain it either. The 35 register sites keep 87% of the padding's steering gain, and 35 random padding tokens only 40% (Tab 9).
  • Astra given the same geometry is comparable, but needs 38 s per decision instead of 177 ms (Sec 5.1, A.9). RegiSteer costs about as much as DynaGuide and FRS, and more than Flow Control (Tab 11).
  • On the real robot, RegiSteer raises MolmoAct2's success from 1/5 and 0/5 to 4/5 and 3/5 with precomputed residuals, and to 5/5 and 4/5 online (Tab 14).
  • New 4-task MolmoAct2 rollouts where initial wrist orientation is not compliant to pick the object, the cases where IK-EE struggles (Tab 14).

[Which tokens we steer] We apologise that the submission described the token sets inconsistently. Sec 4.1 used the symbol R for the candidate set, and A.1.1 and the Fig 12 caption read as if the experiments used other masks. The experiments always used the Fig 12 sites. The caption's "restricted to padding for the π models" described the candidate action registers, and selecting R restricts no image region. The paper finds and tests R in three steps (A.1.1).

  1. Observe. Sec 4.1 finds high-attention, low-information regions and calls them candidate action registers.
  2. Select. A.1.1 keeps the patches among the 10% most attended in all six tasks, using attention alone. These are the action registers R, and all but one GR00T site lie inside the candidate set (A.1.3). Fig 12 draws them and the new Tab 20 lists their coordinates, so R can be reproduced from the paper. We will also release the index files.
  3. Test. Masking R changes the trajectories more than masking content tokens (Fig 4), and a residual on R steers the actions (Tab 9).

The process is not circular. We follow the procedure in A.1.1. Further experiments support our claim:

  • A residual on π0.5's attention sinks reaches only 0.19 alignment, against 0.47 for three random register sites (Tab 10).
  • In paligemma_diffusion, which has the same padding but no registers, padding steers at only 0.70× the gain of content tokens, against 1.71× in π0.5 (Tab 13).

Sec 2 and A.1.3 explain how action registers differ from ViT registers and attention sinks.

This clarification needed no re-runs, since the procedure and the experiments did not change. Please see the corrected Fig 12 caption and A.1.1.


We thank you again for the valuable reviews.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.