- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper identifies “action registers”: visually uninformative image tokens that receive high action attention and strongly influence VLA outputs. It proposes RegiSteer, which optimizes test-time residuals on these tokens to guide a frozen policy toward 3D waypoints while retaining learned manipulation behavior.
Experiments cover three generalist policies on 120 RoboLab tasks and OpenVLA in LIBERO-PRO, showing improved placement and task-dependent success gains. Ablations examine register influence, alternative steering channels, guidance noise, and runtime. Small-scale physical-robot demonstrations use precomputed directional residuals to verify the effectiveness.
Critical strengths
-
Clear motivation and training-free adaptation. The paper addresses the practical need to redirect pretrained VLAs toward new spatial goals while retaining their manipulation skills. RegiSteer keeps policy weights frozen and optimizes only residuals on selected visual tokens, avoiding additional policy training and reducing adaptation costs. This makes the approach convenient to apply to existing policies.
-
Evaluation across multiple policy families. Experiments cover three generalist policies on 120 RoboLab tasks and an autoregressive OpenVLA policy under object relocation in LIBERO-PRO. The placement improvements and recovery of OpenVLA success from 0% to 65% provide meaningful evidence that the approach can redirect existing manipulation skills across different action-generation architectures.
Critical weaknesses
-
Substantial online inference overhead. Online RegiSteer requires two forward generations and one backward pass, increasing measured latency by approximately threefold—for example, from 57 ms to 177 ms per action chunk for on an H200. This weakens its practical appeal for responsive robot control. Comparisons under matched latency or control-frequency budgets are needed to establish whether the performance gains justify this cost.
-
Limited real-world validation. Physical experiments consist of small-scale demonstrations using precomputed directional residuals, rather than the online gradient-based method evaluated in simulation. The paper lacks systematic real-robot task-success comparisons against strong baselines across diverse tasks. Consequently, it remains unclear whether the simulation gains and claimed preservation of manipulation behavior translate reliably to physical deployment.
Questions for authors
- Given the approximately threefold online inference cost, does RegiSteer retain its advantage when competing methods operate under matched latency or control-frequency budgets?
- Has the online gradient-based variant been tested on physical robots? What evidence supports its practical effectiveness beyond the precomputed-residual demonstrations?
- How does RegiSteer compare with directly optimizing a residual on the final action chunk using the same waypoints and kinematic objective?
- Across the 120 RoboLab tasks, what proportions improve, remain unchanged, or deteriorate? Are gains concentrated in particular task types?
- What exact procedure determines the fixed steering masks, and were the tasks used for mask selection separate from those used for evaluation?
5 comments
Thank you for the additional experiments. The latency-aware simulation shows that the placement gains persist despite the inference overhead in the tested settings, and the action-residual baseline directly addresses my question about modifying the final action chunk. The new SO-101 trials also demonstrate that the online method can work on a physical robot.
My main remaining concern is how reliably those gains transfer across physical tasks. The new RoboLab breakdown shows improvements across several task categories, but it does not tell me whether the gains mainly come from tasks where correcting the approach position is enough. Could you report success rates over repeated trials on a few more real-robot tasks or scene layouts, comparing cases where an object is simply relocated with cases where its pose changes and the robot needs a different wrist orientation or grasp? A comparison with the base policy, and with IK-EE using the same waypoints if feasible, would help clarify when RegiSteer is useful in practice.
If the added trials show consistent gains across a broader set of physical tasks, I would consider raising my score.
A few more real-life robot tasks
Thank you for your feedback, we appreciate it. We are happy that the questions are directly addressed.
[A few more real-robot tasks]
"Could you report success rates over repeated trials on a few more real-robot tasks (...) comparing cases where an object is simply relocated with cases where its pose changes and the robot needs a different wrist orientation or grasp? A comparison with the base policy, and with IK-EE using the same waypoints"
Thank you for the recommendation. As you said, we added a few more real-robot tasks (four SO-101 tasks with five trials each), where the task success would need different wrist orientations from the initial position. IK-EE and RegiSteer use the same waypoint. Please see visuals and numbers in Tab 14, App A.5 (page 28-29).
| Task | MolmoAct2 | + IK-EE | + RegiSteer |
|---|---|---|---|
| Marker | 0/5 | 0/5 | 2/5 |
| Granola bar | 2/5 | 0/5 | 4/5 |
| Bottle | 3/5 | 0/5 | 4/5 |
| Pill bottle | 3/5 | 5/5 | 5/5 |
| Total | 8/20 | 5/20 | 15/20 |
The pill bottle can be grasped from above at any wrist angle, allowing both IK-EE and RegiSteer to succeed in all five trials. The other three tasks require a change in wrist orientation. In these tasks, IK-EE retains the initial wrist orientation and fails, whereas RegiSteer allows MolmoAct2 to rotate the wrist as it approaches the object.
"If the added trials show consistent gains across a broader set of physical tasks"
RegiSteer matches or beats both methods on every task, and its gain over IK-EE is significant (p = 0.004). It also improves on MolmoAct2 alone in all six physical tasks, including the two from our first reply.
[Scope clarification]
We would also like to clarify the scope of our contribution. Our main contribution is identifying a novel property of VLAs and using it as an effective steering channel, while preserving their learned manipulation capabilities. We do not try to replace inverse kinematics in robotics as a whole.
We include IK-EE as a classical baseline to assess whether the same waypoint guidance can be used effectively through direct end-effector control. Our experiments highlight its limitations when the wrist orientation is unsuitable for grasping the object, or when noisy waypoints require correction from the VLA's learned visual priors (Sec 5.2, A.5). A more elaborate IK-based pipeline could incorporate perception, grasp planning and other sub-components to operate without a VLA (Sec 4.1). Such a pipeline represents a different approach to manipulation, and falls outside the scope of our work, which focuses on steering VLAs. Nevertheless, we still compare with a collision-free planner that hands off to the VLA, and RegiSteer leads it with noisy or perceived waypoints (Tab 5).
Thank you for making our submission stronger. We would appreciate if you would reconsider your score.
Reply for wjnM [1/1]
Thank you for the constructive review. We are glad you find the motivation clear, the training-free approach convenient, and the results across policy families meaningful. All additions below are in the revised PDF, edited and new content in blue.
The general comment lists the new table numbers and explains which tokens we steer.
[W1, Q1]
We added the runtimes of Flow Control, DynaGuide and FRS to Tab 11 (A.1.4). Latency per query:
To make each method pay for its own latency, we ran π0.5 on a separate set of RoboLab tasks with noisy waypoints and the simulator clock running during inference. Objects placed per episode (Tab 11, bottom):
RegiSteer places the most objects at both noise levels, so its extra latency doesn't erase its lead, and at σ = 8 cm its lead over IK-EE is significant (p = 0.02, Tab 19). IK-EE tracks the noisy target blindly, while RegiSteer only biases a policy that still sees the current images (Sec 5.2). On the real robot, where the steering directions are known in advance, precomputed residuals cut the extra cost to about 0.1% (Sec 5.3).
[W2, Q2]
We agree that the robot part is a demonstration, and Sec 6 says broader physical evaluation is needed. It shows that the register channel can be steered in a different VLA (MolmoAct2) on a different robot (SO-101). We added the online gradient variant on the SO-101 in A.5. From Tab 14:
We also added task success over five trials per task (Tab 14, bottom). For the online variant we set the waypoints ourselves, relative to the end-effector position.
Both variants raise success well above MolmoAct2 alone, and the online variant does slightly better (A.5).
[Q3]
We added it to Tab 1 ("Residual δ to actions"). With the same loss it moves the arm strongly (gain 9.12, alignment 0.903) but places only 0.50 objects, against 2.46 for RegiSteer and 2.08 for IK-EE. The policy does not see an action residual, so its next chunk pulls back toward its own plan. IK-EE avoids this by replacing the actions during transport, which also removes the visual feedback. RegiSteer edits the policy's input instead, so each new chunk carries the correction.
[Q4]
Counts over the 120 tasks in which OBJ improves, stays the same or gets worse against the base policy (A.11):
Most unchanged tasks are ones where neither the base policy nor RegiSteer places an object. By group, OBJ improves in all 18 groups of Tab 15 and success in 13.
[Q5]
It is the A.1.1 procedure, now also stated in Sec 4.1, run once per policy and camera view. We average action-to-image attention over one episode of each of six RoboLab tasks, keep the 10% most-attended patches per task (Eq. 6), and keep the locations shared by all six. This gives the Fig 12 sites, which all main results use, and Tab 20 lists their coordinates.
Thank you for the helpful questions. We would be glad if you would consider raising your score.