acceptodds
Under review as a conference paper at ICLR 2027

The Repair Must Reach the Backbone: A Controlled Comparison of Instruction-Following Fixes for Vision-Language-Action Policies

Abstract

A vision-language-action policy can clear every task in its training suite and still reach for the object its instruction did not name. Repairs exist, and each one arrives on its own base policy, its own demonstrations and its own benchmark, so the field cannot say which ingredient does the work. We hold the base, the demonstrations and the protocol fixed, and compare sixteen arms on one ruler: change one word of the instruction, hold the scene and the robot pose, and record which object the arm reaches, against a random-motion floor. The base reaches the named object in 36% of trials. A swap-differential probe shows that its backbone already picks out the newly named object, so the natural repair is to teach the action expert to read what the backbone computes. That repair does nothing. Training the action expert alone, at either rank, leaves the named-object rate where the base had it, and on the decorrelated demonstrations it also cuts success on the trained tasks by more than half. Given only what the robot observes, no repair that leaves the vision-language backbone frozen passes 53%. Every repair that passes it from the robot's own observations reaches the backbone, and the two that lead decorrelate the instruction from the scene layout: counterfactual relabelling, which edits the word, and low-rank adaptation on position-randomized demonstrations, which moves the object. The second reaches 97.7% across three seeds. Activation patching locates that access: in the pairs the base itself tells apart, swapping the keys and values of six consecutive backbone layers flips its choice 29 times in 32, no other band flips more than once, and the repair keeps that band and makes it more reliable. The recipe then carries off the checkpoint it was tuned on. A second base rises from the same 36% to 77%, a third-party counterfactual benchmark shows the repaired policy following the instruction where the base replays its demonstration, and objects whose name and mesh never occur in training are selected at twice the chance rate. We release the policies, the demonstrations and their generator, and the measurement code.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, instruction following, robot manipulation, controlled comparison, activation patching, imitation learning, dataset confounds

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_zrhkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper compares strategies for improving instruction following in vision–language–action policies. It reports that adapting the backbone with position-randomized demonstrations substantially improves target selection.

Critical strengths
  1. Instruction following is an important problem for reliable robotic manipulation.

  2. The evaluation and ablations appear comprehensive, covering multiple benchmarks, two policy architectures, and physical hardware.

Critical weaknesses
  1. The paper does not consistently respect the ICLR text margins. All tables visibly extends beyond the right margin.

  2. The writing is confusing, making the paper difficult to understand.

  3. Section titles emphasize conclusions rather than clearly describing their contents. Important methodological details are scattered across the main text and appendices.

  4. The claim that backbone adaptation is necessary is stronger than the experiments establish.

Questions for authors

The paper needs substantial reorganization and rewriting. A dedicated methods section should clearly introduce the datasets, adaptation strategies, evaluation metrics, and controls before presenting results. Section titles should describe their contents directly, and all tables should fit within the required margins.

Reviewer_SkfQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why a VLA policy can succeed on familiar tasks while failing to follow changed object instructions. It compares sixteen experimental arms under a shared base policy, demonstration budget, training schedule, and counterfactual evaluation protocol. Action-expert-only updates remain near the baseline on the main counterfactual reaching evaluation, whereas adapting the backbone and action expert using position-randomized demonstrations increases performance from 35.6% to 97.7%. Activation patching identifies a six-layer readout band influencing target selection. Further experiments evaluate transfer to another base policy, new instructions and objects, and a real robot, where the repaired policy reaches the named target in 96/96 trials and lifts it in 90/96.

Critical strengths
  1. A valuable controlled comparison with informative negative results. Holding the base policy, demonstration count, training schedule, and evaluation harness largely fixed makes the differences between interventions easier to interpret. The comparison between confounded and decorrelated demonstrations is particularly useful: allowing backbone adaptation alone gives limited improvement, while combining it with position randomization produces a much larger gain. Reporting damage to original-task performance also prevents an incomplete interpretation based only on target selection.

  2. Careful behavioral measurement complemented by intervention-based analysis. The fixed-scene instruction swaps, camera and nonsense controls, explicit separation of selection from task completion, and per-seed counts strengthen the evaluation. Key/value patching provides evidence beyond attention visualization alone, identifying an influential information pathway on the same checkpoint that is subsequently adapted. The second-backbone and held-out-object experiments further support the practical relevance of the training recipe.

Critical weaknesses
  1. The claim that repair must reach the backbone is stronger than the demonstrated causal conclusion. The experiments show that the tested expert-only adaptations fail under the selected optimization regimes; they do not establish that a frozen backbone cannot support a successful repair with a different readout or optimization regime. Moreover, the successful observation-only adaptation recipes jointly update the backbone and action expert, leaving their individual contributions incompletely separated. The patching study identifies a pathway influencing target choice, but is conditioned on pairs already distinguished by each model: 32/80 for the base and 78/80 for the repaired model, and therefore does not directly explain how previously unresolved pairs become solvable.

  2. Some transfer results do not isolate the recipe evaluated in the main controlled comparison. The LIBERO-CF checkpoint includes an additional region-supervision head absent from Table 1. The physical baseline is unadapted to the hardware and task, as the limitations section acknowledges. These experiments show that the resulting systems can perform well, but provide weaker evidence about the independent contribution of the proposed repair.

Questions for authors
  1. Does the frozen-versus-adapted-backbone comparison also hold on π0.5, beyond transferring only the successful recipe?

  2. Can cross-checkpoint component or activation interventions separate changes to the backbone from changes to the action expert?

  3. What does the Table 1 position-randomization checkpoint achieve on LIBERO-CF without the additional region-supervision head?

Reviewer_oHQLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares instruction-following repairs under a shared policy and evaluation protocol. Updating the backbone together with training on position-randomized demonstrations produces the strongest improvement. Activation patching identifies a layer band through which instruction changes affect action selection.

Critical strengths
  • Holding the scene fixed while changing the requested object exposes failures hidden by ordinary task success. Comparing repairs within the same framework makes the results useful.
  • The main improvement is large and repeated across seeds: target selection rises from 35.6% to 97.7%. The second model and additional evaluations provide evidence beyond the primary checkpoint.
Critical weaknesses
  • “Must reach the backbone” is stronger than the experiments establish. Failure of the tested expert-only updates does not rule out other ways of reading the frozen representation. The second model also lacks the corresponding expert-only control.
  • Activation patching identifies an instruction-sensitive pathway, but does not establish which changes cause the repair. Both models use the same band, and the analysis conditions on pairs they already distinguish. Would training only that band, or freezing it during repair, clarify its role?
Questions for authors

See weaknesses.

Reviewer_GfakOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a vision–language–action policy that finishes the LIBERO object tasks it was trained on while often reaching the object that usually sits in the target position rather than the object named in the instruction. The base is MolmoAct2-Think-LIBERO. The measurement, SWAP-OBEY, keeps the scene and the initial robot pose fixed, changes the object name, and scores whether the end-effector comes within 2 cm of the newly named object.

The main experiment crosses two factors on that one base, with a shared budget of 900 demonstrations and 3000 updates. One factor is whether the update can change the vision–language backbone. The other is whether the demonstrations still let position predict the target. Training only the action expert, including on position-randomized data, stays at the base rate of about 36%. The same rank-32 update on a frozen layout reaches 44.6%. The same update on position-randomized demonstrations reaches 97.7% across three seeds, and a relabeling variant reaches 81.7%. Key/value patching on this checkpoint places the action expert’s read of object identity in backbone layers 18–23. The winning recipe is then moved to π0.5 (36% to 77%), to a third-party counterfactual benchmark, to held-out objects and phrasings, to RoboCasa, and to a real arm.

Critical strengths

The two-factor comparison is the reason to take the paper seriously. Prior repairs are usually reported on different bases, data, and benchmarks. Here the base, the demonstration budget, the schedule, and the instrument are shared, and the two factors are crossed. Neither factor works alone: position randomization with a frozen backbone stays at 35.8%, and a backbone update on confounded demonstrations reaches only 44.6%. Only the combination reaches 97.7%. That pattern is what the printed counts support, and it is consequential because it separates “randomize the positions” and “fine-tune something” from the claim the experiments actually establish.

The action-expert-only controls also make the result hard to dismiss as overfitting or as a generic gain from more training. On the same randomized demonstrations, updating only the action expert drops original-task success from 198/200 to 71, 37, and 35 while leaving named-object reach at the base rate. The winning recipe keeps original-task success near that ceiling. The standard suite being solved is the premise of the measurement, not evidence against it.

Critical weaknesses

The title, abstract, and conclusion state a more general law than the experiment. The title says the repair must reach the backbone. The conclusion says every frozen repair stopped at the base. Table 1 contradicts that wording: counterfactual guidance leaves the backbone frozen and scores 85/160 against the base’s 57/160. The abstract also says the demonstrations were held fixed, while Section 5 shows that the demonstration distribution is the second decisive factor. The full cross was run on one base, MolmoAct2-Think-LIBERO, and every backbone update is rank 32. π0.5 receives only the winning recipe. Its action expert does not read the backbone through per-layer cross-attention, and the read-out band was not measured on it. The limitations section is more careful than the title. As written, the paper asks the reader to accept a general claim about vision–language–action policies from one architecture and one adapter rank.

The objects named by the main claims are not stable. “Repaired policy” covers the three Table 1 seeds, retrains on π0.5 and on the real arm, and a released checkpoint that adds an auxiliary region head no Table 1 arm has. The LIBERO-CF numbers and the stock-benchmark counts of 27/200 and 180/200 come from that extra-head checkpoint. Section 4 never says which checkpoint was patched. The arms that set the frozen-backbone ceiling, including guidance and the oracle jump from 50% to 96.9%, are specified by a citation and a scalar. The submitted SWAP-OBEY harness scores a 5 cm planar comparison rather than the paper’s 2 cm rule on the 160-trial subset. Until those identities are fixed, the 97.7% cell is readable and the broader claims are not.

Questions for authors

Which checkpoint was patched in Section 4 and Figure 3, and which checkpoint produced the LIBERO-CF numbers and the 27/200 and 180/200 counts? Please say whether each one is one of the three Table 1 seeds or the released model with the auxiliary head, and whether that head, or any added tokens, is used at inference.

Which program and which distance rule produced Table 1? The paper defines success as coming within 2 cm of the named object’s origin on 160 trials. The submitted libero_pro_eval.py scores a minimum xy distance under 5 cm, requires that distance to beat the original target, and does not drop those 40 trials. If Table 1 came from a different harness, please point to it.

For the π0.5 base, please report the per-seed named-object counts. The pooled rate printed in Table 2 is the same 57/160 reported for MolmoAct2. An independent count would support the second-base result. A copied cell would not.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.