acceptodds
Under review as a conference paper at ICLR 2027

ManipScope: A Diagnostic Benchmark for Instruction Following and Generalization in Robotic Manipulation

Abstract

Completing a manipulation task does not establish that a robot follows the requested goal rather than a familiar routine. Our proposed ManipScope is an evaluation-only benchmark for robotic instruction following under changed task requirements. Built using RoboTwin 2.0’s simulation resources, it specifies 48 scenes across six task families with explicit source-task links. We deliberately provide no expert training demonstrations for these scenes, targeting skill reuse under changed requests rather than fitting supplied examples. Task-success and selection metrics measure completion and satisfaction of the requested properties. Controlled instruction and layout changes test whether choices track the request, with future-video and internal-intervention probes where access permits. Evaluations of vision–language–action policies, world-action policies, and agents reveal uneven task performance, partial instruction satisfaction, and correct selection without completion. In controlled checkpoint case studies, individual-trial success can coexist with weak paired instruction correctness; forecasts and execution can agree on the wrong target; and internal interventions change predicted actions without establishing useful goal transfer. These findings show why success, consistency, and sensitivity must each be checked against the requested goal.

Primary area:
datasets and benchmarks
Keywords:
Robotic Manipulation, Instruction Following, Benchmarking, Vision-Language-Action Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7bSBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ManipScope, a benchmark designed to evaluate robotic manipulation capabilities, specifically focusing on performance when instruction requirements change. Built upon RoboTwin 2.0, the benchmark encompasses 48 scenarios across six task categories; while these scenarios are clearly linked to source task components, no expert demonstration data is provided for training. The study evaluates twelve policy checkpoints and two agents capable of tool use. Beyond measuring task success rates, the research examines target selection during initial interaction and the extent to which instruction attribute requirements are met. Additionally, the study conducts several controlled experiments, including cross-combining target instructions with object layouts, analyzing the consistency between predicted targets in generated videos and actual execution targets, and substituting internal representations within the video generation strategy. Key findings indicate that robots may select the correct target during the initial phase even if the task is ultimately not completed; predictions and execution outcomes may align on an incorrect target; and changes in action resulting from interventions do not necessarily demonstrate goal-oriented transfer.

Critical strengths
  1. This is an evaluation resource of high practical value that encompasses a wide range of experimental scenarios. ManipScope provides 48 scenarios linked to original tasks—covering various forms of task requirement changes—and includes detailed success criteria, evaluation interfaces, and evaluation results for each task. By evaluating 12 policy checkpoints across 100 episodes per scenario, this resource establishes a robust benchmark for future research. Its evaluation-only design facilitates the study of existing systems without requiring modifications to accommodate the demonstration data provided for new scenarios. Regardless of whether a paper proposes a new policy architecture, this resource contribution is significant in its own right.
  2. The paired-behavior study offers a rigorously controlled case that compellingly demonstrates why isolated success metrics are insufficient. Testing combinations of two valid goal instructions against two object layouts yields far richer information than simply measuring whether actions change in response to input perturbations. In the color-selection scenario tested, Motus followed both instructions simultaneously in only 1 out of 40 paired cases sharing the same layout, whereas π0.5 achieved this in 29 out of 40 cases. A configuration-level bootstrapping approach preserved the correlations between paired conditions. Furthermore, the authors appropriately distinguished between "goal consistency" and "instruction correctness" and avoided misinterpreting "sensitivity to open-loop interventions" as a substantive behavioral improvement. These design choices and interpretive strategies make the case study highly persuasive within its scope.
Critical weaknesses

1. The benchmark fails to clearly distinguish between "following instructions" and "executing tasks."

In Appendix A.2 (lines 701–703), the "three-colored block" task always requires selecting the red block. Consequently, a strategy that simply selects the red block can achieve a high score without actually following the instruction. Although a paired test addresses this issue, its scope is limited to a single task and two checkpoints (Section 5.3, lines 376–377). Furthermore, the relationship between source and target tasks does not verify whether the model has mastered the underlying skills or whether it encountered similar combinations during training (Section 3.1, lines 155–157). This makes the benchmark results difficult to interpret: did the model fail because it misunderstood the instruction, or because it was unable to execute the task?

2. Some diagnostic metrics lack sufficient validation.

As shown in Figure 6, in the "bell" task, all models scored 0/5 on the "first target selection" metric, even though Motus and GPT-6 achieved a task success rate of 5/5. As noted in Appendix D.3 (lines 1456–1457), task success does not necessarily trigger the "contact event" upon which this metric relies. This raises concerns about whether the metric accurately reflects selection errors. Internal intervention analysis also has limitations: only two interpretable "donor–recipient" pairs predicted different targets, and both pairs originated from the same initial configuration (Appendix E.3, lines 1663–1665). Therefore, beyond demonstrating that predicted actions do indeed change, it remains unclear whether these experiments provide meaningful evidence of "goal-directed behavior."

Questions for authors

1. Does the benchmark truly measure instruction-following capability?

In Appendix A.2 (lines 701–703), the three-colored-block task always requires selecting the red block; consequently, a model can succeed simply by choosing the red block, even without following the instruction. Aside from the pairwise experiments conducted on a single task and two checkpoints, is there evidence that the models' performance across the 48 scenarios actually depends on the instructions rather than on fixed scenario preferences?

2. How reliable is the "first selection" metric across different tasks?

As shown in Figure 6, all models scored 0/5 on "first target selection" in the bell task, even though Motus and GPT-6 achieved a task success rate of 5/5. Given that a successfully executed action does not necessarily trigger the required contact event (Appendix D.3, lines 1456–1457), is it possible that this metric misclassifies successful behaviors as selection failures? Would excluding such tasks alter the primary conclusions regarding attribute-level performance?

4 comments
AuthorsOpenReview ↗

Reviewer 7bSB, part 1 (W1, Q1)

We sincerely thank you for the thorough review, with its precise references to the paper, and for recognizing the practical value of the benchmark and the care of the paired study. This is the first of two comments: W1/Q1 here, W2/Q2 in the next.

W1 / Q1. The three-color scene always asks for red, so a fixed preference scores. Does performance depend on the instruction?

Response. Thank you for this important question. Yes: ManipScope measures whether a policy understands the instruction{\color{#1f6feb}\text{Yes: ManipScope measures whether a policy understands the instruction}}, and a fixed scene preference cannot score across it. Measured this way, most checkpoints do not follow the instruction, and two do. The evidence is at three levels: across scenes (1), within one scene (2), and with execution ruled out (3); (4) relates this to existing benchmarks.

(1) Scenes with opposite requirements (as in the submission; numbers new). You are right that the three-color scene asks for red. The benchmark also contains the opposite request: a scene that asks for "the block that is not red", which no preference for red can pass.{\color{#1f6feb}\text{a scene that asks for "the block that is not red", which no preference for red can pass.}}

  • The opposite pair. The same twelve checkpoints succeed in 70.2% of episodes in the three-color scene and in 10.1% in the "not red" scene (Table 19).
  • Further pairs. The suite contains several such pairs: one preference yields 77.0% success where the scene asks for the smaller block and 1.2% where it asks for the block on the plate.
  • Requested value not fixed. In the bottle scene the requested color (brown or green) is drawn at random in each episode, so no single color can be preferred into success (App. A.2).

A preference therefore cannot produce a high score across the benchmark; it shows up as exactly this contrast.

(2) Instructions exchanged within one scene (extended in the revision). Target reassignment replays each frozen initial state under instructions that name each candidate in turn, so a fixed preference is counted as fixed, not as success. The submission ran this on the scene you cite with two checkpoints; the revision covers twelve checkpoints and ten selection scenes with 25 instructions (Sec. 5.3, Table 3).

SystemStates on which every instruction is followed
Ten policy checkpoints3.9% (39/1,000)
Instruction-blind chooser16%
Positive control (GPT-6 agent)88% (44/50)

The positive control shows that the requests are solvable and how well they can be followed, and the test discriminates: LingBot-VA and π0.5 do follow color (10/10 and 6/10 states of the three-color scene). The paper now reads the success of a selection scene together with this result, not alone.

(3) Misunderstanding, not inability to execute (new experiment). In a new run of the source tasks under training instructions in the same evaluator, nine of ten checkpoints complete at least nine of the ten linked source tasks in 70% or more of episodes (Table 30). The tested combinations are absent from the RoboTwin demonstrations: none of the 48 scenes is a RoboTwin task, and each RoboTwin block task has one fixed goal (Sec. 3.1, App. G.1).

(4) Scene preference in existing benchmarks. The concern you raise is an important one, and it does arise elsewhere. In LIBERO-based benchmarks such as LIBERO-PRO, the object layout of a task barely varies (2-5 cm), so a policy can rely on the scene: in our new probe on LIBERO-Object, policies follow a changed instruction only when the named object sits at a trained target position (99-100% of runs there, 49/400 and 0/400 elsewhere; Table 32).

Changes made. Sec. 3.1 (Remark), App. A.2, Sec. 5.3 (Tables 2-3, Figure 2), App. G.1-G.3.

AuthorsOpenReview ↗

Reviewer 7bSB, part 2 (W2, Q2)

W2 / Q2. Are the diagnostic metrics validated? Do they measure what they are claimed to measure? (Bell task: 0/5 on first selection but 5/5 success.)

Response. Thank you for this careful observation. On the tasks where we use it, the selection metric measures the choice of object{\color{#1f6feb}\text{On the tasks where we use it, the selection metric measures the choice of object}}, and this is validated in three ways, two of them new in the revision. It does not apply to tasks that are completed without a loaded contact, such as the bell task you found; the revision removes these tasks from selection statements and reports how the numbers change.

(1) Validation of the selection metric.

CheckResult
Force threshold varied from 0.1 to 1 N (as in the submission)Correct-target counts change by at most 3 of 35 per system
Stricter event (new): commitment, the first candidate displaced by at least 5 cmEpisodes that fail after a correct choice: 47/210 by first contact, 39/210 by commitment
Independent record of the gripper state (new experiment, 2,400 episodes)A non-target is grasped in 83% of failed episodes and in 5.8% of successful ones, so a recorded wrong selection is a wrong grasp, not a touch

The metric also registers correct behavior where it occurs: under target reassignment, LingBot-VA follows all three color instructions on 10/10 states.

(2) Where the metric does not apply: your bell example. You are right: in this task the metric missed successful behavior.

  • What happens. The press is judged from the position of the gripper, and no loaded event is recorded on the bell in any of the 35 episodes, 26 of which succeed (new Figure 9 shows one such episode: the gripper presses the bell, success is reported, and the recorded force is zero).
  • How it was counted. The submission did not count these episodes as wrong selections (it reported them as no event, App. D.3), but they entered the denominator of the attribute rates. The same holds for the two hammer tasks.
  • What the revision does. It restricts selection statements to the six pick-type tasks (new analysis, App. D.5), with this effect:
Attribute correctnessAll tasksWithout the task that has no event
Category53.3%80.0% (without bell-press)
Color65.7%73.3% (without color-hammer)
Size43.8%52.9% (without size-hammer)
  • What changes. The low category rate of the submission came from the bell task, and we no longer state it.
  • What does not change. Size remains the lowest attribute, and a correct choice does not guarantee completion (47/210, or 39/210 under commitment).

Changes made. Sec. 4.1, Sec. 5.2, App. D.5 (new Figure 9).

W2, second half. Is the evidence from the intervention enough to support goal-directed behavior? (Two readable pairs, one configuration.)

Response. We agree with your assessment: the evidence is not enough for that claim, and the paper does not make it. The intervention is reported as a negative result: the predicted actions change, but not toward the goal. The paper's statements about following a goal rest on behavior, not on this analysis.

(1) What the evidence shows (as in the submission).

MeasureFuture representation replacedFull instruction changed
Change in predicted joint commands0.030 rad0.096 rad
Shift of preference toward the donor's goal0.560 mm, 95% interval [−0.019, 1.299]2.529 mm

The actions respond to the intervention, but the interval of the shift toward the goal includes zero.

(2) What this supports, and what it does not. It supports one methodological point: a change in actions is not evidence of goal following, so the direction of the change must be evaluated. It does not show goal-directed behavior, and it does not show the absence of goal influence either: only two readable contrasts predict different targets, both from one configuration, and no closed-loop run is made. The paper states these limits (Sec. 5.3, Sec. 6, App. E.3).

(3) Where goal following is tested with enough evidence. Behaviorally: target reassignment scores which object is moved under each instruction, on 1,000 initial states for ten checkpoints (extended in the revision; see W1 in our first comment). The conclusions about instruction following come from there.

Changes made. No new data for the intervention. The main text now carries one sentence with these numbers (Sec. 5.3); the full paragraph is in App. E.3.

We hope the new results and the revised paper address your concerns, and we would be glad to clarify any remaining point. Thank you again for your time.

i've fully read the author's supplement information, thank you for more experimental and impressive information, base on information above, i will kept my original score as weak accept

AuthorsOpenReview ↗

Thank You, Reviewer 7bSB!

Thank you for taking another careful look at our work! We’re glad you found the additional experiments helpful, and we really appreciate your thoughtful feedback and continued support.

Reviewer_ruEKOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ManipScope, a diagnostic benchmark for evaluating instruction following and generalization in robotic manipulation. It includes 48 scenes across six task families and evaluates different VLA, WAM and agent systems. Beyond task success, the authors introduce target selection metrics and several diagnostic tests to better understand whether models actually follow instructions. The experiments reveal some intresting findings about the limitations of current robotic models.

Critical strengths
  1. I think the motivation is quite good. Task success does not always mean that a robot really understands and follows the instruction. The paper tries to seperate instruction understanding from actual task execution, which is a meaningful problem. I also like the design of changing task requirements while reusing existing manipulation skills. This is a good prespective for evaluating VLA models.

  2. The benchmark is relatively comprehensive, with 48 scenes, multiple task families and a large number of evaluated models. The experiments are quite solid, and the authors provide detailed analaysis beyond simple success rates. Especially, the paired instruction experiments and the distinction between correct selection and task completion are quite intresting. I think these findings are useful for future reserach.

Critical weaknesses
  1. The generalization evaluation could be more convincing. Although the authors construct new tasks based on existing skills, it is not always clear whether the evaluated models have actually learned these skills during pretraining. If a model fails on a new task, it may simply lack the required manipulation ability rather than having poor instruction understanding. I think this distinction needs more careful analaysis, and some direct comparisions with the original tasks would be helpful.

  2. Some comparisons are not fully fair. Different VLA models and agents use different training data, control interfaces and inference settings. The agent evaluation also uses fewer episodes than the policy evaluation. Although the authors discuss these differences, I think it is still difficult to directly compare their performace. More controlled experiemnts would make the conclusions more convincing.

  3. The diagnostic experiments are still somewhat limited. The detailed instruction-pairing and world-model analysis mainly focus on a few models and specific tasks. It is unclear whether these findings can generalize to other tasks and architecutres. Also, all experiments are conducted in simulation, without real-world robot validation. I think more diverse settings would further strenghten the benchmark.

Questions for authors

No more further questions

2 comments
AuthorsOpenReview ↗

Reviewer ruEK, part 1

We sincerely thank you for the careful reading and for the encouraging assessment of the motivation, the benchmark design and the experiments. Each of your three concerns led to a new experiment or analysis in the revision. This is the first of two comments: W1 and W2 here, W3 in the next.

W1. Have the models learned the required skills? Compare with the original tasks.

Response. Thank you for this suggestion. The direct comparison with the original tasks is now a new experiment in the revision, and it shows that missing ability does not explain the failures{\color{#1f6feb}\text{missing ability does not explain the failures}}.

CheckResultWhat it rules out
Original tasks (new experiment). Every checkpoint runs the RoboTwin source tasks linked to the probed scenes, under training instructions in the same evaluator (Table 30)Nine of ten checkpoints complete at least nine of the ten tasks in 70% or more of episodesThe skills were not learned
Evaluation scenes (new analysis). What the policy does in failed episodes (Table 19)84.6% move a distractor; only 12.4% move nothingThe policy cannot act in the new scene
Doubtful cases (new analysis). Without the 6 of 100 checkpoint-scene cells whose source task is below 30%Every instruction is followed on 3.1% of states, against 3.9% with themThe result rests on weak cells

Conclusion. The policies have the skills and use them in the new scenes; they apply them to the wrong object.

Changes made. Sec. 5.2-5.3 (new Table 2, the diagnostic chain), App. D.6 (Table 19), App. G.3 (Table 30).

W2. Comparisons are not fully fair (training data, interfaces, inference settings; fewer agent episodes).

Response. We agree that agents and policies differ in training and interface, which is why the benchmark keeps two leaderboards, not one. Where the two are compared, both use the same five episodes per scene. The agent serves first as a positive control for the policies' instruction following, not as a competitor, and a new feasibility test shows that the two can be combined.

(1) Two leaderboards, because the systems differ (as in the submission).

  • Agents and policies. Agents call tools and are not trained on RoboTwin; policies are. They are therefore ranked separately (Sec. 1).
  • Among the policies. Conditions are matched: same embodiment, same initial states, 100 episodes per scene. Nine of the twelve report identical training data, and restricted to these nine the results are unchanged{\color{#1f6feb}\text{restricted to these nine the results are unchanged}} (new analysis: every instruction followed on 4.3% of states, against 3.9%; App. B.1).

(2) The agent-policy comparison uses equal episodes. Agents run five episodes per scene. The comparison therefore takes the policy's first five episodes, with the same seeds, so both sides have five episodes on the same initial states; the revision adds 95% intervals.

240 shared episodesSuccesses95% interval
GPT-6 agent185 (77.1%)71.4-81.9
G0.5, the best policy107 (44.6%)38.4-50.9

The intervals do not overlap, so five episodes per scene support this suite-level statement.{\color{#1f6feb}\text{The intervals do not overlap, so five episodes per scene support this suite-level statement.}} On the 156 episodes with identical instruction text the gap remains (77.6% against 36.5%; App. B.5, B.6).

(3) What the agent is for: a control first, a component second.

  • A positive control, not a competitor. In the diagnostics, the GPT-6 agent is the reference against which the policies' instruction following is judged: it follows every reassigned instruction on 88% of states, where ten policies do so on 3.9% (Sec. 5.3, Table 3). The comparison is therefore not a ranking. The agent has its own cost: 13.2 s of model time per call, against 1.3 s for G0.5 (Table 11).
  • A component to combine with policies (new feasibility test). The diagnostics show complementary strengths: the agent selects the right target, and the policy executes. In a simple handover, GPT-6 selects the object and brings the gripper to it, and the policy DM0.5 performs the grasp. Over three scenes, the states on which every instruction is followed rise from 0/30 to 16/30, and successes from 35/80 to 66/80 (App. G.4; table in our reply to aZif, W4). This is only a feasibility test; we hope the diagnostics help others build such combinations.

Changes made. Sec. 5.1 and Table 1 (two checkpoints marked), App. B.1 (matched training data), App. B.6 (Table 11), App. G.4 (handover experiment).

AuthorsOpenReview ↗

Reviewer ruEK, part 2 (W3)

W3. The diagnostics cover few models and tasks; simulation only.

Response. We agree that broader coverage makes the findings more convincing. The revision extends the two diagnostics that apply to every policy to all twelve checkpoints, and the findings hold in both architecture families. The forecast and intervention probes are policy-specific tools, released with the code for users of the benchmark. Simulation is what makes this controlled comparison possible; real-robot validation remains future work. These three parts are detailed in (1) to (3).

(1) Diagnostics that apply to every policy are extended (new experiments).

DiagnosticSubmissionRevision
Instruction changed on a frozen state2 checkpoints, 1 scene12 checkpoints on 10 scenes (Table 3); also 3 training tasks (Table 29: 12 checkpoints on the two color tasks, 10 on the size task)
First-decision sensitivity (Figure 13)2 checkpoints12 checkpoints

The finding generalizes, with an informative exception: seven checkpoints follow every instruction on at most 2% of states, while LingBot-VA (world-action) and π0.5 (vision-language-action) follow color instructions.

(2) The forecast and intervention probes are policy-specific (as in the submission). They read a policy's predicted video or internal representations, so they exist only for models that expose these and must be adapted to each model. We therefore offer them as protocols: the released code, updated with the revision, includes them so that users of the benchmark can run them on their own policies.

(3) Why simulation. Simulation makes the comparison controlled. Many open checkpoints trained on RoboTwin can be compared side by side (twelve here, nine with identical training data), and one initial state can be replayed exactly under different instructions, which the diagnosis requires. Real-robot validation remains future work (Sec. 6).

Changes made. Sec. 5.3 (Table 3), App. E.2 (Figure 13), App. G.2-G.3.

We hope the new results and the revised paper address your concerns, and we would be glad to clarify any remaining point. Thank you again for your time.

Reviewer_aZifOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents ManipScope, a benchmark for testing instruction following and generalization in robotic manipulation. Besides task success, it records which object the robot touches first. It also includes controlled probes that vary the instruction and the object layout. It finds that correct initial target interaction does not always lead to task completion, while initial non-target interaction can still lead to success.

Critical strengths

The motivation of studying the instruction following of robotics policies is strong. The detailed analysis of success and failure cases makes the evaluation more useful to the community than reporting aggregate success rates alone. I particularly think the analysis of non-target/success being interesting, which are often overlooked.

Critical weaknesses
  1. My major concern is the lack of training samples on the benchmark. Without this, its hard to decouple the failure due to out of distribution generalization and failure due to instruction following. The missing of training samples could make this benchmark invalid for the analysis it intends to do.
  2. The novelty of this benchmark is limited. It looks like many of these instruction and layout probes could be run in LIBERO-Pro or LIBERO-Plus. What makes a new benchmark necessary? A direct comparison would help, showing what the tasks and diagnostics add beyond applying the same probes to existing environments.
  3. The non-target/success is interesting but its unclear whether the non-target is because the policy decided on the wrong object or because the policy has bad collision avoidance and touches obstacles by accident.
  4. The main conclusions felt somewhat expected to me. Succeeding at a task doesn’t mean the policy followed the instruction, and predictions and actions can agree while both being wrong. The experiments show these points clearly, but they say little about why the failures happen or how to fix them.
Questions for authors
  1. How do you decouple the failure due to generalization to out of distribution scenes and the failure due to true instruction following?

  2. Can existing benchmark such as LIBERO be used for the same instruction following analysis?

===

Post-rebuttal

Thank you for the detailed rebuttal and additional analyses. The new controls substantially address my concern about decoupling scene generalization from instruction-following failures, and I will raise my score to 2.

My remaining concern is the significance of the contribution. The joint diagnostics are useful, but the benchmark still appears to be an incremental extension of existing instruction-probing work such as LIBERO-CF. The additional analyses document the failures more clearly, but I feel it still provides limited new insight into instruction following.  Since the source tasks have fixed goals, their training instructions doesn’t have to distinguish among objects. The finding that policies prefer demonstrated targets is therefore somewhat expected under this supervision. Overall, I appreciate the improved experiments and analysis, but am still unconvinced that the benchmark and its findings offer sufficient insight to the community.

3 comments
AuthorsOpenReview ↗

Reviewer aZif, part 1 (W1, Q1, W3)

We sincerely thank you for the detailed review, and for recognizing the motivation of the work and the value of the success and failure analysis. Your comments prompted most of the new experiments in the revision. This is the first of three comments: W1/Q1 and W3 here, W2/Q2 in the second, and W4 in the third.

W1 / Q1. No training samples; how do you separate out-of-distribution failure from instruction-following failure?

Response. Thank you for raising this central point. We agree that a held-out scene adds a confound, but a fully in-domain benchmark has the opposite problem.{\color{#1f6feb}\text{We agree that a held-out scene adds a confound, but a fully in-domain benchmark has the opposite problem.}} ManipScope takes the middle ground, and the controls added in the revision separate the two: the failures remain when nothing is out of distribution, so they are failures to follow the instruction.

(1) Both extremes confound. In an out-of-distribution scene, a policy can fail for reasons unrelated to language. In an in-domain scene, it can succeed without language, by replaying a trajectory from the position or the scene it remembers. LIBERO-based probes run in training scenes, and policies there map object names to positions (see W2). A new control on the RoboTwin training tasks shows the same: with an empty instruction, checkpoints still move the red block first in 110/120 runs.

(2) The middle ground. As in the submission, objects and skills are drawn from the RoboTwin 2.0 training tasks (89 links to 34 tasks; Figure 5), on which every checkpoint is trained; the scene composition and the request are new. A request such as "Pick up the block that is not red." has no memorized answer, so it tests whether the words are understood: ten checkpoints pick the right block first in 11/100 runs of the new reassignment experiment, whereas the positive control, a GPT-6 agent, does so on 5/5 states.

(3) Two new control experiments separate the remaining confound. Each removes one source of failure.

ControlWhat it removesResult
Unmodified RoboTwin training task, instruction changed to "Pick up the blue block." (Figure 3, Table 29)Scene shiftFive of six checkpoints: blue block first in 0/100 runs, red in 99
Same held-out state; the instruction names the block the policy prefers, or the other block (Table 28)Inability to act in the scenePreferred block: 99/100 runs. Other block: 2/100

The failure appears with nothing out of distribution, and the policy can act in the held-out scene; its choice does not follow the words.

Conclusion. These are instruction-following failures, not out-of-distribution failures.{\color{#1f6feb}\text{These are instruction-following failures, not out-of-distribution failures.}} The same tests identify the checkpoints that do follow: π0.5 moves the blue block first in 15/20 training-scene runs, and LingBot-VA follows all three color instructions on 10/10 held-out states.

Changes made. Sec. 3.1 (training data stated; Remark on the middle ground), Sec. 5.3 (new Tables 2-3 and Figures 2-3), App. G.1 (why the scenes come without demonstrations), App. G.2-G.3 (Tables 27-30).

W3. Is a non-target contact a wrong decision or poor collision avoidance?

Response. Thank you for this question. In the non-target/success episodes you ask about, the contact is mostly an incidental brush, not a decision{\color{#1f6feb}\text{In the non-target/success episodes you ask about, the contact is mostly an incidental brush, not a decision}}; in failed episodes it is almost always a grasp of the wrong object, a decision. The revision shows this with a new three-way classification of non-target contacts, built to separate a non-target grasp from a brush.

(1) Three classes (new in the revision). From the gripper state recorded at every step, each non-target contact is a grasp (pinched by a gripper and carried), a push (no pinch, displaced by at least 5 cm) or a brush (touched, displaced by less than 5 cm). For this we reran 2,400 episodes from ten checkpoints with the gripper state recorded (App. D.5, Table 18). To make the classes concrete, new Figure 8 shows keyframes of one episode for each class.

(2) Result.

Episodes with a non-target contactGraspPushBrush
Failed (1,530)1,380 (90.2%)58 (3.8%)92 (6.0%)
Successful (308)43 (14.0%)17 (5.5%)248 (80.5%)

The non-target contacts of failed episodes are therefore wrong decisions, not poor collision avoidance, while the non-target/success cases you point to are mostly brushes.

(3) Change to the metric (new). To keep brushes out of the selection metric, the revision also scores commitment, the first candidate displaced by at least 5 cm (Sec. 4.1). The paper's result that failure can follow a correct choice holds under it.

Changes made. Sec. 4.1 (commitment), Sec. 5.2, App. D.5 (Table 18, Figure 8).

AuthorsOpenReview ↗

Reviewer aZif, part 2 (W2, Q2)

W2 / Q2. Could these probes be run in LIBERO-PRO or LIBERO-Plus? What makes a new benchmark necessary?

Response. Thank you for suggesting this comparison; we ran a new experiment in LIBERO to answer it. Probes run in LIBERO-PRO or LIBERO-Plus cannot isolate instruction following{\color{#1f6feb}\text{Probes run in LIBERO-PRO or LIBERO-Plus cannot isolate instruction following}}, because their policies are trained on demonstrations that tie object identity to position (1-2). ManipScope removes this tie (3) and adds diagnostics that neither reports (4).

(1) In the LIBERO demonstrations, identity and position are tied. Across the 50 initial states of a task, the manipulated object moves by about 2-5 cm in all four suites, and in LIBERO-Object the target sits at one of two positions. A policy can therefore pass without understanding the instruction, by memorizing where each named object sits.{\color{#1f6feb}\text{A policy can therefore pass without understanding the instruction, by memorizing where each named object sits.}} A new experiment, our reassignment probe run on LIBERO-Object, shows that it does (Table 32).

The changed instruction names an objectπ0.5 picks itStarVLA picks it
at a trained target position99/100100/100
elsewhere49/4000/400

In the second row, 295 and 391 of the 400 runs go to where the named object sits in its own training task.

(2) LIBERO-PRO and LIBERO-Plus inherit this tie. With the trained layout kept, a memorized position is enough. With the layout perturbed, the policies fail on position alone: on LIBERO-PRO's position-perturbation leaderboard, OpenVLA and π0 score 0.00 on all four suites, and π0.5 0.08-0.38. Neither case isolates instruction following; the second is the confound of W1.

(3) ManipScope removes the tie. In the RoboTwin block tasks the demonstrations place each block anywhere in a 56 × 13 cm region, so a memorized position cannot replace the instruction{\color{#1f6feb}\text{a memorized position cannot replace the instruction}}. Moreover, the requests must be understood, not recalled: "Pick up the block that is not red." matches no trained instruction.

(4) Direct comparison (new; Table 31).

LIBERO-PROLIBERO-PlusManipScope
Target changed to another object✓✓✓
Empty or meaningless instruction✓✓✓
Visual and layout perturbations✓✓✗
All instructions followed on one state; instruction-blind chooser✗✗✓
Moved object recorded; failure decomposition✗✗✓
Measured positive control✗✗✓
Object position range in training data2-5 cm2-5 cm56 × 13 cm (block tasks)
Bimanual platform✗✗✓

Reading the table:

  • Complementary aims. The two test robustness; ManipScope diagnoses instruction following.
  • Wrong choice or failed execution. LIBERO-PRO scores at most 0.10 under its task perturbation, but success alone cannot tell a wrong choice from a failed execution; ManipScope can (84.6% of failures move a distractor; Table 19).
  • Positive control. A GPT-6 agent follows every instruction on 88% of states: the changed requests are solvable, and this is the upper reference.
  • Two arms. Two arms expose a failure one arm cannot: one gripper holds the target, the other a non-target, and the episode still succeeds (Figure 8a).

(5) LIBERO-CF is closest: it also changes instructions and scores contact with the instructed object, but in training layouts and without a joint score over one state.

Changes made. Sec. 2, App. H (Tables 31-32).

AuthorsOpenReview ↗

Reviewer aZif, part 3 (W4)

W4. The conclusions are expected and say little about why failures happen or how to fix them.

Response. Thank you for this comment, which pushed us beyond diagnosis. The revision now answers both questions, and each diagnostic points to a specific fix{\color{#1f6feb}\text{each diagnostic points to a specific fix}}.

  • Why and where (points 1-2): the failures are traced to the demonstrations and to target selection.
  • How to fix (points 3-5): the diagnosis implies specific fixes, and a new experiment confirms one of them: handing target selection to a vision-language agent raises a checkpoint from 0/10 to 7/10 states on which every instruction is followed.
  • On "expected": we agree that success need not imply instruction following. The size of the gap was not expected: ten checkpoints follow reassigned instructions less often than an instruction-blind chooser (3.9% against 16%).

(1) Why: the demonstrations (new experiments on the training tasks). The object a policy picks under every instruction is the one the training demonstrations handle first{\color{#1f6feb}\text{The object a policy picks under every instruction is the one the training demonstrations handle first}}: the red block in two tasks, the smallest in a third. For the size task we logged this prediction before collecting the data, and it held: the requested largest block is moved first in 1/160 runs, the smallest in 145. In these training tasks each task has one fixed goal, so the instruction never has to distinguish the objects.

(2) Where: target selection, before execution. Execution is not the bottleneck: a new decomposition shows that in 84.6% of failures the policy moves a distractor (Table 19). The instruction reaches the action but does not select the target: in the first-decision test, extended from two to twelve checkpoints, changing the color word for 9 of the 12 changes the first 16 commands by about three times the repeat baseline, yet only LingBot-VA's first motion heads for the named block (Figure 13). The prediction-action agreement you mention, reported in the submission, points the same way: 46 of 77 readable Motus forecasts already show the wrong object, so the error precedes execution.

(3) How to fix (new in the revision).

FindingWhat it implies for a fix
The default object is the one handled first in the demonstrations; each training task has one fixed goalData: demonstrations in which the instruction decides which object is handled, with first targets balanced
Failures move the wrong object, and the forecast already shows itModel: improve target selection, not grasping or control (tested in (4))
π0.5 and LingBot-VA follow color although trained on the same demonstrationsThe data do not force the failure, so a remedy without new data is possible (tested in (4))

(4) A fix tested (new experiment). Following the table, we supplied the missing target selection without any training. A vision-language agent (GPT-6) reads the instruction and brings the open gripper to the object it refers to; the policy then takes over and performs the grasp. The policy is DM0.5, which alone follows every instruction on 0/10 states of each scene below.

Scene (10 frozen states)All instructions followed: DM0.5 aloneWith selection handed overSuccess: aloneWith handover
Three colors0/107/1012/3027/30
Negation ("not red", "not blue")0/106/1010/2016/20
Order (leftmost, second from left, rightmost)0/103/1013/3023/30

In contrast, amplifying the effect of the instruction on the action (guidance weights 2 to 5) leaves the policy at 0/10. The instruction signal is not merely too weak; the selection is missing, as the diagnosis in (2) says. This handover is a simple illustration, not a tuned method (App. G.4).

(5) How to verify a fix. A fix should raise the share of states on which every instruction is followed above the 16% of the instruction-blind chooser; LingBot-VA and π0.5 show that this is attainable (10/10 and 6/10 states of the three-color scene). The experiment in (4) is scored in exactly this way. Training-based fixes are beyond a diagnostic benchmark; ManipScope provides the measurements on which they are evaluated.

Changes made. Sec. 5.3, App. E.2 (Figure 13), App. G.3, App. G.4 (directions for a remedy, and the handover experiment).

We hope the new results and the revised paper address your concerns, and we would be glad to clarify any remaining point. Thank you again for your time.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (to all reviewers)

We sincerely thank the three reviewers for their careful and constructive comments. They led us first to state more precisely what each part of ManipScope measures, and then to test it. The revision therefore has two layers. First, the design is refined: the metrics and the diagnostic tools are made more precise, each with what it measures and where it applies. Second, the design is backed by new evidence: more than 10,000 new policy episodes, ten new tables and six new figures (new text in blue). Detailed replies follow under each review.

The shared question. In different forms, the three reviews ask: when a policy fails a ManipScope scene, is it failing to follow the instruction, is the scene out of distribution, or is a skill missing? New control experiments on the same checkpoints show that it is the first.{\color{#1f6feb}\text{New control experiments on the same checkpoints show that it is the first.}}

1. Design: what ManipScope measures, and how the revision refines it.

  • Scenes: a middle ground. A fully out-of-distribution scene can make a policy fail for reasons unrelated to language, and a fully in-domain scene lets it succeed from memory. ManipScope uses trained objects and skills with a new composition and a new request (aZif W1, W2).
  • Requests: opposite requirements. The suite pairs requests such as the red block in one scene and "the block that is not red" in another, so a fixed preference cannot score across it (7bSB W1).
  • Diagnostics, refined: each control removes one confound at a time{\color{#1f6feb}\text{each control removes one confound at a time}} (the chain is summarized in the new Table 2). Target reassignment replays one frozen state under instructions that name each candidate in turn, and is read against an instruction-blind chooser (chance reference) and a positive control, a GPT-6 agent (upper reference). The same test on unmodified training tasks removes scene shift; the source tasks remove missing skill (aZif W1, ruEK W1, 7bSB W1).
  • Metrics, refined: a recorded selection must be a choice. Non-target contacts are now classified from the gripper state as grasp, push or brush; a stricter event, commitment, is added; and selection statements are limited to the tasks in which a selection event exists (aZif W3, 7bSB W2).
  • Comparisons: two leaderboards. Agents and policies differ in training and interface, so they are ranked separately and compared only on the same five episodes per scene (ruEK W2).

2. Evidence: what the new experiments show.

QuestionAnswerResult
Do policies follow the instruction?Mostly notTen checkpoints follow every instruction on 3.9% of states (39/1,000), below the instruction-blind chooser (16%); the positive control, a GPT-6 agent, reaches 88% (Table 3)
Is scene shift the cause?NoOn unmodified training tasks, five of six checkpoints move the requested blue block first in 0/100 runs (Table 29)
Is a missing skill the cause?No84.6% of failed episodes move a distractor; nine of ten checkpoints complete at least nine of their ten source tasks (Tables 19, 30)
Does the selection metric record a choice?YesA non-target is grasped in 83% of failed episodes and in 5.8% of successful ones (Table 18)
Does the benchmark discriminate?YesLingBot-VA and π0.5 follow color instructions on 10/10 and 6/10 states (Sec. 5.3, Table 27)
Can the failure be fixed?A first stepWith target selection handed to a vision-language agent, a policy goes from 0/10 to 7/10 states (App. G.4)

Also in the revision. Our probe run on LIBERO-Object and a direct comparison with LIBERO-PRO and LIBERO-Plus (App. H; aZif W2); first-decision sensitivity for all twelve checkpoints, from two (App. E.2; aZif W4, ruEK W3); results for the nine checkpoints with identical training data (App. B.1; ruEK W2); and two checkpoints marked and kept out of pooled statements (X-VLA, whose release takes task names as its language input, and a community π0 that does not complete its source tasks; App. B.1).

We are grateful for the time the reviewers have given to our paper, and we would be glad to provide any further detail.

AuthorsOpenReview ↗

A Small Update: Evaluation of Three Recent Agents

We have updated the manuscript to include evaluations of three recent agents: GPT-6.2 Astra, Gemini 4 Argon, and Opus 5.5. Given the evaluation cost, each model is tested on five episodes per scene, totaling 240 episodes across all 48 scenes. Their results are integrated into the existing comparison tables, with model-specific configurations and budget adjustments documented in the appendix. The new additions are highlighted in red, while the previous revisions remain in blue. Once the benchmark is publicly released, we welcome community contributions to expand evaluation to a broader range of models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.