- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ManipScope, a benchmark designed to evaluate robotic manipulation capabilities, specifically focusing on performance when instruction requirements change. Built upon RoboTwin 2.0, the benchmark encompasses 48 scenarios across six task categories; while these scenarios are clearly linked to source task components, no expert demonstration data is provided for training. The study evaluates twelve policy checkpoints and two agents capable of tool use. Beyond measuring task success rates, the research examines target selection during initial interaction and the extent to which instruction attribute requirements are met. Additionally, the study conducts several controlled experiments, including cross-combining target instructions with object layouts, analyzing the consistency between predicted targets in generated videos and actual execution targets, and substituting internal representations within the video generation strategy. Key findings indicate that robots may select the correct target during the initial phase even if the task is ultimately not completed; predictions and execution outcomes may align on an incorrect target; and changes in action resulting from interventions do not necessarily demonstrate goal-oriented transfer.
Critical strengths
- This is an evaluation resource of high practical value that encompasses a wide range of experimental scenarios. ManipScope provides 48 scenarios linked to original tasks—covering various forms of task requirement changes—and includes detailed success criteria, evaluation interfaces, and evaluation results for each task. By evaluating 12 policy checkpoints across 100 episodes per scenario, this resource establishes a robust benchmark for future research. Its evaluation-only design facilitates the study of existing systems without requiring modifications to accommodate the demonstration data provided for new scenarios. Regardless of whether a paper proposes a new policy architecture, this resource contribution is significant in its own right.
- The paired-behavior study offers a rigorously controlled case that compellingly demonstrates why isolated success metrics are insufficient. Testing combinations of two valid goal instructions against two object layouts yields far richer information than simply measuring whether actions change in response to input perturbations. In the color-selection scenario tested, Motus followed both instructions simultaneously in only 1 out of 40 paired cases sharing the same layout, whereas π0.5 achieved this in 29 out of 40 cases. A configuration-level bootstrapping approach preserved the correlations between paired conditions. Furthermore, the authors appropriately distinguished between "goal consistency" and "instruction correctness" and avoided misinterpreting "sensitivity to open-loop interventions" as a substantive behavioral improvement. These design choices and interpretive strategies make the case study highly persuasive within its scope.
Critical weaknesses
1. The benchmark fails to clearly distinguish between "following instructions" and "executing tasks."
In Appendix A.2 (lines 701–703), the "three-colored block" task always requires selecting the red block. Consequently, a strategy that simply selects the red block can achieve a high score without actually following the instruction. Although a paired test addresses this issue, its scope is limited to a single task and two checkpoints (Section 5.3, lines 376–377). Furthermore, the relationship between source and target tasks does not verify whether the model has mastered the underlying skills or whether it encountered similar combinations during training (Section 3.1, lines 155–157). This makes the benchmark results difficult to interpret: did the model fail because it misunderstood the instruction, or because it was unable to execute the task?
2. Some diagnostic metrics lack sufficient validation.
As shown in Figure 6, in the "bell" task, all models scored 0/5 on the "first target selection" metric, even though Motus and GPT-6 achieved a task success rate of 5/5. As noted in Appendix D.3 (lines 1456–1457), task success does not necessarily trigger the "contact event" upon which this metric relies. This raises concerns about whether the metric accurately reflects selection errors. Internal intervention analysis also has limitations: only two interpretable "donor–recipient" pairs predicted different targets, and both pairs originated from the same initial configuration (Appendix E.3, lines 1663–1665). Therefore, beyond demonstrating that predicted actions do indeed change, it remains unclear whether these experiments provide meaningful evidence of "goal-directed behavior."
Questions for authors
1. Does the benchmark truly measure instruction-following capability?
In Appendix A.2 (lines 701–703), the three-colored-block task always requires selecting the red block; consequently, a model can succeed simply by choosing the red block, even without following the instruction. Aside from the pairwise experiments conducted on a single task and two checkpoints, is there evidence that the models' performance across the 48 scenarios actually depends on the instructions rather than on fixed scenario preferences?
2. How reliable is the "first selection" metric across different tasks?
As shown in Figure 6, all models scored 0/5 on "first target selection" in the bell task, even though Motus and GPT-6 achieved a task success rate of 5/5. Given that a successfully executed action does not necessarily trigger the required contact event (Appendix D.3, lines 1456–1457), is it possible that this metric misclassifies successful behaviors as selection failures? Would excluding such tasks alter the primary conclusions regarding attribute-level performance?
4 comments
Reviewer 7bSB, part 2 (W2, Q2)
W2 / Q2. Are the diagnostic metrics validated? Do they measure what they are claimed to measure? (Bell task: 0/5 on first selection but 5/5 success.)
Response. Thank you for this careful observation. , and this is validated in three ways, two of them new in the revision. It does not apply to tasks that are completed without a loaded contact, such as the bell task you found; the revision removes these tasks from selection statements and reports how the numbers change.
(1) Validation of the selection metric.
| Check | Result |
|---|---|
| Force threshold varied from 0.1 to 1 N (as in the submission) | Correct-target counts change by at most 3 of 35 per system |
| Stricter event (new): commitment, the first candidate displaced by at least 5 cm | Episodes that fail after a correct choice: 47/210 by first contact, 39/210 by commitment |
| Independent record of the gripper state (new experiment, 2,400 episodes) | A non-target is grasped in 83% of failed episodes and in 5.8% of successful ones, so a recorded wrong selection is a wrong grasp, not a touch |
The metric also registers correct behavior where it occurs: under target reassignment, LingBot-VA follows all three color instructions on 10/10 states.
(2) Where the metric does not apply: your bell example. You are right: in this task the metric missed successful behavior.
- What happens. The press is judged from the position of the gripper, and no loaded event is recorded on the bell in any of the 35 episodes, 26 of which succeed (new Figure 9 shows one such episode: the gripper presses the bell, success is reported, and the recorded force is zero).
- How it was counted. The submission did not count these episodes as wrong selections (it reported them as no event, App. D.3), but they entered the denominator of the attribute rates. The same holds for the two hammer tasks.
- What the revision does. It restricts selection statements to the six pick-type tasks (new analysis, App. D.5), with this effect:
| Attribute correctness | All tasks | Without the task that has no event |
|---|---|---|
| Category | 53.3% | 80.0% (without bell-press) |
| Color | 65.7% | 73.3% (without color-hammer) |
| Size | 43.8% | 52.9% (without size-hammer) |
- What changes. The low category rate of the submission came from the bell task, and we no longer state it.
- What does not change. Size remains the lowest attribute, and a correct choice does not guarantee completion (47/210, or 39/210 under commitment).
Changes made. Sec. 4.1, Sec. 5.2, App. D.5 (new Figure 9).
W2, second half. Is the evidence from the intervention enough to support goal-directed behavior? (Two readable pairs, one configuration.)
Response. We agree with your assessment: the evidence is not enough for that claim, and the paper does not make it. The intervention is reported as a negative result: the predicted actions change, but not toward the goal. The paper's statements about following a goal rest on behavior, not on this analysis.
(1) What the evidence shows (as in the submission).
| Measure | Future representation replaced | Full instruction changed |
|---|---|---|
| Change in predicted joint commands | 0.030 rad | 0.096 rad |
| Shift of preference toward the donor's goal | 0.560 mm, 95% interval [−0.019, 1.299] | 2.529 mm |
The actions respond to the intervention, but the interval of the shift toward the goal includes zero.
(2) What this supports, and what it does not. It supports one methodological point: a change in actions is not evidence of goal following, so the direction of the change must be evaluated. It does not show goal-directed behavior, and it does not show the absence of goal influence either: only two readable contrasts predict different targets, both from one configuration, and no closed-loop run is made. The paper states these limits (Sec. 5.3, Sec. 6, App. E.3).
(3) Where goal following is tested with enough evidence. Behaviorally: target reassignment scores which object is moved under each instruction, on 1,000 initial states for ten checkpoints (extended in the revision; see W1 in our first comment). The conclusions about instruction following come from there.
Changes made. No new data for the intervention. The main text now carries one sentence with these numbers (Sec. 5.3); the full paragraph is in App. E.3.
We hope the new results and the revised paper address your concerns, and we would be glad to clarify any remaining point. Thank you again for your time.
i've fully read the author's supplement information, thank you for more experimental and impressive information, base on information above, i will kept my original score as weak accept
Thank You, Reviewer 7bSB!
Thank you for taking another careful look at our work! We’re glad you found the additional experiments helpful, and we really appreciate your thoughtful feedback and continued support.
Reviewer 7bSB, part 1 (W1, Q1)
We sincerely thank you for the thorough review, with its precise references to the paper, and for recognizing the practical value of the benchmark and the care of the paired study. This is the first of two comments: W1/Q1 here, W2/Q2 in the next.
W1 / Q1. The three-color scene always asks for red, so a fixed preference scores. Does performance depend on the instruction?
Response. Thank you for this important question. Yes: ManipScope measures whether a policy understands the instruction, and a fixed scene preference cannot score across it. Measured this way, most checkpoints do not follow the instruction, and two do. The evidence is at three levels: across scenes (1), within one scene (2), and with execution ruled out (3); (4) relates this to existing benchmarks.
(1) Scenes with opposite requirements (as in the submission; numbers new). You are right that the three-color scene asks for red. The benchmark also contains the opposite request: a scene that asks for "the block that is not red", which no preference for red can pass.
A preference therefore cannot produce a high score across the benchmark; it shows up as exactly this contrast.
(2) Instructions exchanged within one scene (extended in the revision). Target reassignment replays each frozen initial state under instructions that name each candidate in turn, so a fixed preference is counted as fixed, not as success. The submission ran this on the scene you cite with two checkpoints; the revision covers twelve checkpoints and ten selection scenes with 25 instructions (Sec. 5.3, Table 3).
The positive control shows that the requests are solvable and how well they can be followed, and the test discriminates: LingBot-VA and π0.5 do follow color (10/10 and 6/10 states of the three-color scene). The paper now reads the success of a selection scene together with this result, not alone.
(3) Misunderstanding, not inability to execute (new experiment). In a new run of the source tasks under training instructions in the same evaluator, nine of ten checkpoints complete at least nine of the ten linked source tasks in 70% or more of episodes (Table 30). The tested combinations are absent from the RoboTwin demonstrations: none of the 48 scenes is a RoboTwin task, and each RoboTwin block task has one fixed goal (Sec. 3.1, App. G.1).
(4) Scene preference in existing benchmarks. The concern you raise is an important one, and it does arise elsewhere. In LIBERO-based benchmarks such as LIBERO-PRO, the object layout of a task barely varies (2-5 cm), so a policy can rely on the scene: in our new probe on LIBERO-Object, policies follow a changed instruction only when the named object sits at a trained target position (99-100% of runs there, 49/400 and 0/400 elsewhere; Table 32).
Changes made. Sec. 3.1 (Remark), App. A.2, Sec. 5.3 (Tables 2-3, Figure 2), App. G.1-G.3.