- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Spotter keeps a frozen embodied policy running while a VLM monitors its actions and intervenes with a short repair plan when a failure is confirmed. The paper reports meaningful success-rate gains, but does not sufficiently isolate which component produces them.
Critical strengths
Spotter keeps a frozen embodied policy running while a VLM monitors its actions and intervenes with a short repair plan when a failure is confirmed. The paper reports meaningful success-rate gains, but does not sufficiently isolate which component produces them.
Critical weaknesses
- The incremental contribution over prior monitoring and recovery systems remains unclear. A matched synchronous-versus-asynchronous comparison using the same judge, repair primitives, context, information, and step budget is missing.
- The strongest results combine a powerful judge, repair primitives, a depth-based executor, and a retrieved demonstration. Even the “zero-shot” setting retrieves from 77 GPT-derived general lessons. Their separate contributions and the offline cost of building these resources are insufficiently characterized.
- The overall efficiency trade-off is substantial: with the Qwen full-context setting, mean episode time rises from 63 to 181 seconds for Cosmos Policy and from 37 to 126 seconds for π0.5. The broad claim about recovery is also supported mainly by a diagnosis of failed grasps.
Questions for authors
- Under matched models, inputs, primitives, and budgets, how much benefit comes specifically from asynchronous supervision?
- What performance remains without the general-lesson library, retrieved example, and depth-based repair executor?
- Can the authors report recovery by failure type and compare success rate at matched wall-clock or token budgets?
5 comments
Rebuttal to Reviewer 338G (Part 2/2)
W3: Efficiency trade-off
The overall efficiency trade-off is substantial: with the Qwen full-context setting, mean episode time rises from 63 to 181 seconds for Cosmos Policy and from 37 to 126 seconds for π0.5.
Response: The increase comes from failed episodes (Table 2). On successful episodes Spotter adds only 13–16 s (34→47 s on Cosmos Policy, 23→39 s on π0.5), while failed episodes, where the judge keeps trying until the step limit, go from 125 to 521 s and from 62 to 313 s. An early exit, in which the judge ends hopeless episodes, cuts this time by about a third with success nearly unchanged (tested on RoboTwin with the GPT-6 Astra judge; Appendix F.3), and as repairs succeed more often, fewer episodes reach this case.
W3 & Q3: Recovery by failure type, and success at matched budgets
The broad claim about recovery is also supported mainly by a diagnosis of failed grasps. Can the authors report recovery by failure type and compare success rate at matched wall-clock or token budgets?
Response: We agree. By the judge's diagnosis at its first intervention (main RoboTwin run, GPT-6 Astra judge as in the paper; Table 10, Appendix F.5):
| Failure diagnosed | Share of intervened episodes | Repaired to success |
|---|---|---|
| Missed grasp | 63% | 21% |
| Stuck / no progress | 22% | 14% |
| Dropped object | 4% | 0% |
| Wrong object | 1% | 0% |
| Other | 11% | 8% |
Recovery comes mainly from grasp failures; drops and wrong objects are not repaired with the current primitives. We have narrowed the claim in the Conclusion (Section 6) to say this.
All methods already share the same step budget (1.8× the official limit). For time and token budgets, an episode counts only if it succeeds within the budget. The table below (Table 8, Appendix F.2) compares Spotter with Harness VLA using the same LLM, on RoboTwin 2.0 with π0.5. Unlike the ablations above, which use GPT-6.1, these runs use GPT-6 Astra as in the paper.
| Budget per episode | Spotter (GPT-6 Astra) | Harness VLA (GPT-6 Astra) | Spotter (Qwen) | Harness VLA (Qwen) |
|---|---|---|---|---|
| 60 s | 43.3 | 10.0 | 38.0 | 8.7 |
| 120 s | 50.0 | 32.0 | 48.0 | 24.0 |
| 300 s | 52.0 | 46.0 | 51.3 | 35.3 |
| 100k tokens | 48.7 | 2.0 | 49.3 | 25.3 |
| 500k tokens | 50.7 | 38.0 | 51.3 | 38.0 |
| 1M tokens | 52.7 | 46.0 | 52.7 | 42.0 |
| 2M tokens | 55.3 | 52.7 | 52.7 | 42.0 |
Time budgets use modelled rather than measured time. GPT responses include network delays that vary from call to call, so to compare all methods fairly we charge each the same: simulator steps × 0.24 s, plus LLM time at a fixed single-request speed measured for Qwen on one dedicated H200 (8,000 input and 75 output tokens per second, plus 0.3 s per call).
Spotter is ahead at every finite budget, because Harness VLA consults the VLM at every step. With an unlimited budget the GPT systems tie, but Harness VLA needs 2.2× the time and 4.6× the tokens per successful episode.
We hope these results address your concerns, and we are happy to discuss further.
Follow-up: Additional Experiments Added
Dear Reviewer 338G,
We have added the relevant experiments and kindly ask you to take another look at our revised submission. We would be grateful to know whether these additions address your concerns.
Rebuttal to Reviewer 338G (Part 1/2)
Dear Reviewer 338G,
Thank you for the precise questions on attribution and cost. W1/Q1 and W2/Q2 both ask how much each part of Spotter contributes, so we answer them together with one set of new ablation experiments on RoboTwin 2.0 (Hard), with π0.5. All changes are now in the revised PDF, marked in red.
W1 & Q1, W2 & Q2: Contribution of each component, including asynchronous supervision
Response: Because of cost, these ablations use GPT-6.1 as the judge instead of GPT-6 Astra used in the paper, on a 150-episode subset of RoboTwin 2.0 that covers all 50 tasks (3 episodes per task). Every variant, including Spotter itself, runs on exactly the same episodes with the same judge. Table 3 changes one part of Spotter at a time (variants are defined in Appendix F.1), keeping the judge, primitives, context, information and step budget fixed.
Table 3 (Section 4.4 of the revised PDF). Matched comparison and ablations (RoboTwin 2.0 Hard, π0.5, GPT-6.1 judge)
Asynchronous supervision (W1/Q1). In synchronous supervision, execution stays in lockstep with the screener: every window is checked, and the policy continues only after it is confirmed correct. This is why it is slightly higher than asynchronous supervision (58.3% vs 56.7%), but at a large cost in time: running supervision in parallel hides 36–38 s per episode on Cosmos Policy and 29–30 s on π0.5 on RoboCasa, about 60% and 80% of the policy's own execution time (Table 2). This is the main difference from prior VLM-led systems, in which execution waits on the VLM at every step (Table 7).
Screener. Likewise, removing the screener so that the judge checks every window raises success slightly, by 1.0 point on RoboTwin and 2.0 points on RoboCasa, but at a time cost that is hard to afford: the embodied model generates an action chunk in about 100 ms, while a GPT call takes 4.3 s on average, so the judge would sit on the critical path at every window. On RoboCasa it also takes 2.9× the tokens (Tables 1 and 2). The screener is what lets a slow judge run in parallel.
Lessons and retrieved example (W2/Q2). Without them success falls to 47.3%, below the policy alone, so they carry much of the gain. Depth-based aiming matters little (−0.3), so removing all three would not do better.
Execution history and expectation checking. Removing them lowers success to 52.7% and 56.0%.
Judge and primitives. A frontier judge is not required: with Qwen as the judge, Spotter raises RoboCasa success from 67.9% to 71.7% (Cosmos Policy) and from 64.3% to 68.4% (π0.5). The repair primitives matter: letting the policy retry after a detected failure gives no gain, even with an oracle detector (67.1% and 63.1%, against 67.9% and 64.3%; Table 1).
Offline cost is low and paid once (Appendix F.6). We run Spotter on development episodes disjoint from the test seeds and repeatedly summarize the judge's experience into general lessons: the 77 RoboCasa lessons took about 7 h (16 rounds on 367 episodes), and the 16 RoboTwin lessons about 3.7 h. We also manually reviewed 10 episodes in each of three rounds to catch bugs and add general advice, such as varying the motion after a failed grasp.