- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces RoboGaze which is a training free method that evaluates synthetically generated robot demonstrations in regards to different failure modes. The method uses different VLMs to first generate context, discover candidates of failures and verify&report them. Further, the paper introduces a benchmark of over 900 video clips which are hand annotated and can be used to evaluate the proposed method and different baseline models. RoboGaze outperforms all other baselines on the proposed benchmark regardless of chosen VLM.
Critical strengths
- Thorough experiments with ablations which show where the gains come from.
- Proposed benchmark is useful for future work in this direction.
- RoboGaze outperforms all other baselines
Critical weaknesses
Weaknesses
- It is not clear if all methods receive the same amount of frames for their evaluation. When frames are uniformly sampled that would mean that for longer clips models receive far less frames compared to RoboGaze. It would help to evaluate a windowed B1/CoT baseline that uses the same 4s window.
- The method is only evaluated on the introduced benchmark and not on any external ones.
- The critic itself does a lot of performance boost, showing another ablation where the critic is run on top of B1 or B4b could help to clarify how much contribution (ii) actually helps (routed, dimension-specific specialist
- It is not clear how the method helps for downstream applications. Actually training a robot policy using filtered data through RoboGaze would be interesting and supporting the value of the paper.
Minor Weaknesses
- Table 3 spacing between different metrics is not clear, results from RG are right beside CoT of the next metric, which makes interpreting the table on first glance difficult.
- There are a lot of terms in the paper which are switched in between. Keep them consistent (GLiDE+ and B4b or RG and RoboGaze, etc)
- Figure two font size is too small, maybe the phase 3 part can be put below 1 and 2 so that in general there is more space
Questions for authors
- Could you report precision and recall separately? This would help to see where the gain comes from.
- What prompts did you use? They should be provided in the Appendix.
- Could you provide more evaluation results on other benchmarks for your method?
5 comments
Response to Reviewer wpUt (Part 2/3)
W3 and Q1: What the Specialists Add Beyond the Critic
"The critic itself does a lot of performance boost, showing another ablation where the critic is run on top of B1 or B4b could help to clarify how much contribution (ii) actually helps."
"Could you report precision and recall separately?"
You are right that the critic accounts for a large share of the result, and since precision and recall are the clearest way to see what it does, we answer both questions together. We ran the ablation you proposed: with identical frames and the same critic attached to both, replacing windowed B1 by RoboGaze's routed, dimension-specific specialists (innovation (ii) in Sec. 1) is worth 7.3 F1 [5.9, 8.6].
To make this comparison fair, we attached RoboGaze's full verification stage (critic prompt, acceptance threshold, span re-inspection and merge rule) to B1 and to GLiDE+, on both their own and the shared frames. The acceptance threshold was re-tuned for each arm on the 150-clip development pool, and for GLiDE+ we placed the critic both in place of and after its debate step, reporting whichever was better for GLiDE+.
Gemma4-31B, 914 clips. Columns are per-clip means, so F1 is not the harmonic mean of P and R. RoboGaze without the critic is one run shown in both blocks, since its observer and specialists already read only the shared frames.
| Method | F1 | P | R | Clean |
|---|---|---|---|---|
| Own frames | ||||
| B1 / + critic | 39.3 / 45.3 | 34 / 49 | 47 / 44 | .58 / .79 |
| GLiDE+ / + critic | 45.9 / 47.9 | 43 / 47 | 50 / 49 | .69 / .76 |
| RoboGaze without / with critic | 44.1 / 57.3 | 36 / 60 | 59 / 55 | .21 / .86 |
| Shared frames | ||||
| Windowed B1 / + critic | 42.8 / 49.6 | 34 / 47 | 59 / 54 | .33 / .80 |
| GLiDE+ / + critic | 47.2 / 48.7 | 44 / 49 | 52 / 51 | .67 / .77 |
| RoboGaze without / with critic | 44.1 / 56.9 | 36 / 59 | 59 / 55 | .21 / .86 |
The critic is useful on its own. It improves every baseline by 1.5 to 7 F1 and lifts clean accuracy to between .76 and .80, almost entirely by raising precision. It helps GLiDE+ least, since GLiDE+ already ends in a debate verifier.
The specialists, in turn, make the critic far more effective. On shared frames, windowed B1 and the routed specialists are nearly equal before verification, with the same recall and similar precision. Yet the same critic raises precision by 13 points on windowed B1 and by 23 on the specialists, which amounts to +6.8 and +12.8 F1, a difference of 6.0 [4.0, 7.9]. We attribute this to what the critic receives: a specialist's hypothesis comes attached to a subtask, with its objects, expected outcome and completion criterion, so the critic has a concrete expectation to check, whereas a windowed proposal does not.
Where the gain comes from (Q1). Relative to GLiDE+, RoboGaze gains mostly in precision while also recalling more. The two stages play complementary roles: the routed specialists recover more failures than any baseline but over-report, and the critic then removes unsupported reports, trading a few points of recall for a large gain in precision, a trade-off that Fig. 14 analyses.
Table 21 offers the complementary view, keeping the critic and removing parts of innovation (ii): dropping the router costs 5.1 F1, and dropping the dimension-specific specialists costs 6.4. Taken together, the specialists' value appears mainly once their hypotheses are verified, so Sec. 1, Sec. 3.4 and Sec. 6 now describe the contribution as this coupling rather than as separate components. App. E.8 reports this ablation, and Table 19 now adds precision and recall for the methods compared here.
Response to Reviewer wpUt (Part 3/3)
W2 and Q3: Evaluation Beyond Our Benchmark
"The method is only evaluated on the introduced benchmark and not on any external ones."
We evaluated RoboGaze on an external benchmark, as you suggested. We chose Spotlight because none of the robot-video benchmarks we examined (RBench, RoboWM-Bench, WorldArena, RoboTrustBench, H2R-Bench) provides time-stamped failure labels, and Spotlight is the closest benchmark that does: 600 text-to-video clips from Veo 3, Seedance and LTX-2, mostly of people, animals and scenes. RoboGaze ran on it with its architecture, prompts and schema unchanged, and in the other direction we ran Spotlight's two strongest strategies on RoboGazeBench.
Description F1, Gemma4-31B. Spotlight's error types are mapped onto our dimensions only to compute F1.
| Method | Spotlight (600 clips) | RoboGazeBench (914 clips) |
|---|---|---|
| RoboGaze | 38.5 | 57.3 |
| Spotlight Multi-Agent | 36.4 | 43.8 |
| Spotlight Sliding Window | 34.1 | 41.6 |
| GLiDE+ | 33.4 | 45.9 |
RoboGaze scores highest on both benchmarks, but on Spotlight's data its difference from Multi-Agent is within noise (+2.1 F1 [−0.9, 4.8]; S+P .271 vs. .262 on its native metric). We therefore report parity there and keep the superiority claim for robot video (App. E.9). This is consistent with the design: Spotlight's clips rarely involve a multi-step task to verify against, and RoboGaze's weakest categories there, human anatomy and logical errors, lie outside a manipulation schema. Beyond external data, we also broadened RoboGazeBench itself with a new generator, Seedance, on which the margin over GLiDE+ is 11.6 F1 [6.1, 17.9] (our reply to Reviewer axGC, Part 2).
W4: Downstream Use
"Actually training a robot policy using filtered data through RoboGaze would be interesting and supporting the value of the paper."
We agree that a policy trained on RoboGaze-filtered data would be the most convincing evidence, and we have not run that experiment.
The downstream evidence in the paper is clip selection, which now carries a paired interval: among six candidates per condition, the top half ranked by RoboGaze contains usable demonstrations at .80 [.70, .88], against .71 [.61, .80] for GLiDE+, a paired gain of 9 points [2, 17] (App. E.5, Table 24). A policy study, however, requires matched amounts of retained data and several training seeds per arm, which we leave to follow-up work.
In the meantime, Sec. 6 now sets out how event-level reports would enter such pipelines, for instance by masking the action loss inside a reported span rather than discarding a whole rollout, and our reply to Reviewer axGC (Part 5) gives this discussion in full.
Q2 and Presentation
Prompts. App. G now lists the prompts of all RoboGaze components; the baseline prompts are released with the code.
Presentation. Thank you for these suggestions, which we addressed as follows:
- Table 3 separates the metric blocks with rules and extra spacing.
- The paper now uses one name per method ("RoboGaze", "GLiDE+"), and the labels B1 to B6 appear only in the two appendix tables that define and compare the controls (Tables 17 and 19).
- For Fig. 2, we tried placing Phase 3 below Phases 1 and 2, but the stacked layout broke the left-to-right reading of the pipeline. We therefore kept the layout and enlarged the type throughout, so that the figure is now legible at print size. Thank you for pointing this out.
We hope these additions address your concerns. If they do, we would be grateful if you would consider strengthening your support, and we would be glad to clarify anything further during the discussion period.
Response to Reviewer wpUt (Part 1/3)
Thank you for your constructive review, and for finding our experiments thorough and the benchmark useful for future work. Your suggestions were concrete enough that we could carry out nearly all of them during the discussion period: the windowed baselines (W1), the critic on top of the baselines (W3), precision and recall (Q1), an external benchmark (W2, Q3) and the prompts (Q2). Revisions are highlighted in blue in the updated PDF, and brackets denote 95% bootstrap intervals.
W1: Equalizing the Frames Seen by Each Method
They did not, and your reading of the long clips is correct. The submission fixed the visual interface of each call (Table 16), but a method that makes several calls can still read more distinct frames per clip: on Gemma4-31B, a single call reads 30 frames on average, GLiDE+ 48 and RoboGaze 88.
We therefore ran the windowed baselines you suggested. Each clip is sampled once at 4 FPS and cut into the same non-overlapping 4 s windows that RoboGaze's observer uses, and every image-reading stage of every method is restricted to these frames. Windowed CoT and windowed B1 run CoT and B1 once per window, and their events are merged with RoboGaze's own rule.
Gemma4-31B, 914 clips. Clean: share of the 210 failure-free clips with no reported failure.
RoboGaze's margin over GLiDE+ (F1) under each control.
†Scored like the recomputed Table 23 (gold references, clip-level averaging).
Windows help the baselines find failures, but also make them over-report. Running B1 once per window raises its recall from 47 to 59, yet its clean accuracy falls from .58 to .33, and windowed CoT leaves almost no failure-free clip clean. Each window is another opportunity to report a failure, and nothing in these baselines checks the additional proposals. This is precisely the over-reporting that RoboGaze's critic is designed to remove, which we return to under W3.
RoboGaze's advantage survives equal frames. Equalizing frames narrows the margin over GLiDE+ only slightly, so unequal frame access accounted for a small part of it, and the margin holds when GLiDE+ is also given RoboGaze's token budget and on a second backbone. Notably, windowed B1 sees exactly what RoboGaze's observer sees and still trails by 14.1 F1, which shows that what the frames are used for matters more than how many are read.
Long clips. As your review anticipated, the margin's growth with duration under native frame limits mostly reflects coverage, because a single call must subsample a long clip. With shared frames, RoboGaze still leads above 40 s by 7.8 F1 [5.1, 10.9] and localizes events better (mIoU .49 vs. .37), but no longer by more than on short clips. Sec. 5.3 now reports the two effects separately, and the abstract no longer claims that the advantage is largest on long clips.
App. E.7 reports every arm of this control, and our reply to Reviewer axGC (Part 1) describes the token-matched protocol in full.