acceptodds
Under review as a conference paper at ICLR 2027

RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis

Abstract

Robot world models now generate synthetic manipulation videos, but evaluating them is hard: realistic-looking outputs often violate physical laws, temporal consistency or task logic, while conventional metrics and monolithic Vision-Language Model (VLM) judges return scores that do not say what failed. We present RoboGaze, a training-free multi-agent VLM evaluator that grounds the task and scene, routes each subtask to dimension-specific specialists and verifies their findings with a critic, producing temporally localized diagnostic reports that state what failed, when, why and how severely, under a schema of 6 dimensions and 30 failure types whose annotation reliability and coverage we measure. To benchmark it, we introduce RoboGazeBench, 914 human-annotated clips from five generators, conditioned on simulated and real-world data and running from 4.9 to 148.9 s. Across eight open-source and proprietary VLM backbones, RoboGaze outperforms every baseline; even a published agentic detector given the same schema and a slightly larger inference budget trails it by 10.1–12.3 description- points, a margin that blinded human raters confirm. On the strongest backbone it closes 83% of the gap between chain-of-thought prompting and human agreement, and its critic curbs the "cry-wolf" false positives of standard VLMs, lifting mean clean-clip accuracy from 17% to 85%. The advantage is largest on held-out generators and long clips, and carries over to selecting generated demonstrations (80% vs. 71% precision). Project page: https://robogazeproject.github.io/RoboGazeProject/

Primary area:
datasets and benchmarks
Keywords:
Benchmarks and datasets for robot learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wpUtOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces RoboGaze which is a training free method that evaluates synthetically generated robot demonstrations in regards to different failure modes. The method uses different VLMs to first generate context, discover candidates of failures and verify&report them. Further, the paper introduces a benchmark of over 900 video clips which are hand annotated and can be used to evaluate the proposed method and different baseline models. RoboGaze outperforms all other baselines on the proposed benchmark regardless of chosen VLM.

Critical strengths
  • Thorough experiments with ablations which show where the gains come from.
  • Proposed benchmark is useful for future work in this direction.
  • RoboGaze outperforms all other baselines
Critical weaknesses

Weaknesses

  • It is not clear if all methods receive the same amount of frames for their evaluation. When frames are uniformly sampled that would mean that for longer clips models receive far less frames compared to RoboGaze. It would help to evaluate a windowed B1/CoT baseline that uses the same 4s window.
  • The method is only evaluated on the introduced benchmark and not on any external ones.
  • The critic itself does a lot of performance boost, showing another ablation where the critic is run on top of B1 or B4b could help to clarify how much contribution (ii) actually helps (routed, dimension-specific specialist
  • It is not clear how the method helps for downstream applications. Actually training a robot policy using filtered data through RoboGaze would be interesting and supporting the value of the paper.

Minor Weaknesses

  • Table 3 spacing between different metrics is not clear, results from RG are right beside CoT of the next metric, which makes interpreting the table on first glance difficult.
  • There are a lot of terms in the paper which are switched in between. Keep them consistent (GLiDE+ and B4b or RG and RoboGaze, etc)
  • Figure two font size is too small, maybe the phase 3 part can be put below 1 and 2 so that in general there is more space
Questions for authors
  • Could you report precision and recall separately? This would help to see where the gain comes from.
  • What prompts did you use? They should be provided in the Appendix.
  • Could you provide more evaluation results on other benchmarks for your method?
5 comments
AuthorsOpenReview ↗

Response to Reviewer wpUt (Part 1/3)

Thank you for your constructive review, and for finding our experiments thorough and the benchmark useful for future work. Your suggestions were concrete enough that we could carry out nearly all of them during the discussion period: the windowed baselines (W1), the critic on top of the baselines (W3), precision and recall (Q1), an external benchmark (W2, Q3) and the prompts (Q2). Revisions are highlighted in blue in the updated PDF, and brackets denote 95% bootstrap intervals.

W1: Equalizing the Frames Seen by Each Method

"It is not clear if all methods receive the same amount of frames [...]. When frames are uniformly sampled that would mean that for longer clips models receive far less frames compared to RoboGaze. It would help to evaluate a windowed B1/CoT baseline that uses the same 4s window."

They did not, and your reading of the long clips is correct. The submission fixed the visual interface of each call (Table 16), but a method that makes several calls can still read more distinct frames per clip: on Gemma4-31B, a single call reads 30 frames on average, GLiDE+ 48 and RoboGaze 88.

We therefore ran the windowed baselines you suggested. Each clip is sampled once at 4 FPS and cut into the same non-overlapping 4 s windows that RoboGaze's observer uses, and every image-reading stage of every method is restricted to these frames. Windowed CoT and windowed B1 run CoT and B1 once per window, and their events are merged with RoboGaze's own rule.

Gemma4-31B, 914 clips. Clean: share of the 210 failure-free clips with no reported failure.

MethodF1, own framesF1, shared framesClean, ownClean, shared
CoT21.824.6.11.04
B139.342.8.58.33
GLiDE+45.947.2.69.67
RoboGaze57.356.9.86.86

RoboGaze's margin over GLiDE+ (F1) under each control.

SettingMargin
Submission (each method's own frames)11.4
Shared frames9.7 [8.6, 11.0]
Shared frames, GLiDE+ at RoboGaze's token budget9.3
Gemini 3.1 Pro, gold subset†, shared frames8.8 [5.9, 11.4]

†Scored like the recomputed Table 23 (gold references, clip-level averaging).

Windows help the baselines find failures, but also make them over-report. Running B1 once per window raises its recall from 47 to 59, yet its clean accuracy falls from .58 to .33, and windowed CoT leaves almost no failure-free clip clean. Each window is another opportunity to report a failure, and nothing in these baselines checks the additional proposals. This is precisely the over-reporting that RoboGaze's critic is designed to remove, which we return to under W3.

RoboGaze's advantage survives equal frames. Equalizing frames narrows the margin over GLiDE+ only slightly, so unequal frame access accounted for a small part of it, and the margin holds when GLiDE+ is also given RoboGaze's token budget and on a second backbone. Notably, windowed B1 sees exactly what RoboGaze's observer sees and still trails by 14.1 F1, which shows that what the frames are used for matters more than how many are read.

Long clips. As your review anticipated, the margin's growth with duration under native frame limits mostly reflects coverage, because a single call must subsample a long clip. With shared frames, RoboGaze still leads above 40 s by 7.8 F1 [5.1, 10.9] and localizes events better (mIoU .49 vs. .37), but no longer by more than on short clips. Sec. 5.3 now reports the two effects separately, and the abstract no longer claims that the advantage is largest on long clips.

App. E.7 reports every arm of this control, and our reply to Reviewer axGC (Part 1) describes the token-matched protocol in full.

AuthorsOpenReview ↗

Response to Reviewer wpUt (Part 2/3)

W3 and Q1: What the Specialists Add Beyond the Critic

"The critic itself does a lot of performance boost, showing another ablation where the critic is run on top of B1 or B4b could help to clarify how much contribution (ii) actually helps."

"Could you report precision and recall separately?"

You are right that the critic accounts for a large share of the result, and since precision and recall are the clearest way to see what it does, we answer both questions together. We ran the ablation you proposed: with identical frames and the same critic attached to both, replacing windowed B1 by RoboGaze's routed, dimension-specific specialists (innovation (ii) in Sec. 1) is worth 7.3 F1 [5.9, 8.6].

To make this comparison fair, we attached RoboGaze's full verification stage (critic prompt, acceptance threshold, span re-inspection and merge rule) to B1 and to GLiDE+, on both their own and the shared frames. The acceptance threshold was re-tuned for each arm on the 150-clip development pool, and for GLiDE+ we placed the critic both in place of and after its debate step, reporting whichever was better for GLiDE+.

Gemma4-31B, 914 clips. Columns are per-clip means, so F1 is not the harmonic mean of P and R. RoboGaze without the critic is one run shown in both blocks, since its observer and specialists already read only the shared frames.

MethodF1PRClean
Own frames
B1 / + critic39.3 / 45.334 / 4947 / 44.58 / .79
GLiDE+ / + critic45.9 / 47.943 / 4750 / 49.69 / .76
RoboGaze without / with critic44.1 / 57.336 / 6059 / 55.21 / .86
Shared frames
Windowed B1 / + critic42.8 / 49.634 / 4759 / 54.33 / .80
GLiDE+ / + critic47.2 / 48.744 / 4952 / 51.67 / .77
RoboGaze without / with critic44.1 / 56.936 / 5959 / 55.21 / .86

The critic is useful on its own. It improves every baseline by 1.5 to 7 F1 and lifts clean accuracy to between .76 and .80, almost entirely by raising precision. It helps GLiDE+ least, since GLiDE+ already ends in a debate verifier.

The specialists, in turn, make the critic far more effective. On shared frames, windowed B1 and the routed specialists are nearly equal before verification, with the same recall and similar precision. Yet the same critic raises precision by 13 points on windowed B1 and by 23 on the specialists, which amounts to +6.8 and +12.8 F1, a difference of 6.0 [4.0, 7.9]. We attribute this to what the critic receives: a specialist's hypothesis comes attached to a subtask, with its objects, expected outcome and completion criterion, so the critic has a concrete expectation to check, whereas a windowed proposal does not.

Where the gain comes from (Q1). Relative to GLiDE+, RoboGaze gains mostly in precision while also recalling more. The two stages play complementary roles: the routed specialists recover more failures than any baseline but over-report, and the critic then removes unsupported reports, trading a few points of recall for a large gain in precision, a trade-off that Fig. 14 analyses.

Table 21 offers the complementary view, keeping the critic and removing parts of innovation (ii): dropping the router costs 5.1 F1, and dropping the dimension-specific specialists costs 6.4. Taken together, the specialists' value appears mainly once their hypotheses are verified, so Sec. 1, Sec. 3.4 and Sec. 6 now describe the contribution as this coupling rather than as separate components. App. E.8 reports this ablation, and Table 19 now adds precision and recall for the methods compared here.

AuthorsOpenReview ↗

Response to Reviewer wpUt (Part 3/3)

W2 and Q3: Evaluation Beyond Our Benchmark

"The method is only evaluated on the introduced benchmark and not on any external ones."

We evaluated RoboGaze on an external benchmark, as you suggested. We chose Spotlight because none of the robot-video benchmarks we examined (RBench, RoboWM-Bench, WorldArena, RoboTrustBench, H2R-Bench) provides time-stamped failure labels, and Spotlight is the closest benchmark that does: 600 text-to-video clips from Veo 3, Seedance and LTX-2, mostly of people, animals and scenes. RoboGaze ran on it with its architecture, prompts and schema unchanged, and in the other direction we ran Spotlight's two strongest strategies on RoboGazeBench.

Description F1, Gemma4-31B. Spotlight's error types are mapped onto our dimensions only to compute F1.

MethodSpotlight (600 clips)RoboGazeBench (914 clips)
RoboGaze38.557.3
Spotlight Multi-Agent36.443.8
Spotlight Sliding Window34.141.6
GLiDE+33.445.9

RoboGaze scores highest on both benchmarks, but on Spotlight's data its difference from Multi-Agent is within noise (+2.1 F1 [−0.9, 4.8]; S+P .271 vs. .262 on its native metric). We therefore report parity there and keep the superiority claim for robot video (App. E.9). This is consistent with the design: Spotlight's clips rarely involve a multi-step task to verify against, and RoboGaze's weakest categories there, human anatomy and logical errors, lie outside a manipulation schema. Beyond external data, we also broadened RoboGazeBench itself with a new generator, Seedance, on which the margin over GLiDE+ is 11.6 F1 [6.1, 17.9] (our reply to Reviewer axGC, Part 2).

W4: Downstream Use

"Actually training a robot policy using filtered data through RoboGaze would be interesting and supporting the value of the paper."

We agree that a policy trained on RoboGaze-filtered data would be the most convincing evidence, and we have not run that experiment.

The downstream evidence in the paper is clip selection, which now carries a paired interval: among six candidates per condition, the top half ranked by RoboGaze contains usable demonstrations at .80 [.70, .88], against .71 [.61, .80] for GLiDE+, a paired gain of 9 points [2, 17] (App. E.5, Table 24). A policy study, however, requires matched amounts of retained data and several training seeds per arm, which we leave to follow-up work.

In the meantime, Sec. 6 now sets out how event-level reports would enter such pipelines, for instance by masking the action loss inside a reported span rather than discarding a whole rollout, and our reply to Reviewer axGC (Part 5) gives this discussion in full.

Q2 and Presentation

Prompts. App. G now lists the prompts of all RoboGaze components; the baseline prompts are released with the code.

Presentation. Thank you for these suggestions, which we addressed as follows:

  • Table 3 separates the metric blocks with rules and extra spacing.
  • The paper now uses one name per method ("RoboGaze", "GLiDE+"), and the labels B1 to B6 appear only in the two appendix tables that define and compare the controls (Tables 17 and 19).
  • For Fig. 2, we tried placing Phase 3 below Phases 1 and 2, but the stacked layout broke the left-to-right reading of the pipeline. We therefore kept the layout and enlarged the type throughout, so that the figure is now legible at print size. Thank you for pointing this out.

We hope these additions address your concerns. If they do, we would be grateful if you would consider strengthening your support, and we would be glad to clarify anything further during the discussion period.

Reviewer_axGCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RoboGaze, a training-free multi-agent VLM framework for diagnosing failures in generated robot-manipulation videos. RoboGaze constructs task and scene memories, decomposes videos into temporal windows and subtask segments, routes segments to six dimension-specific specialists, and uses a critic to verify and merge failure hypotheses. Its output specifies what failed, when, why, and with what severity.

Critical strengths
  1. Important and well-motivated problem. Event-level diagnosis is substantially more useful for robot-video analysis than a single quality score.

  2. High-quality benchmark construction. The benchmark includes temporal spans, severity, descriptions, double annotation with senior adjudication, a three-senior gold subset, and explicit schema-coverage analysis.

  3. Transparent limitations and reporting. The appendices carefully document annotation reliability, generator composition, inference cost, failure cases, and the limits of single-view RGB evidence.

Critical weaknesses
  1. Visual evidence is not fully controlled. RoboGaze examines videos through dense four-second windows, while the baselines may see fewer distinct frames, especially for long videos. Therefore, the improvement cannot be attributed solely to the proposed reasoning structure. A baseline using exactly the same frames and temporal windows is needed.

  2. Limited generator diversity. Most benchmark videos come from three systems sharing the Cosmos model family, while only 160 clips come from architecturally independent held-out generators. This limits the strength of the generalization claim across robot world models. More generators,such as seedance, need to be provided.

  3. Methodological novelty is moderate. RoboGaze mainly combines existing prompting components, temporal decomposition, specialist agents, routing, and critic verification. Comparisons with additional event-level evaluators such as Spotlight or VideoHallu would help establish whether the contribution goes beyond effective system integration.

Questions for authors
  1. How large was the development pool used for prompt design and selection of the verifier threshold, and how were these clips separated from benchmark conditions and source instructions?

  2. Can the authors report confidence intervals for the 40-condition screening experiment and clarify whether improvements translate to policy learning or task execution, rather than only human-rated visual usability?

10 comments
AuthorsOpenReview ↗

Response to Reviewer axGC (Part 1/5)

Thank you for your careful and constructive review, and for recognizing the importance of event-level diagnosis, the quality of our benchmark, and the transparency of our reporting. All revisions are highlighted in blue in the updated PDF.

Your three weaknesses can be read as three alternative explanations for RoboGaze's advantage: that it sees more frames (W1), that the benchmark favours one generator family (W2), or that it simply stacks more components (W3). We address each in turn, running the experiments you suggested wherever they apply. Brackets denote 95% bootstrap intervals.

W1: Controlling the Visual Evidence Available to Each Method

"RoboGaze examines videos through dense four-second windows, while the baselines may see fewer distinct frames, especially for long videos. [...] A baseline using exactly the same frames and temporal windows is needed."

We agree, and this was a gap in our protocol. The submission equalized the visual interface per call (Table 16) but not per clip, so methods that issue several calls read more distinct frames: on Gemma4-31B, 30 per clip for a single call, 48 for GLiDE+ and 88 for RoboGaze.

We ran the control you proposed (also requested by Reviewer wpUt). Each clip is sampled once at 4 FPS and cut into non-overlapping 4 s windows, and every image-reading stage of every method may read only these frames. We also added the baseline you describe: our single-call baseline B1, with RoboGaze's schema and rubric, run once per window.

Gemma4-31B, 914 clips, shared frames. Columns are per-clip means, so F1 is not the harmonic mean of P and R.

MethodPRF1Clean (210 failure-free clips)
B1, windowed345942.8.33
GLiDE+445247.2.67
RoboGaze595556.9.86

RoboGaze's margin over GLiDE+ (F1) under each control.

SettingMargin
Submission (each method's own frames)11.4
Shared frames9.7 [8.6, 11.0]
Shared frames, GLiDE+ at RoboGaze's token budget9.3
Shared frames, clips above 40 s7.8 [5.1, 10.9]
Gemini 3.1 Pro, gold subset†: own frames / shared12.2 / 8.8 [5.9, 11.4]

†Scored like the recomputed Table 23 (gold references, clip-level averaging).

These results lead to four findings.

Coverage explains only a small part of the margin. Sharing frames helps GLiDE+ slightly (+1.3 F1) and costs RoboGaze slightly (−0.4), because RoboGaze's re-inspection can no longer read at 8 FPS. Most of the margin therefore survives. Coverage matters more on Gemini 3.1 Pro, whose API accepts only 16 frames per call, as one would expect if part of the margin reflected coverage.

More frames alone mostly add false alarms. Windowed B1 reads exactly the windows that RoboGaze's observer reads, and it indeed recovers more failures than B1 run once (recall 59 vs. 47). Yet its clean accuracy drops sharply (.33 vs. .58), because each additional view is another chance to report a failure that did not occur, and nothing in a single call checks these proposals against what the task required. What separates RoboGaze is therefore how it decides which candidates to keep, which we analyze in Part 3.

On long clips, part of our original claim reflected coverage. As you anticipated, a single call must subsample a long clip, and this explains most of the margin's growth with duration under native frame limits. With shared frames RoboGaze still leads above 40 s and localizes events better (mIoU .49 vs. .37), but no longer by more than on short clips. Sec. 5.3 now reports both frame protocols, and the abstract no longer claims that the advantage is largest on long clips.

Matching compute does not close the gap either. Because RoboGaze also read more input tokens under the original protocol (85k vs. 52k), we raised GLiDE+'s cap on re-inspection and zoom steps from 4 to 7, which matches RoboGaze's token budget on the development pool. The added budget helps GLiDE+ only marginally (47.6 F1, from 47.2).

In summary, when inputs and compute are equalized, most of RoboGaze's advantage remains, and the controls point to how candidate failures are verified rather than to how much of the video is seen. The revision reports every arm of this control (App. E.7), frames per clip (Table 16), and cost in tokens, time and requests (Fig. 5, Table 19).

AuthorsOpenReview ↗

Response to Reviewer axGC (Part 2/5)

W2: Generalization Beyond the Cosmos Family

"Most benchmark videos come from three systems sharing the Cosmos model family, while only 160 clips come from architecturally independent held-out generators. [...] More generators such as seedance, need to be provided."

We agree that evidence beyond the Cosmos family was the thinnest part of the benchmark, a limitation that App. C.3 of the submission already acknowledged. We therefore added Seedance, as you suggested, which raises the number of clips from general-purpose generators from 160 to 240.

To keep the comparison controlled, we rendered Seedance 1.0 Pro (image-to-video, 8 s, no robot-specific training) on the same 80 conditions as Wan 2.2 and Veo, so that the three differ only in the generator, not in the task mix. The new clips were annotated with the benchmark's own procedure, two independent passes followed by senior adjudication, with detection κ = .72 (.73 on the benchmark).

Gemma4-31B, description F1 on the 80 shared conditions. The paper's +12.4 and +13.0 for Wan 2.2 and Veo are pooled over eight backbones.

GeneratorCoTGLiDE+RoboGazeMargin over GLiDE+
Wan 2.218.940.852.9+12.1 [6.0, 18.5]
Veo18.239.552.6+13.1 [7.4, 19.6]
Seedance 1.0 Pro (new)19.441.052.6+11.6 [6.1, 17.9]

The margin does not depend on the generator. RoboGaze leads GLiDE+ by a similar amount on all three generators, and on each by at least as much as on the full benchmark. The result also holds with Gemini 3.1 Pro as the evaluator (+12.9 [6.2, 19.0] on Seedance). Moreover, because these clips last only 5 to 8 s, every method on an open-weight backbone reads them in full, so frame coverage (W1) cannot account for the gap here.

Nor does it depend on a link to Cosmos. To our knowledge, Veo and Seedance share no component with the Cosmos family, yet their margins match that of Wan 2.2, whose only overlap is the Wan 2.1 VAE adopted by Cosmos-Predict2.5 (App. C.3).

Why the core benchmark builds on Cosmos. Most open robot world models used today to generate training data are built on Cosmos variants, so a benchmark that samples the generators robot-learning pipelines actually use is necessarily weighted toward this family. In particular, the three core systems cover three distinct generation mechanisms (App. C.3, Table 9):

  • Cosmos-Predict2.5 (14B) is fine-tuned per task and rolls out each clip in one shot.
  • DreamGen (on Cosmos-Predict2, 2B) renders synthetic trajectories segment by segment to expand policy-training data.
  • DreamDojo (Cosmos-family action-conditioned DiT, 2B) acts as a simulator and rolls out autoregressively for as long as its actions run.

Since the rollout mechanism determines how long a clip can be, these systems give the benchmark its long horizon, up to 148.9 s with 130 clips above 40 s. General-purpose generators, which stop at 5 to 8 s, could not provide this. Sec. 5.3 now states the resulting division explicitly: evidence across model families comes from short clips, and long-horizon evidence from robot-specific generators.

Extending the benchmark. Seedance is also the first use of an extension protocol that the revision documents in App. C.13, covering conditions, rendering settings, annotation and adjudication. Because the 914 benchmark clips are left untouched, every reported number remains comparable (App. C.13), and the Seedance results form a separate block (App. E.11). In addition, the "Use & extend" section of the project page shows how to add a generator, a backbone or a comparator.

In summary, RoboGaze's advantage holds on every generator we could test outside the core set, including two with no tie to Cosmos, and the benchmark's reliance on Cosmos reflects the robot world models in current use rather than a choice that favours our method.

AuthorsOpenReview ↗

Response to Reviewer axGC (Part 3/5)

W3: Clarifying the Methodological Contribution

"RoboGaze mainly combines existing prompting components, temporal decomposition, specialist agents, routing, and critic verification. Comparisons with additional event-level evaluators such as Spotlight or VideoHallu would help establish whether the contribution goes beyond effective system integration."

We agree that temporal windows, specialist agents, routing and verification each have precedents, and we do not claim any of them in isolation. We respectfully emphasize, however, that our contribution lies in what these components are organized around: the robot's task. Existing event-level evaluators, including GLiDE and Spotlight, ask whether a video looks wrong, and even when they are given the instruction and scene, as GLiDE+ and B1 are, they judge the clip as a whole. RoboGaze instead asks whether each part of the video achieved what its subtask was supposed to achieve, and every component is reformulated to answer that question.

Why this matters for robot video. In manipulation, many visual events are failures only relative to the task. A gripper closing on a cup is correct during a grasp, but a failure if the cup should already have been released; similarly, an object leaving the hand is correct at placement, but a failure mid-transport. A task-agnostic detector cannot tell these cases apart, so it either misses them or over-reports, which is the "cry-wolf" behaviour that keeps clean-clip accuracy low for standard VLM judges (Sec. 5.1).

How each component is reformulated around the task.

  • Task and scene memory turns the instruction into an ordered list of subtask specifications, each with its objects, expected outcome and completion criterion (Sec. 3.2). It thus provides an explicit, checkable expectation rather than a generic summary of the video.
  • Subtask segmentation aligns the video with this list through window-level progress signals, so that each segment is the execution of exactly one subtask. Sliding windows, as in Spotlight, cut the video by time and lose this correspondence.
  • Routing selects the failure dimensions relevant to each subtask (a grasp segment is not checked for action order), whereas one-agent-per-category designs examine every category everywhere.
  • The critic verifies each hypothesis against the specification of the subtask that produced it. Generic verifiers, such as GLiDE's debate or a propose-and-verify loop, can only check whether a claimed failure is visually plausible.

The components are therefore coupled rather than stacked: the specialists produce hypotheses tied to a subtask, and that tie is what gives the critic something concrete to falsify. This coupling also explains why their individual ablations (Table 21) are not additive.

Empirical support. Our experiments support this reading: the gain comes from the following factors (Gemma4-31B).

  • Assembling the same ingredients generically is not enough. Given our schema, Spotlight's two strategies, generic propose-and-verify and GLiDE+ reach 41.6 to 47.6 F1 across their own and shared frames, whereas RoboGaze reaches 57.3 and 56.9 under the same two settings (Part 1).
  • Verification is strongest when anchored to a subtask. Before verification, windowed B1 and the routed specialists are nearly equal (recall 59 for both; precision 34 and 36). The same critic then adds 6.8 F1 to the former but 12.8 to the latter, a difference of 6.0 [4.0, 7.9]. Moreover, replacing the subtask specification with the full instruction costs 3.3 F1 [2.1, 4.6] on original frames.
  • Task structure matters more as tasks grow longer. Removing subtask segmentation costs 1.2 F1 on single-subtask clips but 5.9 on clips with three or more subtasks.
  • The critic transfers, yet RoboGaze stays ahead. Attached to the baselines, the critic adds 1.5 to 7 F1, but the best critic-equipped baseline still trails RoboGaze by 7.3 F1 [5.9, 8.6].

Sec. 1, Sec. 3.4 and Sec. 6 now state the contribution as this task-conditioned coupling, rather than as three separate components. Part 4 reports the comparisons with Spotlight and VideoHallu that you requested.

Reviewer_47ScOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RoboGaze, a training-free multi-agent vision-language model (VLM) framework and benchmark for diagnosing failures in generated robot manipulation videos. The benchmark contains 914 clips ranging from 4.9 to 148.9 seconds, covering five video generators and three robot embodiments, with 1,847 annotated failure events. Each clip is independently annotated by two annotators and reviewed by a senior annotator, with an additional 180-clip gold-standard subset. RoboGaze employs a three-stage diagnostic pipeline: (1) task and scene grounding with window-level observation memory, (2) subtask routing to the top three dimension-specific specialists based on a taxonomy of six failure dimensions and 30 failure types, and (3) critic-based verification to accept, merge, or reject failure hypotheses with explicit abstention categories. The paper evaluates eight VLM backbones against CoT, self-consistency, and GLiDE+ baselines using event-description F1, temporal mIoU, severe-event recall, clean-clip accuracy, human validation, and downstream demonstration screening.

Critical strengths
  1. The paper introduces a structured diagnostic framework that moves beyond scalar scores to identify what failed, when, why, and how severely. The six-dimension, 30-type failure taxonomy (Figure 3 and Tables 5-6) and residual error analysis (Figure 11) enable systematic failure characterization and diagnosis.

  2. The paper presents a comprehensive benchmark of 914 clips across multiple video generators, including held-out models and long-horizon rollouts. The controlled B1-B6 baseline comparisons (Tables 17 and 19) help distinguish the contributions of the proposed architecture from those of prompting and inference configurations.

Critical weaknesses
  1. There are inconsistencies in API call accounting and implementation. The reported per-clip call counts in Table 19 and Appendix E.1 appear inconsistent with Algorithms 1-2. For an average 19.8 s clip, Stage 1 requires at least seven calls (two grounding calls and five observation windows), while Stage 2 adds at least four calls, yielding a minimum of 11 calls before verification. However, Table 19 reports only 7.6 and 9.4 calls for RoboGaze without and with the critic, respectively. The discrepancy is more pronounced for longer clips. Furthermore, the released code uses overlapping 2 s windows with a 1 s stride, rather than the non-overlapping 4 s windows described in the paper. The role of window-level observations in subsequent stages is also unclear, as they are absent from the router, specialist, and verifier inputs in the equations and pseudocode. These discrepancies raise concerns about implementation consistency and computational cost reporting.

  2. There are inconsistencies in benchmark statistics and aggregation. Several reported results require clarification. In Table 12, all five agreement metrics for the 619-clip short-horizon subset exactly match those from an earlier 382-clip benchmark, while the overall scores correspond to clip-count-weighted averages rather than the expected event-level aggregation. Similarly, the gold subset F1 scores in Table 23 exactly match the unweighted averages of the six dimension-level scores in Table 22, despite the stated clip-level macro-averaging protocol. Additionally, all 19 bootstrap confidence intervals in Table 20 are symmetric around their point estimates, which warrants verification. These issues raise concerns about the consistency and reproducibility of the reported evaluation results.

Questions for authors
  1. Could the authors explain the discrepancy between the reported 7.6 (B5) and 9.4 (B6) calls per clip and the minimum of 11 calls implied by Algorithms 1-2? Why does the released code use overlapping 2 s windows instead of the stated non-overlapping 4 s windows? Please also clarify whether window-level observations are passed to the router, specialists, and verifier.

  2. Why do the short-horizon agreement scores match an earlier benchmark, and why do the overall metrics correspond exactly to simple averages despite the stated event-matching and clip-level aggregation protocols? Were the confidence intervals in Table 20 computed using bootstrap percentiles or a normal approximation?

  3. Given the 16/32-frame limits in Table 16, what are the average subtask segment lengths, and how frequently are these limits reached during specialist and verifier inference on long videos? Additionally, how would the main results change if the 113 evidence-limiting abstention events (Appendix B.1) were excluded or treated as neutral rather than as failure predictions?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 47Sc (Part 1/5)

Thank you for the detailed reading of our method, tables and code, and for recognizing the diagnostic framework, the taxonomy and the controlled baseline comparisons. Revisions are highlighted in blue in the updated PDF.

Before the point-by-point response, we summarize what your questions concern, since this matters for how the reported results should be read. They fall into two groups. The first group concerns places where the paper described less than the pipeline actually does; the runs themselves are unaffected, and the revision now documents them in full. The second group concerns three secondary tables that had been computed under slightly different rules from those the paper states; we have recomputed them from the raw records under the stated rules.

QuestionWhat it concernsEffect on reported results
Calls per clip (Table 19)the column counted diagnostic stages onlynone; all stages now reported, time and cost unchanged
2 s windows in the public codean outdated development snapshotnone; every run used 4 s windows (manifests, traces)
Window observations, "top three" routinginputs and threshold omitted from Eqs. 2–5 and Algorithms 1–3none; the description now matches the implementation
Table 12 (agreement)one row from an earlier release; pooling of duration bandsvalues move by at most .02
Table 23 (gold subset)aggregation rule and reference setscores move by at most 0.5 F1; margins unchanged; gap closure 83% → 82%
Table 20 (intervals)normal approximation instead of percentilesall intervals still exclude zero

In short, Table 3 and every method comparison in Sec. 5 are reproducible as reported. Parts 1 and 2 cover the first group (W1, Q1), Parts 3 and 4 the second (W2, Q2), and Part 5 the frame limits and abstention events (Q3).

W1 and Q1: Call Accounting

"For an average 19.8 s clip, Stage 1 requires at least seven calls [...], yielding a minimum of 11 calls before verification. However, Table 19 reports only 7.6 and 9.4 calls [...]."

The pipeline runs every stage you list, so your count of the full pipeline is correct; the discrepancy lies in what Table 19's column counted. That column reported only the diagnostic stages (router, specialists, critic and span re-inspection), leaving out task and scene grounding and the window observer. The per-call traces on Gemma4-31B give the complete picture:

StageRoboGaze without criticRoboGazeRoboGaze, clips over 40 s
Grounding / window observer2.0 / 5.52.0 / 5.52.0 / 17.2
Router / specialists2.6 / 5.02.6 / 5.04.1 / 8.2
Critic / span re-inspectionnone0.9 / 0.91.0 / 2.6
Diagnostic stages only (as printed)7.69.415.9
All stages15.116.935.1

Counted this way, RoboGaze issues 16.9 requests per clip, above the minimum of 11 you derived, and GLiDE+ issues 12.5 rather than its nominal 10.0. Table 19 now reports measured requests and input tokens over the full pipeline, and App. E.1 gives the per-stage counts above, including on clips longer than 40 s.

AuthorsOpenReview ↗

Response to Reviewer 47Sc (Part 2/5)

W1 and Q1 (continued): Cost, Routing, Windows and Window Observations

Cost. Wall-clock time and USD in Table 19 were measured end to end, so they already reflect every stage and do not change: 152 s and USD 0.202 per clip for RoboGaze, against 156 s and USD 0.208 for GLiDE+. RoboGaze stays level in time despite issuing more requests because its window and specialist calls run in parallel, whereas GLiDE+'s loop is sequential. Since RoboGaze does read more input tokens (85k vs. 52k), "inference budget" was ambiguous, and the abstract now states the comparison under matched frames instead. With GLiDE+ raised to RoboGaze's token budget, RoboGaze still leads by 9.3 F1 (our reply to Reviewer axGC, Part 1).

Routing. The implementation keeps the dimensions whose router score exceeds θ = .30, up to three, which yields 1.9 specialists per segment on average; Sec. 3.3 had stated only the upper bound. Eq. 3, Algorithm 2 and App. D.1 now give the threshold, and Eq. 5 and Algorithm 3 make explicit that the critic reads the frames each hypothesis cites, makes at most one call per clip, and re-reads an accepted span at 8 FPS only when it crosses a window boundary or lasts longer than 2 s.

Windows. Every reported run uses non-overlapping 4 s windows of 16 frames, as the paper states; the 2 s, 1 s-stride default you found belongs to an earlier development snapshot of the public code, while the reported runs set the window in their configuration files. Three independent records confirm this:

  • Run manifests (App. D.1) record a 4 s window, a 4 s stride and 16 frames for every reported run.
  • Per-call traces record the frame timestamps of each observer call, and each call spans one 4 s interval.
  • Call counts follow from the configuration: non-overlapping 4 s windows give ⌈L/4⌉ observer calls on a clip of length L, 5.5 on average as measured, whereas a 1 s stride would give about L − 1, roughly 19.

We are updating the repository so that its defaults match the reported configuration; the 2 s row of Table 18 likewise used non-overlapping windows.

Where the window observations go. The observations are used twice. Their progress signals first determine the subtask segments (Eq. 2), with every window assigned to exactly one subtask so that the segments partition the video. Each segment then carries a summary of its own windows (object visibility, displacement and identity, robot action and contact), which the router and the specialists read. The critic deliberately does not read these summaries: it sees the task and scene memory, the hypotheses and the frames they cite, so that its verdict rests on visual evidence rather than on another model's description of it. Because Eqs. 2–4 and Algorithms 1–2 had omitted the segment summary, they now include it as an explicit input, and Sec. 3.2 states that the segments partition the video.

AuthorsOpenReview ↗

Response to Reviewer 47Sc (Part 3/5)

W2 and Q2: Benchmark Statistics and Aggregation

"In Table 12, all five agreement metrics for the 619-clip short-horizon subset exactly match those from an earlier 382-clip benchmark, while the overall scores correspond to clip-count-weighted averages [...]. The gold subset F1 scores in Table 23 exactly match the unweighted averages of the six dimension-level scores in Table 22 [...]. All 19 bootstrap confidence intervals in Table 20 are symmetric."

These three tables belong to the second group: in each, the computation followed a slightly different rule from the one the paper describes, and we have now aligned them. They touch the method comparison only at its edges: Table 12 describes annotator agreement, Table 20 changes the shape of the intervals but not the margins they surround, and Table 23 feeds a single headline figure, the gap closure on the gold subset, which moves from 83% to 82%. We recomputed every flagged cell from the raw records; values move by at most .02 in agreement and 0.5 F1 on the gold subset, and no comparison or conclusion changes.

The common cause is that the paper never stated the unit and the pooling rule of each metric, which left room for a table to be computed one way and described another. App. D.2 now defines both for every metric:

MetricUnitPooled across slices by
Detection κclipsrecomputing on the pooled clips
Dimension κ, severity αevents marked by either annotator; an event the other did not mark counts as a disagreement with "no event" (severity 0)recomputing on the pooled events
Temporal IoU, description similarityevent pairs matched between annotatorsweighting by matched pairs
Description F1clips with at least one reference event (704 of 914)weighting by those clips
Clean accuracyfailure-free clips (210 of 914)weighting by those clips

Table 12 (annotator agreement). The ≤15 s row still held the values of an earlier 382-clip release and had not been refreshed for the 619 short clips of the current benchmark. The overall row, in turn, averaged the duration bands by clip count, which closely approximates recomputing on the pooled data but is not identical to it. Every cell is now recomputed from the 914 clips with the rules above:

Det. κDim. κTemp. IoUSev. αDesc. sim.
Overall, printed.74.66.61.58.73
Overall, recomputed.73.64.60.56.72
≤15 s, printed.77.69.64.61.75
≤15 s, recomputed (619 clips).76.68.62.60.75

The other duration bands and the per-dimension rows were already computed on the current clips and do not change, so Fig. 13, which uses the per-dimension κ, is unchanged; Sec. 4 now reports κ = .73.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Strengths and How We Addressed the Main Concerns

Dear Area Chair and Reviewers,

We thank the reviewers for their careful and constructive reviews. We have answered each review point by point and uploaded a revised PDF, with all changes marked in blue. Below, we summarize the strengths the reviewers recognized and how we addressed their main concerns.

Contributions and strengths recognized by the reviewers

  • Contribution
    • Important, well-motivated problem. Event-level diagnosis, which states what failed, when, why and how severely, was seen as substantially more useful for robot video than a single quality score (axGC, 47Sc).
    • Structured diagnostic framework. The six-dimension, 30-type failure schema and the residual error analysis enable systematic failure characterization (47Sc).
    • High-quality, reusable benchmark. RoboGazeBench was valued for its temporal spans, severities and descriptions, double annotation with senior adjudication, gold subset, held-out generators and long-horizon rollouts, and as a resource for future work (axGC, 47Sc, wpUt).
  • Soundness
    • Thorough, controlled experiments. The ablations and the B1–B6 controls show where the gains come from, separating the architecture from prompting and inference choices (wpUt, 47Sc).
    • Consistent results. RoboGaze outperforms every baseline regardless of the VLM backbone (wpUt).
    • Transparent reporting of annotation reliability, cost, failure cases and limitations (axGC).
  • Presentation: clear and easy to follow (47Sc, wpUt).

How we addressed the main concerns

The reviewers' concerns fall into five themes. For each, we ran the control or analysis that the reviewers proposed (brackets denote 95% bootstrap intervals).

  1. Equal visual evidence (axGC W1, wpUt W1). Both reviewers asked whether RoboGaze wins simply because it reads more frames. We therefore restricted every method to the same 4 s windows and added the windowed baselines they requested. Under this control, RoboGaze still leads GLiDE+ by 9.7 F1 [8.6, 11.0], against 11.4 originally, and by 9.3 when GLiDE+ also receives RoboGaze's token budget. For the baselines, extra frames mostly produce false alarms; what sets RoboGaze apart is how it verifies candidate failures (App. E.7).

  2. Contribution beyond system integration (axGC W3, wpUt W3/Q1). This raises the question of what RoboGaze contributes beyond combining known components. Its core idea is to organize evaluation around the robot's task: each specialist ties its hypothesis to a subtask, with that subtask's objects, expected outcome and completion criterion, and the critic checks the hypothesis against this specification. It is this coupling that makes verification effective. With the same critic attached to the baselines, as Reviewer wpUt suggested, the routed specialists still add 7.3 F1 [5.9, 8.6], and the critic improves them nearly twice as much as it improves a windowed baseline. Spotlight's strategies, given our schema, trail RoboGaze by at least 13.5 F1 (App. E.8, E.9).

  3. Generalization (axGC W2, wpUt W2/Q3). A further question is whether this advantage extends beyond our generators and data. We added Seedance, as Reviewer axGC suggested, rendered on the same conditions as Wan 2.2 and Veo. RoboGaze's margin on it (+11.6 F1 [6.1, 17.9]) matches theirs, so the advantage does not depend on the Cosmos family. On Spotlight's external benchmark of non-robot video, RoboGaze runs without any adaptation and matches Spotlight's best strategy (App. C.13, E.9, E.11).

  4. Reporting consistency (47Sc W1/W2). We resolved each question on call accounting and table aggregation. Table 19 now counts the calls of every pipeline stage, not only the diagnostic ones; time and cost were measured end to end and are unchanged. We recomputed Tables 12, 20 and 23 under the protocol the paper states, and App. D.2 now defines the unit and pooling rule of every metric. All method comparisons and conclusions in Sec. 5 remain unchanged.

  5. Downstream use (axGC Q2, wpUt W4). Finally, both reviewers asked whether better diagnosis leads to better training data. We added the paired intervals Reviewer axGC requested for the screening study: clips ranked highly by RoboGaze are usable at .80, against .71 for GLiDE+, a gain of 9 points [2, 17] (App. E.5). Sec. 6 now also describes how event-level reports can feed policy training, for instance by masking the action loss inside a reported failure span.

The remaining minor questions and presentation suggestions are fully addressed in the individual replies.

We would be grateful to know whether these replies address your concerns, and we are happy to run further analyses while the discussion period is open.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.