acceptodds
Under review as a conference paper at ICLR 2027

RoboChrono: A Real Robot Benchmark for Streaming Task Understanding

Abstract

Existing video benchmarks mainly use human centric or passively recorded footage, offering limited direct evaluation of temporal understanding in robot task execution. We introduce RoboChrono, a benchmark with 39 scenarios and 34,754 distinct questions from controlled executions on GIMME_ARM and Marvin Pro, together with bare hand human recordings of five of the same tasks. RoboChrono covers diverse embodiments, end effectors, and viewpoints and evaluates seven capabilities grouped into recognition, alignment, and temporal grounding. Separate scores assess action understanding and the ability to connect observations across time and camera views. Evaluation of 17 vision language models reveals stable overall rankings across pairs of evaluation sets with no shared questions (Spearman –), yet aggregate scores conceal marked differences between capabilities. Doubao-Seed-2.0-Lite achieves the highest observed recognition accuracy (71.7%) and temporal grounding score (0.576 [email protected]). Gemini-3.6-Flash ranks first in alignment (cross view matching and frame ordering) but last in recognition, while placing thirteenth overall. Median frame ordering accuracy across models and sets is 28.9%, close to the 25% chance baseline. These findings expose gaps in temporal and cross view understanding and motivate capability specific evaluation alongside aggregate scores.

Primary area:
datasets and benchmarks
Keywords:
Robot Egocentric Video Understanding, Streaming Temporal Task Understanding, Vision-Language Models, Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_A778OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RoboChrono, a real-robot manipulation benchmark built from 1,816 teleoperated dual-view episodes across 39 scenarios. By truncating videos at intermediate timestamps and pairing them with annotated action and state intervals, the benchmark creates 34,713 instances across six multiple-choice tasks and one action-time localization task, evaluating 18 zero-shot VLMs.

Critical strengths
  1. The dataset is built on 25 hours of quality-controlled real-robot demonstrations with synchronized head and wrist views and manual temporal interval annotations, providing a well-grounded resource across gripper and dexterous embodiments.
  2. The input-modality ablation in Table 5 (comparing full video, current frame only, blind text-only, and shuffled frames) provides a transparent diagnostic of what signals models rely on.
Critical weaknesses
  1. The paper's own input ablation undermines the construct validity of key tasks and the "streaming" premise of the benchmark. In Table 5, removing visual input altogether reduces Next Action accuracy by only 0.7 points (40.0% full video vs. 39.3% blind), Goal-Conditioned Next Action reaches 41.1% blind, and Current Action reaches 40.5% blind against a 25% random baseline, indicating that the anticipation tasks largely measure language priors over option templates rather than visual tracking of task progress. Moreover, shuffling video frames improves Frame Matching by 10.4 points, and the "streaming" protocol simply evaluates models offline on a single truncated clip without multi-turn queries or a full-video comparison. Because a benchmark's value depends on whether its scores reflect the intended visual-temporal capabilities, these shortcuts directly affect the validity of the paper's conclusions.
  2. The evaluation protocol is under-specified and exhibits signs of unexamined parsing or input artifacts, which makes the cross-model capability analysis unreliable. Essential video input details—such as frame sampling count, FPS, and resolution—are omitted, and no human baseline or inter-annotator agreement is reported for the short 4–7 second localization intervals. In addition, several results point to scoring or refusal artifacts: for example, Gemini-3.6-Flash scores at or below chance on Frame Matching (24.5%) and Current Action (28.4%) despite reaching 60.8% on View Matching, and blind Frame Matching drops to 15.8%. Because runs are only excluded when parse failures exceed 50% and per-model failure rates are not reported, the capability profiles compared in Section 4.2 may reflect formatting failures rather than visual understanding.
Questions for authors
  1. Why does the blind text-only baseline perform nearly as well as full video input on Next Action (39.3% vs. 40.0%), and do the authors plan to filter or redesign distractors that can be solved from language priors alone?
  2. Why does random frame shuffling improve Frame Matching accuracy by 10.4 points in Table 5, and how does the single-query truncated clip evaluation differ in conclusions from full-video evaluation?
  3. Could the authors report the frame sampling protocol (frame count, FPS, resolution), per-model parse-failure rates (especially for Gemini-3.6-Flash), and human performance / inter-annotator agreement on the benchmark?
3 comments
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer A778,

Thank you again for your time and constructive review. As the discussion period is drawing to a close, we would like to briefly follow up on our rebuttal and highlight the additional analyses addressing your main questions:

  • Visual evidence and language priors (W1/Q1): Across all 4,880 Next Action questions, visual input improves accuracy by 15.3-24.4 percentage points across four models. After applying the blind-baseline filter, RynnBrain1.1-122B, which did not define the filter, retains 50.4% accuracy versus 51.1% overall.

  • Frame shuffling and causal history (W1/Q2): We added controls at 2 and 4 fps, identity re-encoding, and comparisons using the full causally available episode prefix. These establish task-dependent responses: shuffling improves order-invariant Frame Matching and reduces Next Action accuracy in seven of eight models, while the additional history significantly improves anticipation for four models.

  • Gemini and evaluation protocol (W2/Q3): Our response includes sampling configurations, parsing checks, and the Gemini-native rerun on 8,716 of its 8,720 questions. All 8,716 answers are parseable; Current Action and Frame Matching reach 77.5% and 94.2%, respectively. These results directly address the anomalous scores highlighted in your review.

  • Human baseline and agreement (W2/Q3): Four annotators answered the same 30 questions per multiple-choice task. Mean accuracy reaches 80.0% on Next Action and 95.0% on Frame Matching, with Fleiss' kappa of 0.70-0.90 across the six multiple-choice tasks. Our response also reports temporal IoU agreement for Action Time.

We would be grateful if you could review these additions and let us know whether they address your concerns. We remain happy to provide further clarification on any remaining questions.

Best regards, The Authors

Reviewer_A778OpenReview ↗

Official Comment by Reviewer A778

Thank you for your thoughtful response and for the work you put into addressing my questions during the rebuttal period. After considering your response, I’ve decided to keep my original score.

AuthorsOpenReview ↗

Thank You and Follow-up on Remaining Concerns

Dear Reviewer A778,

Thank you for your reply and for taking the time to consider our rebuttal and additional experiments. We appreciate your continued engagement with our work.

To better understand your decision to maintain the original rating, could you let us know which concerns, if any, remain unresolved after our responses? Identifying these specific points would help us provide a more focused clarification. We would be happy to address any remaining questions or provide further details on the additional analyses.

Thank you again for your time and consideration.

Best regards,
The Authors

Reviewer_MrAPOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The proposed RoboChrono benchmark evaluates visual history understanding for robotic manipulation tasks. It covers seven evaluation tasks grouped into recognition, alignment, and temporal grounding, spanning action understanding and anticipation, visual correspondence, temporal ordering, and action localization. An interesting finding is that removing visual observations does not influence Next Action Prediction.

Critical strengths
  1. The benchmark evaluates the ability to understand visual history across seven types of abilities, including recognition, alignment, and temporal grounding, and covering action understanding and anticipation, visual correspondence, temporal ordering, and action localization. The tasks are conducted across 39 diverse scenarios and 1,816 episodes involving both real robots and humans.
  2. It considers both the ego view and the wrist view, which is particularly important for robot manipulation.
  3. Most state-of-the-art (SOTA) models are included (except for Gemini), and the experimental analysis is comprehensive. The data pipeline also ensures the quality of benchmark construction.
Critical weaknesses
  1. The relationship between task success rate and the seven types of abilities remains unclear. For example, is action localization important, or is merely recognizing the action order sufficient?
  2. The multi-selection QA tasks may underestimate the prediction difficulty. Is there any way to ensure that the negative choices are challenging?
  3. The results show that removing visual observations does not influence Next Action Prediction, and that shuffling the frames improves Frame Match performance. More analysis and case studies may be helpful to understand these phenomena.
Questions for authors
  1. Could you show the relationship between task success rate and the seven types of abilities.
  2. Is there any way to ensure that the negative choices are challenging?
  3. The results show that removing visual observations does not influence Next Action Prediction, and that shuffling the frames improves Frame Match performance. More analysis and case studies may be helpful to understand these phenomena.
Reviewer_WrAMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark of streaming tasks in real-world robot manipulation. The goal is to provide the community with a testbed to evaluate video understanding when the acting agent is not a human, covering a variety of scenarios and tasks as well as two robotic platforms. The authors further evaluated 18 visual-language models in a zero-shot setting, showing substantial room for improvement in temporal progress understanding.

Critical strengths
  • the dataset offers an interesting physical grounding for the study of robotic actions and their temporal video understanding
  • the experimental analysis includes a large number of models and highlights an interesting discrepancy between frame matching and frame ordering
  • overall the paper is well organized and easy to read (although some of the figures' visibility and captions should be improved, in particular Fig. 3 and 5)
Critical weaknesses
  1. It is unclear to what extent the observed performance patterns and trends are specific to the proposed benchmark or reflect broader limitations of current multimodal models. There is an extensive literature documenting the reliance of video models on static appearance cues, their limited sensitivity to temporal order, and the resulting weaknesses in temporal understanding. Therefore, some of the findings reported in this paper are not entirely surprising. A clearer characterization of the benchmark's added value and of the novel insights it provides beyond existing evaluations would strengthen the contribution. [a] TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models. ICLR 2025. [b] Lost in Time: A New Temporal Benchmark for VideoLLMs. BMVC 2025. [c] V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning. CVPR 2026.

  2. The ablation results show that removing visual observations has only a marginal impact on performance for some tasks. This finding deserves a more thorough investigation, as it raises questions about the extent to which the evaluated models genuinely rely on visual evidence to infer task progress. For instance, counterfactual experiments could keep the task instructions fixed while changing the observed manipulation state, allowing the authors to assess whether predictions respond appropriately to changes in the visual evidence. More generally, the paper would benefit from a deeper analysis of this result to identify its underlying causes and derive more informative conclusions, rather than simply reporting the performance gap.

  3. The use of multiple-choice questions is practical and facilitates reproducible comparisons across models. However, this evaluation format may allow models to exploit answer priors, eliminate implausible options, or rely on stereotypical action sequences without genuinely understanding the temporal context. Complementing the current evaluation with open-ended predictions, where applicable, would help assess whether the reported capabilities persist when such shortcuts are less readily available.

  4. While the benchmark reveals differences in performance across tasks and models, the paper provides limited insight into the specific mechanisms underlying these failures. A more fine-grained analysis would make the contribution more informative and actionable. Relevant failure modes to investigate may include failure to track state changes across observations, confusion between completed and ongoing actions, dependence on stereotypical action sequences and task priors, sensitivity to observation sampling frequency and history length, etc...Characterizing these failure modes would help clarify what current models struggle with and how the benchmark can guide future research.

Questions for authors
  • To what extent do the observed performance patterns generalize beyond the proposed dataset or was already observed in other streaming benchmarks?

  • How much of the observed difficulty stems from robot-specific visual evidence and manipulation dynamics, as opposed to general limitations in temporal understanding?

  • Does the benchmark reveal novel failure modes, or does it primarily reproduce known weaknesses of multimodal models in temporal reasoning?

  • Given that removing visual observations has only a marginal impact on some tasks, what explains this behavior? Can counterfactual experiments establish whether predictions are genuinely sensitive to changes in the observed manipulation state?

1 comment
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer WrAM,

Thank you again for your time and constructive review. As the discussion period is drawing to a close, we would like to briefly follow up on our rebuttal and highlight the additional evidence addressing your main concerns:

  • Added value and failure analysis (W1/W4): Our response clarifies RoboChrono's contribution to manipulation-progress understanding and provides analyses of completed-versus-next errors, the stage dependence of task priors, observation history, and sampling frequency. These analyses characterize specific difficulties in tracking manipulation progress beyond aggregate performance differences.

  • Visual dependence (W2/Q4): Across all 4,880 Next Action questions, visual input improves accuracy by 15.3-24.4 percentage points across the four evaluated models. The analysis by episode stage also contextualizes the small gap in the original subset, which is concentrated on initial action segments.

  • Counterfactual state changes (W2/Q4): Following your suggestion, we keep the questions and options identical and replace only the observed clips. On Current Action, all eight models significantly favor the state-consistent answer over retaining the original answer. For RynnBrain1.1-122B, these proportions are 75.8% versus 8.1%, conditional on originally correct answers. We also analyze Next Action swaps separately according to whether the replacement introduces a current-versus-next ambiguity.

  • Multiple-choice shortcuts (W3): After filtering questions answered correctly by at least two of three blind baselines, 3,565 Next Action questions remain. RynnBrain1.1-122B, which was not used to define the filter, achieves 50.4% with video, compared with 51.1% on the complete set.

We would be grateful if you could review these additions and let us know whether they address your concerns, or whether any specific points would benefit from further clarification. We remain happy to discuss any remaining questions.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.