- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces RoboChrono, a real-robot manipulation benchmark built from 1,816 teleoperated dual-view episodes across 39 scenarios. By truncating videos at intermediate timestamps and pairing them with annotated action and state intervals, the benchmark creates 34,713 instances across six multiple-choice tasks and one action-time localization task, evaluating 18 zero-shot VLMs.
Critical strengths
- The dataset is built on 25 hours of quality-controlled real-robot demonstrations with synchronized head and wrist views and manual temporal interval annotations, providing a well-grounded resource across gripper and dexterous embodiments.
- The input-modality ablation in Table 5 (comparing full video, current frame only, blind text-only, and shuffled frames) provides a transparent diagnostic of what signals models rely on.
Critical weaknesses
- The paper's own input ablation undermines the construct validity of key tasks and the "streaming" premise of the benchmark. In Table 5, removing visual input altogether reduces Next Action accuracy by only 0.7 points (40.0% full video vs. 39.3% blind), Goal-Conditioned Next Action reaches 41.1% blind, and Current Action reaches 40.5% blind against a 25% random baseline, indicating that the anticipation tasks largely measure language priors over option templates rather than visual tracking of task progress. Moreover, shuffling video frames improves Frame Matching by 10.4 points, and the "streaming" protocol simply evaluates models offline on a single truncated clip without multi-turn queries or a full-video comparison. Because a benchmark's value depends on whether its scores reflect the intended visual-temporal capabilities, these shortcuts directly affect the validity of the paper's conclusions.
- The evaluation protocol is under-specified and exhibits signs of unexamined parsing or input artifacts, which makes the cross-model capability analysis unreliable. Essential video input details—such as frame sampling count, FPS, and resolution—are omitted, and no human baseline or inter-annotator agreement is reported for the short 4–7 second localization intervals. In addition, several results point to scoring or refusal artifacts: for example, Gemini-3.6-Flash scores at or below chance on Frame Matching (24.5%) and Current Action (28.4%) despite reaching 60.8% on View Matching, and blind Frame Matching drops to 15.8%. Because runs are only excluded when parse failures exceed 50% and per-model failure rates are not reported, the capability profiles compared in Section 4.2 may reflect formatting failures rather than visual understanding.
Questions for authors
- Why does the blind text-only baseline perform nearly as well as full video input on Next Action (39.3% vs. 40.0%), and do the authors plan to filter or redesign distractors that can be solved from language priors alone?
- Why does random frame shuffling improve Frame Matching accuracy by 10.4 points in Table 5, and how does the single-query truncated clip evaluation differ in conclusions from full-video evaluation?
- Could the authors report the frame sampling protocol (frame count, FPS, resolution), per-model parse-failure rates (especially for Gemini-3.6-Flash), and human performance / inter-annotator agreement on the benchmark?
3 comments
Official Comment by Reviewer A778
Thank you for your thoughtful response and for the work you put into addressing my questions during the rebuttal period. After considering your response, I’ve decided to keep my original score.
Thank You and Follow-up on Remaining Concerns
Dear Reviewer A778,
Thank you for your reply and for taking the time to consider our rebuttal and additional experiments. We appreciate your continued engagement with our work.
To better understand your decision to maintain the original rating, could you let us know which concerns, if any, remain unresolved after our responses? Identifying these specific points would help us provide a more focused clarification. We would be happy to address any remaining questions or provide further details on the additional analyses.
Thank you again for your time and consideration.
Best regards,
The Authors
Follow-up on Our Rebuttal
Dear Reviewer A778,
Thank you again for your time and constructive review. As the discussion period is drawing to a close, we would like to briefly follow up on our rebuttal and highlight the additional analyses addressing your main questions:
Visual evidence and language priors (W1/Q1): Across all 4,880 Next Action questions, visual input improves accuracy by 15.3-24.4 percentage points across four models. After applying the blind-baseline filter, RynnBrain1.1-122B, which did not define the filter, retains 50.4% accuracy versus 51.1% overall.
Frame shuffling and causal history (W1/Q2): We added controls at 2 and 4 fps, identity re-encoding, and comparisons using the full causally available episode prefix. These establish task-dependent responses: shuffling improves order-invariant Frame Matching and reduces Next Action accuracy in seven of eight models, while the additional history significantly improves anticipation for four models.
Gemini and evaluation protocol (W2/Q3): Our response includes sampling configurations, parsing checks, and the Gemini-native rerun on 8,716 of its 8,720 questions. All 8,716 answers are parseable; Current Action and Frame Matching reach 77.5% and 94.2%, respectively. These results directly address the anomalous scores highlighted in your review.
Human baseline and agreement (W2/Q3): Four annotators answered the same 30 questions per multiple-choice task. Mean accuracy reaches 80.0% on Next Action and 95.0% on Frame Matching, with Fleiss' kappa of 0.70-0.90 across the six multiple-choice tasks. Our response also reports temporal IoU agreement for Action Time.
We would be grateful if you could review these additions and let us know whether they address your concerns. We remain happy to provide further clarification on any remaining questions.
Best regards, The Authors