- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EgoExo-Next, a visual-option benchmark designed to evaluate dynamic visual-state reasoning in vision-language models. The benchmark contains 2,503 human-curated four-choice questions constructed from six publicly available egocentric and ego–exo video sources. It comprises four interconnected subtasks: Ego Next-State, Ego–Exo Matching, Exo Next-State, and Ego-to-Exo Next-State. A central feature is the use of controlled earlier-state and later-future distractors, intended to prevent models from relying on simple temporal shortcuts. The authors evaluate proprietary, open-source, and spatial reasoning VLMs. The best fully evaluated model achieves 43.81% average accuracy, substantially below the human performance of 98.55%. Additional analyses, including candidate-only evaluation, temporal shuffling, mismatched-observation controls, visual similarity baselines, and oracle-input experiments, are provided to investigate the validity and difficulty of the benchmark.
Critical strengths
- A useful decomposition of temporal and cross-view reasoning. The four subtasks examine temporal progression within each viewpoint and synchronized cross-view matching, then combine these requirements in Ego-to-Exo Next-State. The matched-subset oracle experiments make this design more informative, as providing the current exocentric state or the future egocentric state improves performance, offering insight into the temporal and cross-view requirements of the task.
- Candidate design supported by control experiments. Using candidates from the same action trajectory reduces the usefulness of coarse action labels, while later-future distractors require models to distinguish among future states rather than simply separate past from future. The candidate-only, shuffled-sequence, mismatched-observation, and CLIP/DINO experiments test several plausible shortcuts, while independent human verification helps ensure that the questions are answerable.
Critical weaknesses
Future-state prediction is not clearly distinguished from temporal ordering. The next-state tasks use candidates from the same action trajectory and define the target as the nearest sampled future state. With nonuniform sampling and no fixed prediction horizon, models may identify the action stages represented by the observations and candidates, then select the earliest candidate after the observation sequence. The controls and similarity baselines test several potential shortcuts but do not directly address whether temporal ranking could explain the results.
Questions for authors
Could the authors clarify how the current experiments distinguish future visual-state prediction from a strategy that identifies the action stages of the observations and candidates, then selects the earliest candidate after the observation sequence?
4 comments
Response to Reviewer pJah
Q1: Distinction Between Future Visual-State Prediction and Temporal Ordering
We thank the reviewer for this important question. We do not claim that next-state identification and temporal ordering are entirely independent. Rather, our focus is on visual-state progression and its composition with cross-view correspondence. Inferring temporal order from images itself requires understanding fine-grained changes in object configurations, human poses, and action progression, rather than relying on explicit temporal annotations.
EgoExo-Next evaluates this reasoning through visual-option next-state identification, where models must distinguish the subsequent state from earlier and later-future candidates along the same action trajectory. Importantly, Subtask 4 further requires identifying future exocentric states from egocentric observations, integrating temporal state progression with viewpoint transformation.
As shown in W1, explicit temporal ordering does not improve accuracy over direct next-state identification (40.37% vs. 41.77%), indicating that recovering temporal relationships remains challenging for the evaluated VLM. Thus, our contribution lies in evaluating visually grounded state progression and its integration with cross-view reasoning, rather than treating temporal ordering as an entirely separate capability. We will clarify this focus in the revised manuscript.
Follow-up on our rebuttal
Dear Reviewer pJah,
As the discussion phase is approaching its end, we would greatly appreciate it if you could let us know whether our clarifications and additional experiments have adequately addressed your remaining concerns.
We would be happy to address any further questions before the discussion period ends. Thank you again for your time and constructive engagement throughout the discussion!
Best regards,
All Authors
Follow-up on our rebuttal
Dear Reviewer pJah,
With the rebuttal deadline approaching, we wanted to briefly follow up on our response. If you have a chance to review it, we would appreciate a brief reply letting us know whether our clarifications and additional results address your main concerns or if anything remains unclear.
Thank you for your time and thoughtful feedback.
Best regards,
The Authors
Response to Reviewer pJah
W1: Future-State Prediction vs. Temporal Ordering
We thank the reviewer for highlighting this important distinction. To directly investigate whether temporal ordering provides an effective shortcut for EgoExo-Next, we conduct two additional experiments using Qwen3-VL-235B on all 1,899 questions from Subtasks 1, 3, and 4.
Accuracy (%). In/Out denote Indoor/Outdoor. Avg. denotes the mean across six settings.
Explicit Temporal Ordering achieves 40.37% accuracy, compared with 41.77% for direct next-state identification, with consistently lower performance across all six settings. Oracle Future Mask improves accuracy to 45.13%, indicating that identifying past and future candidates provides useful information, while selecting the earliest future state remains challenging.
These results demonstrate that explicitly applying temporal ordering does not improve next-state identification for the evaluated VLM. Even with ground-truth past/future information, the model continues to struggle with fine-grained temporal distinctions.