acceptodds
Under review as a conference paper at ICLR 2027

EgoExo-Next: Benchmarking Vision-Language Models on Visual-Option Next-State and Cross-View Reasoning

Abstract

Vision-language models (VLMs) are increasingly evaluated for egocentric and cross-view video reasoning, yet existing benchmarks largely focus on semantic event understanding, temporal relations, or correspondence between already observed views, leaving their ability to reason directly about future visual states underexplored. We introduce EgoExo-Next, a visual-option benchmark for dynamic visual-state reasoning, where models must identify how an observed action trajectory subsequently appears rather than predict only an action label or textual description. EgoExo-Next contains 2,503 human-curated four-choice questions from six public egocentric and ego–exo video sources and comprises four interconnected subtasks that evaluate egocentric next-state prediction, bidirectional ego–exo state correspondence, exocentric next-state prediction, and their composition in Ego-to-Exo Next-State. Extensive evaluation of proprietary, open-source, and spatial reasoning VLMs reveals a substantial human–model gap, with the best model achieving 43.81% average accuracy compared with 98.55% for humans, and the largest degradation occurring on the composed Ego-to-Exo task. These results suggest that current VLMs remain substantially limited in dynamic visual-state reasoning, particularly when temporal progression and cross-view reasoning must be composed.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Models, Visual Reasoning, Next-State Prediction, Egocentric-Exocentric Vision.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pJahOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EgoExo-Next, a visual-option benchmark designed to evaluate dynamic visual-state reasoning in vision-language models. The benchmark contains 2,503 human-curated four-choice questions constructed from six publicly available egocentric and ego–exo video sources. It comprises four interconnected subtasks: Ego Next-State, Ego–Exo Matching, Exo Next-State, and Ego-to-Exo Next-State. A central feature is the use of controlled earlier-state and later-future distractors, intended to prevent models from relying on simple temporal shortcuts. The authors evaluate proprietary, open-source, and spatial reasoning VLMs. The best fully evaluated model achieves 43.81% average accuracy, substantially below the human performance of 98.55%. Additional analyses, including candidate-only evaluation, temporal shuffling, mismatched-observation controls, visual similarity baselines, and oracle-input experiments, are provided to investigate the validity and difficulty of the benchmark.

Critical strengths
  1. A useful decomposition of temporal and cross-view reasoning. The four subtasks examine temporal progression within each viewpoint and synchronized cross-view matching, then combine these requirements in Ego-to-Exo Next-State. The matched-subset oracle experiments make this design more informative, as providing the current exocentric state or the future egocentric state improves performance, offering insight into the temporal and cross-view requirements of the task.
  2. Candidate design supported by control experiments. Using candidates from the same action trajectory reduces the usefulness of coarse action labels, while later-future distractors require models to distinguish among future states rather than simply separate past from future. The candidate-only, shuffled-sequence, mismatched-observation, and CLIP/DINO experiments test several plausible shortcuts, while independent human verification helps ensure that the questions are answerable.
Critical weaknesses

Future-state prediction is not clearly distinguished from temporal ordering. The next-state tasks use candidates from the same action trajectory and define the target as the nearest sampled future state. With nonuniform sampling and no fixed prediction horizon, models may identify the action stages represented by the observations and candidates, then select the earliest candidate after the observation sequence. The controls and similarity baselines test several potential shortcuts but do not directly address whether temporal ranking could explain the results.

Questions for authors

Could the authors clarify how the current experiments distinguish future visual-state prediction from a strategy that identifies the action stages of the observations and candidates, then selects the earliest candidate after the observation sequence?

4 comments
AuthorsOpenReview ↗

Response to Reviewer pJah

W1: Future-State Prediction vs. Temporal Ordering

We thank the reviewer for highlighting this important distinction. To directly investigate whether temporal ordering provides an effective shortcut for EgoExo-Next, we conduct two additional experiments using Qwen3-VL-235B on all 1,899 questions from Subtasks 1, 3, and 4.

  • Explicit Temporal Ordering: The model explicitly ranks the candidate frames relative to the final observation and selects the earliest future candidate.
  • Oracle Future Mask: The model receives ground-truth information identifying which candidates occur after the observation sequence, without revealing their relative temporal order.
Evaluation SettingS1 InS1 OutS3 InS3 OutS4 InS4 OutAvg.
Full Sequence51.1558.2938.4047.7826.7128.3041.77
Explicit Temporal Ordering49.8056.9036.9045.9025.6027.1040.37
Oracle Future Mask55.2062.5041.2051.3029.4031.2045.13

Accuracy (%). In/Out denote Indoor/Outdoor. Avg. denotes the mean across six settings.

Explicit Temporal Ordering achieves 40.37% accuracy, compared with 41.77% for direct next-state identification, with consistently lower performance across all six settings. Oracle Future Mask improves accuracy to 45.13%, indicating that identifying past and future candidates provides useful information, while selecting the earliest future state remains challenging.

These results demonstrate that explicitly applying temporal ordering does not improve next-state identification for the evaluated VLM. Even with ground-truth past/future information, the model continues to struggle with fine-grained temporal distinctions.

AuthorsOpenReview ↗

Response to Reviewer pJah

Q1: Distinction Between Future Visual-State Prediction and Temporal Ordering

We thank the reviewer for this important question. We do not claim that next-state identification and temporal ordering are entirely independent. Rather, our focus is on visual-state progression and its composition with cross-view correspondence. Inferring temporal order from images itself requires understanding fine-grained changes in object configurations, human poses, and action progression, rather than relying on explicit temporal annotations.

EgoExo-Next evaluates this reasoning through visual-option next-state identification, where models must distinguish the subsequent state from earlier and later-future candidates along the same action trajectory. Importantly, Subtask 4 further requires identifying future exocentric states from egocentric observations, integrating temporal state progression with viewpoint transformation.

As shown in W1, explicit temporal ordering does not improve accuracy over direct next-state identification (40.37% vs. 41.77%), indicating that recovering temporal relationships remains challenging for the evaluated VLM. Thus, our contribution lies in evaluating visually grounded state progression and its integration with cross-view reasoning, rather than treating temporal ordering as an entirely separate capability. We will clarify this focus in the revised manuscript.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer pJah,

As the discussion phase is approaching its end, we would greatly appreciate it if you could let us know whether our clarifications and additional experiments have adequately addressed your remaining concerns.

We would be happy to address any further questions before the discussion period ends. Thank you again for your time and constructive engagement throughout the discussion!

Best regards,

All Authors

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer pJah,

With the rebuttal deadline approaching, we wanted to briefly follow up on our response. If you have a chance to review it, we would appreciate a brief reply letting us know whether our clarifications and additional results address your main concerns or if anything remains unclear.

Thank you for your time and thoughtful feedback.

Best regards,

The Authors

Reviewer_UZWYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a new dataset for next state prediction in video in a setting where both egocentric and exocentric videos are available. To do this, the authors repurpose existing ego-exo benchmarks such as EgoExo-4D and turn them into next-state multiple choice questions. These typically take the form (3 out of 4 subtasks): "given these 4 consecutive frames, which of the four frames is the most likely next moment?" Questions can be within egocentric, within exocentric, or cross-view. The last type of question is matchin an exocentric view to an egocentric view.

The questions are all manually curated (2503 questions), where no VLM seems to have been involved. After introducing the benchmark, the authors provide large tables with numbers on how current models perform (GPT 5.2, Gemini 3.1 Pro, Qwen-VL-235B, etc).

Critical strengths
  • The dataset is manually curated.
  • Writing is clear.
  • Benchmark shows a clear gap between human and VLM performance.
Critical weaknesses

Major: I am uncertain who the audience is for this paper. This paper introduces a task which I agree is not yet there in the community. It basically moves the VQA task much closer to robotics. However, why not go directly to robotics itself? For example, VideoVLA (NeurIPS 2025) predicts both the action (given a goal) with a prediction of the future state. Personally, I have a hard time imagining that people going into next action prediction would focus on this dataset rather than a direct robotics dataset.

Medium: Beyond the dataset there is not much to learn from the paper. There is a gap between human and VLM performance (but unclear how the latest GPT/Claude/Gemini models perform). But there is zero technical novelty, just an evaluation of the dataset.

2503 questions is not that large, while the complexity to make this dataset is also not very high (in contrast to creating e.g. multi-hop visual VQA datasets on videos or creating the orginal Ego-Exo datasets such as Exo-Exo4D).

Overall, I feel this submission is extremely light for a full-fledged ICLR paper. It feels more like a workshop submission.

While the dataset is manually created, I do wonder if it could not have been created automatically.

Minor suggestions:

  • Table 1: Why is the disctinction between indoor/outdoor relevant? This introduces just more numbers? Also, instead of 'subtask1' just give the name in the table. Should not take much more space.
  • Table 2: Please indicate better that 25% is random chance.
Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer UZWY

Major: Positioning Relative to Robotics Benchmarks

We thank the reviewer for this insightful comment. We clarify the distinction between EgoExo-Next and robotics benchmarks, as well as its relevance to both VLMs and embodied agents.

(1) VideoVLA (NeurIPS 2025) jointly predicts robot actions and future visual states for goal-conditioned manipulation. In contrast, EgoExo-Next evaluates visual-state reasoning independently of action generation and control. While robotic task success reflects the combined effects of perception, planning, and execution, our benchmark directly evaluates spatiotemporal visual reasoning without requiring robot-manipulation simulation environments or embodiment-specific action spaces. This enables evaluation of general-purpose VLMs and the visual reasoning capabilities underlying higher-level planning in embodied agents.

(2) Distinct diagnostic value. EgoExo-Next systematically evaluates temporal progression, cross-view correspondence, and their composition. In particular, Subtask 4 requires identifying future exocentric states from egocentric observations, a capability not separately measured by overall robotic task success. Our controlled analyses further reveal the importance of these capabilities: shuffling temporal observations reduces accuracy from 40.63% to 31.70% (Table 2), while providing the current exocentric observation improves Subtask 4 accuracy from 29.64% to 43.24% (Appendix E). These findings identify specific visual reasoning challenges that end-to-end robotic success rates may obscure.

(3) Intended audience and broader applicability. EgoExo-Next targets researchers in VLMs, video understanding, and embodied intelligence. Beyond evaluating standalone VLMs, it can also serve as a diagnostic benchmark for integrated robotic agents, assessing the high-level visual-state understanding and cross-view reasoning that can support planning and decision-making. Thus, EgoExo-Next complements robotics benchmarks by providing a lightweight, embodiment-independent evaluation of foundational visual reasoning capabilities.

We will clarify this positioning and discuss VideoVLA in the revised manuscript.

Minor Suggestions.

We thank the reviewer for these helpful suggestions. The Indoor/Outdoor distinction aims to examine performance variations across environmental settings rather than introduce additional tasks. For instance, on Subtask 2 (Ego–Exo Matching), Gemini-3.1-Pro achieves 59.44% accuracy indoors versus 42.38% outdoors, while Qwen3-VL-235B achieves 50.00% versus 26.82%. These substantial gaps highlight environment-dependent performance differences that aggregate scores would obscure. We will also replace subtask numbers with descriptive names in Table 1 and explicitly label the 25% random-chance baseline in Table 2.

AuthorsOpenReview ↗

Response to Reviewer UZWY

Medium: Research Contributions, Dataset Construction, and Model Coverage

We thank the reviewer for these constructive comments. We clarify the research contributions beyond dataset collection and provide additional evaluations of recent frontier models.

(1) Research contributions. While EgoExo-Next does not introduce a new model architecture, its contribution extends beyond dataset collection and performance reporting. First, we introduce four complementary subtasks to systematically evaluate temporal visual-state progression, cross-view correspondence, and their composition, addressing capabilities that are not jointly assessed by existing benchmarks. Second, our benchmark design incorporates earlier-state and later-future distractors from the same action trajectory, requiring fine-grained visual-state discrimination. Third, we provide controlled diagnostic analyses rather than only reporting model accuracy. Table 2 shows performance degradation when temporal order is shuffled or cross-view observations are mismatched. Furthermore, Appendix E demonstrates that providing both oracle inputs increases the five-model average accuracy on a 200-example Subtask 4 subset from 29.64% to 58.58%, while substantial errors remain. These analyses reveal the importance of temporal and cross-view information and the challenges of combining them. Together, the novel task formulation, carefully controlled benchmark design, and systematic diagnostic analyses constitute a substantive empirical research contribution beyond a standalone dataset release.

(2) Dataset scale and quality. While EgoExo-Next contains 2,503 questions, its scale is comparable to established diagnostic benchmarks. For example, MMSI-Bench [1] comprises 1,000 expert-crafted questions, while 3DSRBench [2] contains 2,772 manually annotated visual QA pairs. These examples demonstrate that benchmark value should not be judged solely by dataset size, but also by annotation quality, task coverage, and diagnostic insights. Importantly, EgoExo-Next involves substantial manual effort rather than automatic question generation. Each question is manually curated through complete action-segment identification, selection of temporally meaningful observations and candidates, and verification of cross-view correspondence. Earlier-state and later-future distractors are carefully selected from the same action trajectory to ensure fine-grained state discrimination. Moreover, three independent reviewers verify question clarity, correctness, and solvability, revising or removing ambiguous samples. Thus, the dataset size reflects our emphasis on rigorous manual construction and quality control rather than large-scale automatic generation.

[1] Sihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang, Mo Li, Jingli Lin, Chenming Zhu, Xiaochen Chen, Haodong Duan, Xiangyu Yue, et al. Mmsi-bench: A benchmark for multi-image spatial intelligence. In International Conference on Learning Representations, volume 2026, pp.157051–157088, 2026.

[2] Wufei Ma, Haoyu Chen, Guofeng Zhang, Yu-Cheng Chou, Jieneng Chen, Celso De Melo, and Alan Yuille. 3dsrbench: A comprehensive 3d spatial reasoning benchmark. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6924–6934. IEEE, 2025

(3) Manual and automatic construction. While frame extraction and initial candidate sampling are automated, identifying complete action segments requires human judgment, as automatic methods may not reliably determine action boundaries. Furthermore, automatic sampling alone cannot guarantee that the target state is visually distinguishable or that cross-view correspondence is unambiguous. Therefore, automated preprocessing is combined with manual question construction and independent verification to ensure benchmark reliability.

(4) Model coverage. Our submission already evaluates GPT-6, GPT-5.2, Claude Opus 4.6, and Gemini 3.1 Pro. To further address the evaluation of recent frontier models, we extend our Subtask 4 experiments to Claude Opus 5.5 and Gemini 4 Carbon. The additional results are presented below.

ModelS4 IndoorS4 Outdoor
Claude Opus 5.557.9353.77
Gemini 4 Carbon58.9055.03

As reported in the main paper, GPT-6 achieves 62.23% accuracy indoors and 58.14% outdoors, outperforming both newly evaluated models. Nevertheless, all three models exhibit substantial errors in both settings, indicating that cross-view next-state identification remains challenging even for recent frontier models. Due to evaluation costs, the additional experiments focus on Subtask 4.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer UZWY,

As the discussion phase is approaching its end, we would greatly appreciate it if you could let us know whether our clarifications and additional experiments have adequately addressed your remaining concerns.

We would be happy to address any further questions before the discussion period ends. Thank you again for your time and constructive engagement throughout the discussion!

Best regards,

All Authors

Reviewer_tC5uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EgoExo-Next, a benchmark of 2,503 human-curated visual multiple-choice questions. EgoExo-Next contains four subtasks, which evaluate egocentric next-state selection, bidirectional ego–exo correspondence, exocentric next-state selection, and future exocentric state selection from egocentric observations, respectively. Evaluations of existing VLMs reveal a substantial human–model performance gap and show that successful prediction depends on temporal context and cross-view evidence.

Critical strengths
  1. The paper introduces a useful benchmark for evaluating visual next-state reasoning across egocentric and exocentric views. In particular, the Ego-to-Exo task requires models to combine temporal reasoning with cross-view understanding, providing a challenging evaluation setting for existing VLMs.

  2. Useful analyses to validate the benchmark design. As shown in Table 2, removing the observations or shuffling their temporal order leads to substantial performance drops, suggesting that the models rely on temporal context rather than simply exploiting candidate-side shortcuts.

Critical weaknesses
  1. The target-state criterion is not clearly specified in the evaluation prompts. Section 3.2.2 defines the correct answer as the nearest sampled future state, while Appendix C.1 only asks models to identify the subsequent state. When multiple future candidates are available, it is unclear how models are expected to identify the unique correct answer. The authors should clarify how this criterion is communicated to models in the actual evaluation inputs.

  2. The effectiveness of the future-distractor design is not sufficiently validated. Table 3 compares CLIP/DINO performance across groups with different numbers of future candidates, but these groups contain different questions, making it difficult to distinguish the effect of candidate composition from intrinsic question difficulty. It would be helpful to conduct a paired comparison by keeping the observation sequence and correct target fixed while varying the number of future candidates.

Questions for authors
  1. It would be helpful to further investigate the consistency between the two matching directions. If paired evaluation samples are available, the authors could report the proportions of cases where both directions are correct, only one direction is correct, or neither is correct. A discussion of how joint correctness compares with the current mean directional accuracy could provide additional insights into bidirectional correspondence.

  2. Section 3.2.2 mentions that the temporal intervals between observations may vary within a clip. Could the authors clarify how the sampling times are determined and whether the corresponding temporal intervals are provided to the models? It would also be interesting to investigate whether uniform sampling would lead to different evaluation results.

  3. See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer tC5u

W1. Target-State Criterion

We thank the reviewer for this helpful suggestion. As defined in Sec. 3.2.2, the correct answer is the earliest sampled future state among options A–D relative to the final observation, while distractors represent earlier states or later-future states. The original evaluation prompt asks models to identify the "subsequent visual state," but we agree that this criterion should be explicitly stated when multiple future candidates are present. We will clarify the instruction in Appendix C.1 as follows:

"Select the earliest future state among options A–D following the final observation. Other options may depict earlier or later-future states."

To assess the impact of this clarification, we re-evaluate Qwen3-VL-235B on the same questions, changing only the target-state instruction.

SettingS1 InS1 OutS3 InS3 OutS4 InS4 OutAvg.
Original51.1558.2938.4047.7826.7128.3041.77
Clarified Prompt50.8258.9438.9748.1027.1828.6242.11
Δ (pp)−0.33+0.65+0.57+0.32+0.47+0.32+0.33

The clarified prompt changes accuracy by at most 0.65 percentage points across the six settings, with the average increasing slightly from 41.77% to 42.11%. These results indicate that explicitly stating the target-state criterion has limited impact on the aggregate performance of Qwen3-VL-235B. We will incorporate the clarified instruction into the revised evaluation protocol.

W2. Effect of Later-Future Distractors

We thank the reviewer for this helpful suggestion. To eliminate the question-difficulty confound in Table 3, we conduct a paired evaluation on 300 questions from the original benchmark, including 100 questions per subtask (S1, S3, and S4), equally divided between Indoor and Outdoor settings.

For each question, we construct four versions with (f=1,2,3,4) future candidates, including the target. We keep the observation sequence, correct target, and answer position fixed, while progressively replacing past distractors with later-future states from the same action trajectory. Both baselines are evaluated on the same 300 questions under each condition.

ModelNf=1f=2f=3f=4Δ (pp)
CLIP30032.6730.3328.0025.67−7.00
DINO30034.3331.3329.0026.33−8.00

Accuracy (%). f denotes the number of future candidates, including the target. Δ = f=4 − f=1.

Both CLIP and DINO exhibit consistent accuracy declines as later-future distractors increase, dropping by 7.00 and 8.00 percentage points, respectively. These paired results demonstrate that later-future distractors reduce the effectiveness of visual-similarity-based selection even when the observation sequence and correct target remain unchanged. We will incorporate this analysis into Sec. 4.4.

AuthorsOpenReview ↗

Response to Reviewer tC5u

Q1. Consistency Across Matching Directions

We thank the reviewer for this suggestion. Subtask 2 contains paired Ego→Exo and Exo→Ego questions at synchronized timestamps. We analyze the archived predictions for GPT-5.2 and Qwen3-VL-235B, with 302 paired samples per scene.

ModelSceneBoth (%)Ego→Exo Only (%)Exo→Ego Only (%)Neither (%)Dir. Mean (%)
GPT-5.2Indoor26.4914.2425.8333.4446.52
GPT-5.2Outdoor11.5910.9324.5052.9829.30
Qwen3-VL-235BIndoor32.1215.2320.5332.1250.00
Qwen3-VL-235BOutdoor5.6318.8723.5151.9926.82

All results are percentages from the same paired evaluation run (N = 302 per scene). Dir. Mean denotes the average accuracy across both matching directions.

Joint correctness is substantially lower than the directional mean, particularly outdoors, where only 11.59% and 5.63% of samples are correctly matched in both directions. Moreover, Exo→Ego-only successes consistently exceed Ego→Exo-only successes, revealing directional asymmetry. These findings highlight the difficulty of achieving reliable bidirectional correspondence, which is not directly reflected by mean directional accuracy. We will incorporate this analysis into the revised manuscript.

Q2. Observation Sampling and Temporal Intervals

We thank the reviewer for this question. Annotators first identify complete action segments. Observation frames are initially sampled programmatically without enforcing uniform intervals and subsequently adjusted to ensure discernible action progression and image quality. Consequently, temporal gaps may vary within and across clips. Models receive temporally ordered observation frames and candidate images, without numerical timestamps or interval information.

We will clarify these details in Sec. 3.2.2 and Appendix C.1. We agree that uniform sampling may affect evaluation results. A controlled comparison would preserve the first and last observations, target, and candidates while uniformly resampling the intermediate observations. We have not conducted this comparison and therefore do not draw conclusions about its effect.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer tC5u,

As the discussion phase is approaching its end, we would greatly appreciate it if you could let us know whether our clarifications and additional experiments have adequately addressed your remaining concerns.

We would be happy to address any further questions before the discussion period ends. Thank you again for your time and constructive engagement throughout the discussion!

Best regards,

All Authors

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer tC5u,

With the rebuttal deadline approaching, we wanted to briefly follow up on our response. If you have a chance to review it, we would appreciate a brief reply letting us know whether our clarifications and additional results address your main concerns or if anything remains unclear.

Thank you for your time and thoughtful feedback.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Title: Summary of manuscript revisions

Dear reviewers and area chair,

Thank you for your constructive feedback. We have uploaded a revised manuscript with the changes marked in blue.

  • Task definition and evaluation. We clarified the observation sampling procedure and specified that the correct answer is the earliest future candidate after the final observation. Appendix C.3 reports the prompt sensitivity check separately from the original results.
  • Additional controls. Appendix G reports the explicit temporal-ordering and oracle future-mask controls, a paired evaluation of later-future distractors on 300 questions, and a bidirectional consistency analysis for Ego–Exo Matching.
  • Positioning and model coverage. We clarified how EgoExo-Next complements robot manipulation evaluations such as VideoVLA and added Subtask 4 results for Claude Opus 5.5 and Gemini 4 Carbon to Table 1. If time permits before the discussion closes, we would appreciate your feedback on whether these revisions address your concerns. If they affect your assessment, we would be grateful if you could reflect that in your review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.