acceptodds
Under review as a conference paper at ICLR 2027

EMBER-Bench: Benchmarking Cross-Event Causal Memory in Long-Horizon Embodied Tasks

Abstract

Lifelong physical agents must reason over extended interactions where past events continue to shape the world long after they disappear from view. Beyond recalling what happened, agents must infer how history changes the current state and constrains future actions. Yet existing embodied and video-memory benchmarks largely focus on historical retrieval and summary, leaving such history-dependent causal reasoning underexplored. We introduce **EMBER-Bench**, an egocentric benchmark for cross-event causal reasoning in long-horizon embodied tasks, for which we newly created the task design, video recording, and data annotation. It contains 189 household tasks and 699 QA pairs, spanning task progress, failure recovery, external interventions, and compound long-horizon tasks with distant dependencies and prerequisites, with fine-grained event and causal-chain annotations. EMBER-Bench evaluates reasoning in both directions: next-action prediction selects the next action from history, and causal traceback, given that action, identifies the historical event that makes it necessary. Input ablations that add action logs or privileged cause-and-consequence annotations to the video indicate which kind of historical information models fail to use. Among the 16 evaluated models, the highest overall accuracy is 61.2%, compared with a mean of 98.3% across two human evaluators. At paired decision points, correct traceback is not associated with correct next-action prediction. Adding action logs yields a gain of 1.6 points, whereas cause-and-consequence annotations yield an additional gain of 13.0 points on top of that. These results suggest that extracting causal information from past events and converting it into constraints on current actions remains a key difficulty for long-horizon embodied agents.

Primary area:
datasets and benchmarks
Keywords:
embodied AI, egocentric memory, causal reasoning, long-horizon reasoning, vision-language models, benchmarks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8BQGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EMBER-Bench, which introduces 189 recorded household tasks and 699 multiple-choice questions targeting history-dependent embodied reasoning. It covers task progress, self-induced anomalies, external interventions, and compound tasks, pairing next-action prediction with causal traceback at 151 decision points. Across 16 models, the best overall accuracy is 61.2%, compared with 98.3% for human evaluators. Additional experiments examine historical information, temporal dependencies, and causal prompting, finding larger gains from privileged cause-and-consequence annotations than from action logs.

Critical strengths
  1. Focused and useful benchmark design. The emphasis on persistent consequences of anomalies and interventions provides a concrete setting for studying how historical events constrain subsequent actions. Structured event annotations support interpretable analysis.

  2. Meaningful validation of history dependence. The current-frame-only control achieves approximately chance accuracy, while human evaluators perform near ceiling with history. Together with input-boundary checks and manual review, these results support the benchmark’s answerability and the relevance of historical information.

  3. Substantial diagnostic evaluation. The paper evaluates 16 models and goes beyond a leaderboard through paired questions, information ablations, and temporal analyses. Task-clustered uncertainty estimates and explicit caveats about privileged annotations and action-cued traceback are welcome.

Critical weaknesses
  1. The evaluation does not fully isolate long-horizon memory requirements. Each question provides the complete history in a stateless call, primarily testing full-context video reasoning. Moreover, the shortest event-to-decision interval tertile spans only 0–7.97 seconds. Current-frame failure establishes that some history is needed, but not that distant-event retention or persistent state tracking is necessary. Recent-window and selective-history controls would strengthen this distinction.

  2. The prediction–traceback mismatch has limited mechanistic interpretation. Traceback supplies the correct next action and explicitly marks candidate historical clips, whereas prediction requires selecting the action without these cues. Consequently, correct traceback may reflect action–event matching rather than independently recovered causal knowledge. The paper acknowledges this limitation, but the mismatch alone cannot establish a failure to convert retrieved causes into action constraints.

  3. The annotation ablation does not cleanly separate causal reasoning from missing state information. In Table 10, the action log states that an arm hits an apple, while the consequence annotation adds that the apple rolled onto the ground. This supplies an additional physical outcome, not merely a causal organization of otherwise identical information. The improvement therefore supports the usefulness of explicit consequences, but does not isolate causal extraction as the main bottleneck.

  4. The causal-prompting result needs a stronger control. Causal CoT is compared with direct answering, without a generic reasoning prompt matched for output length or inference budget. Only one of three models improves significantly. Thus, the evidence does not yet distinguish a benefit specific to causal guidance from a general benefit of additional reasoning, and repeated runs would help assess robustness to the reported API nondeterminism.

Questions for authors
  1. How does performance change with only a recent video window, only the relevant event, or the history with that event removed? What fraction of questions demonstrably requires distant evidence?

  2. Can models first identify relevant events without receiving the reference action, then predict the next action using their own retrieved evidence? Does this produce the same prediction–traceback mismatch?

  3. Would equally detailed state descriptions, without explicit causal labels, provide comparable gains? How well do action logs and consequence annotations perform without video?

4/ Does causal CoT outperform generic CoT under matched inference budgets and repeated runs? Are its intermediate consequence assessments systematically accurate?

4 comments
AuthorsOpenReview ↗

[Author Response Section 1/4]

We thank the reviewer for the detailed and constructive review. Our motivation is that past events keep constraining a physical agent's actions after they leave view. EMBER-Bench therefore tests whether models use such history to select the next action, not only whether they can retrieve it. Following your suggestions, we ran new controls on the 151 paired questions with five models: Qwen3.8 Omni Flash (Omni), Qwen3.8 Flash (Flash), Gemini 3.8 Flash (Gemini), GPT-6 Sol (GPT) and Qwen3.5 35B-A3B (Q3.5). Differences are in percentage points (pp). Because the API versions and prompts differ from the submission and between control groups, absolute accuracies differ by a few points from the paper and across tables; comparisons should be read within each table.

Each question supplies the full history in one stateless call, the shortest event-to-decision tertile spans only 0–7.97 s, and current-frame failure shows that some history is needed, not that distant events must be retained (W1). How does accuracy change with only a recent window, only the relevant event, or the event removed, and what fraction of questions requires distant evidence (Q1)?

Thanks a lot for this suggestion. Experiment setup: Recent window: only the last 8 s or 30 s of the history, taken from the same frame sequence as the full history; goal, current frame, question and options unchanged. Only the earlier history is removed. Text / Event window: the history as text (action log plus neutral state descriptions, L+S), keeping only events that start and end inside the window.

Table I. Experiment Results with different lengths of history window

HistoryWindowOmniFlashGPTQ3.5
Videofull38.434.430.523.8
Videolast 30 s30.521.223.219.9
Videolast 8 s28.524.519.920.5
Textfull5554.368.953.6
Textlast 30 s48.344.45142.4
Textlast 8 s36.436.437.135.8

How many items depend on evidence outside the window. By annotation, the key event ends more than 8 s before the decision in about two thirds of the 151 pairs, and more than 30 s before it in about one third (Appendix E.3, Table 8). The windows therefore remove the key event from a large share of items, and from more items at 8 s than at 30 s.

When the history is given as text, models can actually use it (53.6–68.9% with full history). Keeping only the last 30 s lowers accuracy by 6.7–17.9 pp (mean 11.4). Keeping only the last 8 s lowers it by 17.8–31.8 pp (mean 21.5), to 35.8–37.1%, significant in every model. The drop roughly doubles from the 30 s to the 8 s window, as the share of items whose key event falls outside the window grows. This graded pattern is what we expect if answers depend on earlier events, and not if they depend on the most recent frames.

Video history. All eight model–window cells drop: by 3.3–10.6 pp at 8 s and 3.9–13.2 pp at 30 s (means 8.4 and 8.1 pp), to 19.9–30.5%, around chance (25%). Distractors are built as the actions a model would choose if it ignored or misread the earlier event (Sec. 3.3), and a recent window alone leads models to them. Because full-history video accuracy is already low (23.8–38.4%), there is little room for a graded difference between the two windows, and Q3.5, near chance even with full history, changes least. We therefore treat the text condition as the more sensitive measure of what the items require. The video condition shows that models also draw on earlier history when it is presented as video.

This control asks a different question from the windows: not whether the item needs earlier evidence, but whether a model’s answer currently depends on that specific clip. Given only the gold clip and the current frame, the prediction is 25.2–37.7%, which does not exceed full-history prediction (29.8–37.1%; Table II in our response to W2). This matches our responses to W2 and W3: handing models the gold clip does not improve their decisions, whereas stating its consequence in text raises accuracy by 15.5–16.0 pp in every model. Together, the controls separate two findings:

  1. The items require distant evidence (text windows above; 26.5% with the current frame only, Appendix D).
  2. Current models draw on earlier history in general but do not use the specific event whose consequence decides the answer unless that consequence is stated.

The second finding is the gap EMBER-Bench is designed to measure.

In summary, we agree that each question is answered in one stateless call with the full history, and this is deliberate. Full access is the most favorable condition for retention, so the failures we report concern the use of past events, not their storage. We will state this scope explicitly in Sec. 3 and the Limitations, and add Table I together with all controls above to the Appendix. Thank you again for this suggestion, which has made the distinction considerably sharper.

AuthorsOpenReview ↗

[Author Response Section 2/4]

Traceback supplies the correct next action and marks candidate clips, so a correct traceback may reflect action–event matching, and the mismatch alone cannot establish a failure to convert retrieved causes into action constraints (W2). Can models first identify the event without the action, then predict from their own evidence (Q2)?

Thank you for your valuable suggestion. The ablation settings are shown below:

Action-free traceback: same marked history and current frame; we delete the reference action and only the wording that depends on it. Two-stage: the clip the model selected without the action, with markers removed, is given in a new call together with the current frame and the original goal, question and options. Gold clip: the same, with the gold clip. Association: traceback correctness vs. independent full-history prediction correctness, adjusted for item and model.

Table II

ConditionOmniFlashGeminiGPTQ3.5
C, with action (original)44.441.753.063.641.1
C, without action28.535.155.052.333.1
P, full history35.133.137.137.129.8
P, self-selected clip27.227.836.437.126.5
P, gold clip31.825.237.731.827.8
TracebackP acc., C correctP acc., C wrongAdjusted Δ (pp)
With action34.0% (125/368)34.9% (135/387)−2.5
Without action33.8% (104/308)34.9% (156/447)−5.0

1.First, the mismatch does not depend on the cue. With action-free traceback, P accuracy is 33.8% when C is correct and 34.9% when it is not (adjusted −5.0 pp, not significant), in line with the action-cued condition (−2.5) and the paper's 16-model estimate (−3.5). Four of five models show no positive association; GPT shows a positive direction (39.6% vs 32.7%) but still pairs about 60% of its correct tracebacks with a wrong prediction.

2.Locating the event, or being handed it, does not improve the decision. Predictions from the self-selected clip (26.5–37.1%) or the gold clip (25.2–37.7%) do not exceed full-history prediction (29.8–37.1%); gold minus self-selected is −0.1 pp. Because a clip alone drops the surrounding history, we also highlighted the gold clip inside the full history, which adds only 1.2 pp (not significant).

3.Stating the event's consequence does improve it: describing the consequence in text raises prediction by 15.5–16.0 pp on average, significantly in every model (Response 3/4).

4.The action does help traceback itself. Removing it lowers traceback by a significant 7.9 pp on average, mainly in L3 (18.3 pp) and L4 (13.3 pp), where several candidates are anomalous. Gemini and GPT still find the event in 55.0% and 52.3% of items without the action.

As your notes and as Sec. 4.3 cautions, the correlation alone is not direct evidence. Results 2 and 3 add interventional evidence: giving the model the right event leaves its action unchanged, while giving it the event's consequence changes it. Thus, even when the correct past event is located or directly provided, models often fail to exploit its consequence when choosing the action, unless that consequence is stated. In the revision, we will present these results together with the cue effect and state the finding in these terms. Thank you again for your constructive suggestion!

AuthorsOpenReview ↗

[Author Response Section 3/4]

In Table 10 the consequence annotation adds a new physical outcome (the apple rolls onto the floor), so the gain supports explicit consequences but does not isolate causal extraction (W3). Would equally detailed state descriptions without causal labels give comparable gains, and how do logs and annotations perform without video (Q3)?

Thanks a lot for this suggestion. Here's ablation settings for your concern. S: a neutral rewrite of E that states the same events and outcomes as plain observations ("Observation 1: the arm hits the clean apple. Observation 2: the apple rolls onto the floor."), with no cause/consequence fields or causal connectives. S matches E in length (5,202 vs 5,225 words; every item within ±10%) and was screened for answer-revealing wording. S is placed inline with the log, or in a block at E's position. Renamed fields: E verbatim, with cause/consequence renamed to field_1/field_2. S2: an end-state sentence for every completed action, not only the relevant one. Text only: L, L+S, L+E.

Table III.

ComparisonOmniFlashGeminiGPTQ3.5Mean
V+L − V (log)4.666.614.60.76.5
V+L+S − V+L12.612.621.219.913.916
V+L+E − V+L14.611.317.915.917.915.5
E − renamed fields0.7−2.0n/a−2.70.7none significant
Acc. %, L (no video)33.837.74538.436.4
Acc. %, L+S (no video)5758.374.869.557.6
Acc. %, L+E (no video)54.35172.266.255.6

From results in Table III, we can see that E indeed adds outcome information beyond the log, and this is the information Contribution 2 refers to: what a past event changed. The controls show that this content, not the causal labelling, drives the gain: S gives +16.0 pp and E +15.5 pp over V+L, each significant in all five models. E vs renamed the fields changes accuracy by −2.7 to +0.7 pp; neither is significant.

More state information does not explain the gain by itself. S2 gives an end state for every completed action, including the relevant one, and so contains more state information than S, yet it helps less (+10.2 pp), a significant 5.8 pp below S. What helps most is knowing which past event matters and what it changed, the distinction EMBER-Bench is built around.

Without video, we can see that the log alone gives 33.8–45.0%, while L+S and L+E give 57.0–74.8% and 51.0–72.2%. Adding the video to L+E does not help (−5.0 pp). Humans reach 97.8% on the same video input, so the information is in the video, yet models use it only when it is stated in text.

On top of the log, consequences add 11.3–17.9 pp in every model, while the log itself adds 0.7–6.6 pp (not significant) in four models; GPT is the exception (+14.6 pp, significant). About three quarters of this gain comes from L2 (+29.3 pp), where the self-induced anomaly itself appears as a log entry: once the triggering event is written down, GPT infers its consequence and acts on it.

We will make precise that by causal information we mean what a decision-relevant past event changed and whether that change still holds, not causal labelling. Concretely, we will (a) describe E as explicit descriptions of past events and their persisting consequences; (b) add S, the renaming control and S2, and state that causal labels add no measurable gain; (c) clarify that "causal extraction" means extracting and using the persisting consequences of past events; and (d) report log gains per model. With this precision Contribution 2 stands: what changes decisions is the consequence of the relevant past event, not the action history. Thank you again for your constructive suggestion!

AuthorsOpenReview ↗

[Author Response Section 4/4]

Causal CoT is compared only with direct answering, without a generic reasoning prompt matched in output length or inference budget; only one of three models improves significantly, and repeated runs are needed (W4). Does causal CoT outperform generic CoT under matched budgets and repeated runs, and are its intermediate consequence assessments accurate (Q4)?

Thanks for pointing this out. Here's the setup. Direct answering, generic CoT and causal CoT were submitted interleaved in the same batch, in two runs, on 50 prediction questions (17,17,16 with L2/L3/L4 respectively) within 151 questions for better ablation, testing five models. Both CoT variants use the same two-line format (a short assessment, then the answer) and the same cap of two sentences or 60 words. The generic prompt asks only for a brief justification; the causal prompt is the paper's Prompt B.5. Visual input, reasoning setting and output limit are fixed within each model. Since the Qwen3.8-27B API we applied is unstable recently, we changed to different qwen models to elaborate.

Table IV:

ModelDirectGeneric CoTCausal CoTCausal − generic (pp)
Omni353833−5.0
Flash3341432
Gemini5047558
GPT4543452
Q3.52931376
Mean38.44042.62.6

From Table IV, we can compare the performance of causal CoT and generic CoT. Under a matched format and budget, causal CoT is higher than generic CoT in four of five models, by 2.0–8.0 pp (largest in Gemini, +8.0, and Q3.5, +6.0), and lower in Omni (−5.0).

Generic CoT changes accuracy by −3.0 to +8.0 pp relative to direct answering (mean +1.6) and lowers it in Gemini and GPT; causal CoT adds 4.2 pp on average. In Gemini and Q3.5, the two models with the largest causal advantage, generic CoT adds little or nothing (−3.0 and +2.0), so their advantage is not explained by extra reasoning. Flash is the case you anticipated: a generic justification already gives most of its gain (+8.0 of +10.0 over direct).

Model capability. The gain does not track model strength: causal − direct is +5.0 for Gemini and 0.0 for GPT, against +10.0 for Flash and +8.0 for Q3.5. We will remove the capability-dependent reading in Sec. 4.4.

We will restate the CoT finding as: causal CoT gives a small advantage over a matched generic CoT in most models (mean +2.6 pp) and we will report the generic control, both runs and per-model results in the paper. This was a secondary observation, and the main findings are unchanged.


Appreciate it for your patience and effort. Thank you once again for your valuable reviews. We are looking forward to hearing from you.


Reviewer_Ddn8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary
  • this paper introduces EMBER-Bench, an egocentric benchmark comprising 699 QA pairs for cross-event causal reasoning.
  • especially this paper aims to explore the use of memory in causal reasoning task, the questions are constructed to evaluate the use of 4 types of memories such as: Task Progress Memory, Self-induced Anomaly Memory, External Intervention Memory, Compound Long-Horizon Memory.
  • Their benchmark is evaluated on several recent and popular video llms. their analysis suggests that extracting causal information from past events and converting it into constraints on current actions remains a key difficulty for long-horizon embodied tasks.
Critical strengths
  • this paper attempts to tackle an important challenge in the field: benchmarking video llms in long horizon embodied tasks
  • this paper evaluates a series of popular video llms on their benchmark and performs related analysis
  • this paper attempts to create new videos for their benchmark -- which is a good idea to ensure the benchmark samples are not contaminated with the training data of the existing models.
Critical weaknesses
  • the readability of this paper is very poor. it is very hard to follow the paper's writing.
  • i have read the entire main paper and did not get a clear idea of what their task looks like, especially how it is different from the existing literature!
  • the idea of L1-L4 memory is not clear, i do not see a clear definition with some intuitive examples beyond their introduction in l.096. it would be of interest to clearly explain how does different memory requirements differ from each other, and how different memory plays a role in certain question types.
  • the benchmark is extremely small and lacks diversity, it combines 699 QA pairs belonging to only 189 videos.
  • based on figure 3 it seems that the majority of the videos are fairly short (0-120 seconds), the notion of causality in long video that this paper claims to address is missing.
  • their paper seems to tackle egocentric video understanding tasks -- their claim for a benchmark of long horizon embodied task is somewhat misleading.
  • this paper uses some markers like L1-L4 or P, C, E -- which is very difficult to keep track of since they are not very common terms. sometime they are not even properly defined, e.g., P and C are defined for the first time in table 2 caption l.378, but their usage is identified at l.351.
  • l.367. what is this priviledge information referring to?
  • table 3: should not the net gain be negative for Qwen3.8 27B?
  • l.161 the authors mentioned that the task videos were originally recorded by them. it is not clear what the setup was for those recordings. and why can't we use existing videos and annotate their causal events for the creation of this benchmark?
Questions for authors

i summarized them under Critical Weaknesses.

6 comments
AuthorsOpenReview ↗

[Author Response Section 1/2]

We thank the reviewer for the time spent on our submission, and for recognizing that the problem is important and that newly recorded videos avoid contamination from training data. Several comments concern the scope of the work, so we first restate what the paper contributes. We then list the points we will have revised (below) and the points that need clarification (Section 2/2).

Hierarchical VLAs such as π0.5 (Physical Intelligence et al., 2025) now carry out multi-stage household tasks. In such tasks the right next step often depends on an event that is no longer visible: something was spilled, or another person took an item. Can a model use the history to determine which effects of past events still hold, and choose its next action accordingly (l.038–045)?

Video-memory benchmarks test whether a model can retrieve or summarize what happened. Embodied-memory benchmarks test task progress and, in a few cases, the agent's own mistakes. None of the 18 benchmarks in Table 1 explicitly evaluates changes made by other people (L3), or their combination with the agent's own mishaps within one long task (L4), and pairing action selection with causal attribution at the same decision point has been limited (l.085–088).

EMBER-Bench provides:

  • 189 household tasks that a 12-member team designed, recorded and annotated from scratch, with 699 questions and a cause–consequence annotation for every anomaly (Sec. 3).
  • A strict admission test: a question is kept only if the goal and current frame do not determine the answer. Without the history, 16 models average 26.5% on next-action questions (chance 25%); with it, human evaluators reach 98.3% (Sec. 3.4, Appendix D). Every question went through three rounds of model screening and evidence verification against the raw videos.
  • Paired questions at 151 decision points: which action now, and which past event made it necessary.

The best of 16 models reaches 61.2%, against 98.3% for humans. Adding a log of past actions to the video raises accuracy by 1.6 points; adding the consequences of past events raises it by a further 13.0 (Sec. 4.2). Current models fail to analyse what past events changed and to track whether those changes still constrain the next action; recording what was done is not enough. This points memory design for embodied agents toward tracking the lasting consequences of events rather than logging actions (Sec. 5).

Reviewer C8Ld notes that it "addresses an important gap in existing embodied memory benchmarks" and calls the paired evaluation "a useful design". Reviewer 8BQG points to a "meaningful validation of history dependence" and a "substantial diagnostic evaluation". Both rated the presentation "3: Clear and easy to follow".

We have nevertheless addressed each specific point raised in this review in the revised paper (which will be updated later), and will keep refining the wording for the final version:

  • Notation (W7). P and C are defined at first mention (Sec. 1, Fig. 1 caption), and a notation table opens Sec. 3.
  • L1–L4 (W3). The definitions (l.096–104) and examples (Fig. 1a) are expanded into one criterion with a definition and example per type (Sec. 1, Appendix H). All four types have next-action questions; traceback questions exist only for L2–L4, where one traceable anomalous event exists (l.258–261).
  • Recording setup (W10). We will explain it in Sec. 3.1 and appendix, the videos are camera phone hanged on the neck with 1920×1080 at 30 fps or higher; 12 camera wearers in 8 households; written task plans recorded continuously; L3 and L4 tasks recorded by two people; a quality-control pass on every recording.
  • Terminology and Table 3 (W8, W9). "Privileged information" is defined at first use, and Table 3 reports the overall and the subgroup net gain in separate columns.
AuthorsOpenReview ↗

[Author Response Section 2/2]

The remaining comments rest on a reading of the paper that we would like to clarify.

1. What the task looks like, and how it differs from prior work (W2). Fig. 1(b) shows a complete paired example: goal, history, current frame, four candidate actions, and the paired traceback question. The difference from prior work is stated in l.078–088 and Table 1: long-video QA asks what happened, and the answer is in the video; EMBER asks what to do next, when the answer depends on how a past event changed the task state.

2. Video length and "long-horizon" (W5). Fig. 3(b) does not show the length of our videos. As its caption says, it shows the "video input duration based on all QA pairs", i.e., the history before each decision point. More importantly, the paper does not claim long-video understanding. It states that task duration alone does not establish memory dependence (l.046), and that long-video understanding does not establish how history constrains an action (l.079–081); EgoSchema (Mangalam et al., 2023) makes the same point about clip length. "Long-horizon" is used in the embodied sense of tasks with many dependent steps, as in CALVIN (Mees et al., 2022) and π0.5: every task plan has more than 10 atomic actions, and more than 40 for L4 (l.194–195). The results support this design: when the key event ended less than 8 s before the decision, models still choose the correct action in only 29.2% of cases (Fig. 4b).

3. Embodied scope (W6). EMBER evaluates the high-level decision of an embodied agent, the choice of the next subtask, from first-person observations. The paper states that it does not test closed-loop execution (l.092–094, Sec. 5). This choice is the step π0.5 takes before it generates low-level actions, and the protocol follows established embodied-reasoning benchmarks: EgoPlan-Bench (Chen et al., 2026b) (first titled "Benchmarking Egocentric Embodied Planning"), RoboVQA (Sermanet et al., 2024), and ERQA (Gemini Robotics Team, 2025) from Gemini Robotics.

4. Size and diversity (W4). The size is a design choice stated in Sec. 3.5 (l.320–323): a 12-member team designed and recorded every task, and a question is kept only if its answer is unique and requires the history. The result is in the range of established benchmarks of this kind: ERQA (Gemini Robotics Team, 2025) has 400 questions and EgoPlan-Bench2 has 1,321. EMBER covers 8 functional areas and 12 activity categories (Fig. 3a). Its effects are well resolved: the gap between humans and the best model is 37.1 points, and the gain from consequence annotations has a 95% confidence interval that excludes zero for five of six models (Appendix Table 11).

5. Why new recordings (W10). The events EMBER needs, mishaps with lasting consequences and other people changing task objects, must be designed into a task so that the current frame hides them. Benchmarks that annotate existing videos do not cover them (Table 1); EgoPlan-Bench2 (Qiu et al., 2026), built on Ego4D (Grauman et al., 2022), discards narrations of other people's actions during construction. Scripted new recordings are standard for diagnostic benchmarks such as the Perception Test (Pătrăucean et al., 2023), whose approach we adopt (l.185–186), and CaptainCook4D (Peddi et al., 2024).

6. Table 3 and "privileged information" (W9, W8). The last column of Table 3 is the net gain within the correct-traceback subgroup, as its caption and l.465–467 state. Overall, Qwen3.8 27B changes by −4.0 points, as the reviewer expects. "Privileged information" denotes the human-written cause-and-consequence annotations, which are used only in the diagnostic ablation (l.213–214, l.366–367); the term follows Vapnik & Vashist (2009).

We hope this clarifies the scope and contribution of the paper. The comments above concern scope and presentation; if there is a specific concern about the validity of the experiments or the analysis, we would be glad to address it.

References

Yi Chen, Yuying Ge, Yixiao Ge, Mingyu Ding, Bohao Li, Rui Wang, Ruifeng Xu, Ying Shan, and Xihui Liu. EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning. International Journal of Computer Vision, 134(3):118, 2026.

Gemini Robotics Team. Gemini Robotics: Bringing AI into the Physical World. arXiv preprint arXiv:2503.20020, 2025.

Kristen Grauman, Andrew Westbury, Eugene Byrne, et al. Ego4D: Around the World in 3,000 Hours of Egocentric Video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995–19012, 2022.

Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding. In Advances in Neural Information Processing Systems, volume 36, pp. 46212–46244, 2023.

Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks. IEEE Robotics and Automation Letters, 7(3):7327–7334, 2022.

Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, et al. Perception Test: A Diagnostic Benchmark for Multimodal Video Models. In Advances in Neural Information Processing Systems, volume 36, pp. 42748–42761, 2023.

Rohith Peddi, Shivvrat Arya, Bharath Challa, et al. CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities. In Advances in Neural Information Processing Systems, volume 37, pp. 135626–135679, 2024.

Physical Intelligence, Kevin Black, Noah Brown, et al. π0.5: a Vision-Language-Action Model with Open-World Generalization. arXiv preprint arXiv:2504.16054, 2025.

Lu Qiu, Yi Chen, Yuying Ge, Yixiao Ge, Ying Shan, and Xihui Liu. EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarios. International Journal of Computer Vision, 134(5):222, 2026.

Pierre Sermanet, Tianli Ding, Jeffrey Zhao, et al. RoboVQA: Multimodal Long-Horizon Reasoning for Robotics. In IEEE International Conference on Robotics and Automation, pp. 645–652, 2024.

Vladimir Vapnik and Akshay Vashist. A New Learning Paradigm: Learning Using Privileged Information. Neural Networks, 22(5–6):544–557, 2009.


Appreciate it for your patience and effort. Thank you once again for your valuable reviews. We are looking forward to hearing from you.


Reviewer_C8LdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EMBER-Bench, a benchmark for evaluating cross-event causal memory in long-horizon embodied tasks. It includes 189 household tasks and 699 QA pairs across four memory categories. By pairing next-action prediction with causal traceback, the authors show that identifying relevant historical events does not necessarily lead to correct action selection. Experiments on 16 models further highlight the importance of causal information for decision-making.

Critical strengths
  1. The paper addresses an important gap in existing embodied memory benchmarks by focusing on how historical events affect subsequent decisions, rather than simply testing memory retrieval.

  2. The paired prediction and traceback evaluation is a useful design. Together with the ablation experiments across 16 models, it provides valuable insights into the limitations of current multimodal models.

Critical weaknesses
  1. The causal traceback task provides the next action, which may allow models to rely on event matching rather than genuine causal reasoning. This makes it difficult to determine what the traceback results actually measure.

  2. The improvement from causal annotations may partly come from more explicit event descriptions. The current ablation does not fully separate the benefits of causal information from those of better event understanding.

Questions for authors
  1. Can the authors further distinguish genuine causal reasoning from action-event matching in the traceback task?

  2. Could the authors add a baseline with non-causal event descriptions to better isolate the contribution of causal information?

  3. How would enabling model reasoning affect the main experimental results?

4 comments
AuthorsOpenReview ↗

[Author Response Section 1/3]

We thank the reviewer for the detailed and constructive review. Our motivation is that past events keep constraining a physical agent's actions after they leave view. EMBER-Bench therefore tests whether models use such history to select the next action, not only whether they can retrieve it. Following your suggestions, we ran new controls on the 151 paired questions with five models: Qwen3.8 Omni Flash (Omni), Qwen3.8 Flash (Flash), Gemini 3.8 Flash (Gemini), GPT-6 Sol (GPT) and Qwen3.5 35B-A3B (Q3.5). Differences are in percentage points (pp). Because the API versions and prompts differ from the submission and between control groups, absolute accuracies differ by a few points from the paper and across tables; comparisons should be read within each table.

Traceback provides the next action, so models may rely on event matching rather than genuine causal reasoning, which makes it hard to tell what the traceback results measure (W1). Can causal reasoning be distinguished from action–event matching (Q1)?

The experiments are set below for your concerns. (a) Lexical overlap, no model: the candidate whose description shares the most words with the reference action. (b) Text-only matching by the same model: the action and the four candidate descriptions, no video. (c) No action: the original traceback with the action, and only the wording that depends on it, removed. (d) Salience: "Which clip is most anomalous?", no action.

Table I:

(a) Lexical overlap 32.0%.

ConditionOmniFlashGeminiGPTQ3.5
Original (video + action)44.441.753.063.641.1
(b) Text-only matching46.443.741.743.041.1
(c) No action28.535.155.052.333.1
(d) Most anomalous clip51.048.351.060.344.4

From Table1, we can conduct the following analysis:

Matching the action to the candidate descriptions by shared words does not solve traceback (32.0%).

Gemini and GPT find the event even without the action (55.0% and 52.3%). For them, traceback exceeds their own text-only matching by 11.3 pp (not significant) and 20.5 pp (significant); on items with several anomalous candidates, Gemini exceeds it by a significant 29.0 pp. For these models traceback draws on the video and is not reducible to matching the action to a description. For the three Qwen models, traceback is at the level of text matching (−2.0 to 0.0 pp) and depends on the cue (28.5–35.1% without it). Their text-only matching (41.1–46.4%) is well above lexical overlap (32.0%): these models can judge which event would require the action when events are described in words, but largely fail to locate that event in the video itself. This mirrors Response Section 2 below, where models use past events once they are stated in text.

Moreover, asking only for the most anomalous clip gives 44.4–60.3%, close to action-cued traceback, so salience can locate the event when a single candidate is anomalous. Where several candidates are anomalous and salience cannot decide, the action cue matters (removing it costs 18.3 pp in L3 and 13.3 pp in L4). On these items Gemini still picks the right event without the action (61.3%, vs 41.9% for salience).

Traceback measures whether a model can identify, among candidate past events, the one that makes the given action necessary. In the stronger models this goes beyond lexical overlap and text-level matching, and in Gemini also beyond salience where several candidates are anomalous; in the weaker models it reduces largely to text-level matching with the cue.

The main finding does not rest on the cue. Locating the event is a retrieval-type ability; what EMBER-Bench adds is the test of whether that event then constrains the next action. With traceback run without the action, P accuracy is 33.8% when C is correct and 34.9% when it is not (adjusted for item and model: −5.0 pp, not significant). Thank you again for your constructive suggestion!

AuthorsOpenReview ↗

[Author Response Section 2/3]

The gain from causal annotations may partly come from more explicit event descriptions, and the current ablation does not separate causal information from better event understanding (W2). Could a baseline with non-causal event descriptions isolate the contribution of causal information (Q2)?

Thanks for pointing this out. Here's our experiment setup. On the 151 paired prediction questions we add three inputs of matched length to V+L (video plus action log). E: the paper's cause/consequence annotations. S: a neutral rewrite stating the same events and outcomes as plain observations, with no causal fields or connectives (5,202 vs 5,225 words; every item within ±10%; screened for answer-revealing wording). Renamed fields: E verbatim, with cause/consequence renamed to field_1/field_2. Example from Table 10: S reads "Observation 1: the arm hits the clean apple. Observation 2: the apple rolls onto the floor."; E puts the same two sentences under accident_cause and accident_consequence. We also ran S2 (an end state for every completed action) and the same inputs without video.

Table II:

InputOmniFlashGeminiGPTQ3.5
V+L37.135.146.447.931.1
V+L+S (non-causal)49.747.767.566.945.0
V+L+E (causal)51.746.464.262.949.0
L+S, no video57.058.374.869.557.6
L+E, no video54.351.072.266.255.6
Paired differenceResult
V+L+S − V+L+16.0 pp; significant in all five models
V+L+E − V+L+15.5 pp; significant in all five models
E − renamed fields−2.7 to +0.7 pp in four models; none significant
V+L − V (log only)+6.5 pp; n.s. in four models

The controls answer this directly: the gain comes from the explicit description of what the relevant event changed, and causal labelling adds nothing on top. S matches E (+16.0 vs +15.5 pp); What the event changed is the information Contribution 2 refers to.

Better event description in general does not explain the gain. S2 describes the end state of every completed action, including the relevant one, yet helps less (+10.2 pp), a significant 5.8 pp below describing the relevant event alone. What helps is knowing which past event matters and what it changed, the ability EMBER-Bench targets.

The pattern holds without video: L+S 57.0–74.8% and L+E 51.0–72.2%, against 33.8–45.0% for the log alone. Humans reach 97.8% from the video itself, so the information is in the video, yet models use it only when it is stated in text.

The action log adds little in four models (+0.7 to +6.6 pp, not significant); GPT is the exception (+14.6 pp, significant). About three quarters of this gain comes from L2 (+29.3 pp), where the self-induced anomaly itself appears as a log entry: once the triggering event is written down, GPT infers its consequence and acts on it. On top of the log, consequence information adds 11.3–17.9 pp in every model.

We will describe E as explicit descriptions of past events and their persisting consequences, add S and the renaming control to Sec. 4.2, state that causal field labels add no measurable gain, and report log gains per model. Contribution 2 stands with this precision: what changes decisions is explicit information about the persisting consequence of the relevant past event, not the sequence of past actions and not causal labelling. Thank you again for your constructive suggestion!

AuthorsOpenReview ↗

[Author Response Section 3/3]

Enabling reasoning does not change the main results (Q3)

We first checked with probe calls that each reasoning switch takes effect, using the returned reasoning tokens or reasoning content as evidence. We then ran all 151 P/C pairs with reasoning off and on for the two models whose switch we could verify: Qwen3.8 Omni Flash (low effort) and GPT-6 Sol (medium effort), 604 answers per model. Items, video, prompts, temperature and output limit are identical across the two settings, and invalid or filtered answers count as wrong. An earlier 36-pair pilot covered three models.

Table III:

ModelTaskOffOnOn − off (pp)
OmniP32.533.81.3
OmniC44.449.75.3
GPTP33.138.45.3
GPTC62.965.62.6

36-pair pilot, overall on − off: Omni +5.6, Qwen3.8 Flash −1.4, GPT +8.3 pp; none significant.

Analysis of the results above.

  1. None of the four changes is significant. Even the largest, GPT's +5.3 pp on P, leaves prediction at 38.4%.
  2. The main conclusions are unchanged. With reasoning on, P accuracy is 33.8% and 38.4%, far below the human 97.8%, and C stays above P (49.7% vs 33.8%; 65.6% vs 38.4%). The gap between locating a past event and acting on it remains.

Appreciate it for your patience and effort. Thank you once again for your valuable reviews. We are looking forward to hearing from you.


Thank you for the detailed responses and additional experiments. The new results largely address my concerns, especially regarding action-event matching and the role of explicit event descriptions. I appreciate the clarifications and revisions. While some limitations remain, I find the responses satisfactory and will maintain my rating of 3 (Weak Accept).

Comments on the paper

2 comments
Area_Chair_hUyhOpenReview ↗

Please Engage in the Author-Reviewer Discussion

Dear Reviewers,

The authors have posted their responses to your comments and questions. Please review their replies and share any follow-up comments or questions, especially if any concerns remain unresolved or need further clarification.

Your engagement is important to ensuring a constructive and productive discussion. Thank you for your time and contributions to the ICLR review process.

Best regards,
The AC

AuthorsOpenReview ↗

Response to All Reviewers

Dear all reviewers,

We sincerely thank all reviewers for their careful reading and constructive suggestions. We are grateful for their recognition of the importance of evaluating how past events constrain later actions (Reviewers C8Ld, 8BQG and Ddn8), the focused benchmark design around the persistent consequences of anomalies and interventions (Reviewer 8BQG), the paired next-action prediction and causal traceback evaluation (Reviewers C8Ld and 8BQG), the validation of history dependence through the current-frame-only control and human evaluation (Reviewer 8BQG), the diagnostic evaluation across 16 models (Reviewers 8BQG and C8Ld), the newly recorded videos that avoid training-data contamination (Reviewer Ddn8), and the clarity of the presentation (Reviewers 8BQG and C8Ld).

Following the reviewers' suggestions, we ran new controls on the 151 paired decision points with five models and updated the paper (additions in red). We summarize the main points below.

[Distant history is required] (Reviewer 8BQG) We added recent-window controls that keep only the last 30 s or 8 s of the history. With text history, accuracy drops by 11.4 points on average with the last 30 s and by 21.5 points with the last 8 s (significant in every model), and the drop grows as more key events fall outside the window. With video history, accuracy falls to around chance. We also state explicitly that each question is answered in a single stateless call with the full history, so the benchmark measures the use of past events rather than their storage. (Sec. 3.3, Sec. 4.1, Appendix F.1)

[What traceback measures, and the prediction–traceback gap] (Reviewers 8BQG and C8Ld) Lexical overlap does not solve traceback (32.0%). For the stronger models, traceback goes beyond text-only matching and, where several candidates are anomalous, beyond salience; for the weaker models it reduces largely to text-level matching with the action cue, which we now state. The gap does not depend on the cue: with action-free traceback, correct traceback is still not associated with correct prediction (adjusted −5.0 points, not significant). Giving models the clip they selected, or the gold clip, does not improve prediction, whereas stating the event's consequence in text does. (Sec. 4.3, Appendix F.2)

[Causal information versus more detailed event descriptions] (Reviewers 8BQG and C8Ld) A length-matched neutral rewrite of the annotations, without causal fields or connectives, gives a gain comparable to the causal annotations (+16.0 vs. +15.5 points over V+L, significant in all five models), and renaming the fields has no significant effect. An end-state sentence for every completed action helps significantly less (+10.2 points). We therefore make precise that by causal information we mean what a decision-relevant past event changed and whether that change still holds, not causal labelling, and we revised the abstract, contributions and Sec. 4.2 accordingly. Results without video are also reported. (Abstract, Sec. 1, Sec. 4.2, Appendix F.3)

[Causal CoT and reasoning mode] (Reviewers 8BQG and C8Ld) Against a generic CoT with the same format and output cap, averaged over two runs, causal CoT gives a small mean advantage of 2.6 points (positive in four of five models). We restated the finding in these terms, added the overall net change to Table 3, and removed the capability-dependent interpretation; the accuracy of intermediate consequence assessments is left to future work. Enabling reasoning changes no result significantly, and prediction remains well below traceback and human performance. (Sec. 4.1, Sec. 4.4, Table 3, Appendices F.4–F.5)

[Clarity, scope and follow-up] (Reviewer Ddn8)

  • Clarity: P and C are defined at first mention, and notation is collected in Sec. 4 and Appendix I. L1–L4 share one criterion with examples (Appendix H), the recording setup is described (Sec. 3.1, Appendix G), and privileged information is defined (Sec. 3.2).
  • Long-horizon: the term refers to tasks with many dependent steps, not to video length. Each question receives the full history; 36.5% of questions have more than two minutes of it, and removing the earlier history lowers accuracy in every model tested (Appendix F.1).
  • Text vs. video: the text that outperforms video is human annotations of what past events changed. These annotations are privileged and used only for diagnosis. Without them, a complete action log gives 33.8–45.0%, while humans reach 97.8% from the video alone. Extracting this information from video is the step the benchmark tests (Sec. 4.2, Appendix F.3).
  • Limitations, including dataset scale, are discussed in Appendix J. We thank the reviewers again for their time and suggestions, which have made the paper's claims more precise.

Sincerely,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.