- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes EMBER-Bench, which introduces 189 recorded household tasks and 699 multiple-choice questions targeting history-dependent embodied reasoning. It covers task progress, self-induced anomalies, external interventions, and compound tasks, pairing next-action prediction with causal traceback at 151 decision points. Across 16 models, the best overall accuracy is 61.2%, compared with 98.3% for human evaluators. Additional experiments examine historical information, temporal dependencies, and causal prompting, finding larger gains from privileged cause-and-consequence annotations than from action logs.
Critical strengths
-
Focused and useful benchmark design. The emphasis on persistent consequences of anomalies and interventions provides a concrete setting for studying how historical events constrain subsequent actions. Structured event annotations support interpretable analysis.
-
Meaningful validation of history dependence. The current-frame-only control achieves approximately chance accuracy, while human evaluators perform near ceiling with history. Together with input-boundary checks and manual review, these results support the benchmark’s answerability and the relevance of historical information.
-
Substantial diagnostic evaluation. The paper evaluates 16 models and goes beyond a leaderboard through paired questions, information ablations, and temporal analyses. Task-clustered uncertainty estimates and explicit caveats about privileged annotations and action-cued traceback are welcome.
Critical weaknesses
-
The evaluation does not fully isolate long-horizon memory requirements. Each question provides the complete history in a stateless call, primarily testing full-context video reasoning. Moreover, the shortest event-to-decision interval tertile spans only 0–7.97 seconds. Current-frame failure establishes that some history is needed, but not that distant-event retention or persistent state tracking is necessary. Recent-window and selective-history controls would strengthen this distinction.
-
The prediction–traceback mismatch has limited mechanistic interpretation. Traceback supplies the correct next action and explicitly marks candidate historical clips, whereas prediction requires selecting the action without these cues. Consequently, correct traceback may reflect action–event matching rather than independently recovered causal knowledge. The paper acknowledges this limitation, but the mismatch alone cannot establish a failure to convert retrieved causes into action constraints.
-
The annotation ablation does not cleanly separate causal reasoning from missing state information. In Table 10, the action log states that an arm hits an apple, while the consequence annotation adds that the apple rolled onto the ground. This supplies an additional physical outcome, not merely a causal organization of otherwise identical information. The improvement therefore supports the usefulness of explicit consequences, but does not isolate causal extraction as the main bottleneck.
-
The causal-prompting result needs a stronger control. Causal CoT is compared with direct answering, without a generic reasoning prompt matched for output length or inference budget. Only one of three models improves significantly. Thus, the evidence does not yet distinguish a benefit specific to causal guidance from a general benefit of additional reasoning, and repeated runs would help assess robustness to the reported API nondeterminism.
Questions for authors
-
How does performance change with only a recent video window, only the relevant event, or the history with that event removed? What fraction of questions demonstrably requires distant evidence?
-
Can models first identify relevant events without receiving the reference action, then predict the next action using their own retrieved evidence? Does this produce the same prediction–traceback mismatch?
-
Would equally detailed state descriptions, without explicit causal labels, provide comparable gains? How well do action logs and consequence annotations perform without video?
4/ Does causal CoT outperform generic CoT under matched inference budgets and repeated runs? Are its intermediate consequence assessments systematically accurate?
4 comments
[Author Response Section 2/4]
Traceback supplies the correct next action and marks candidate clips, so a correct traceback may reflect action–event matching, and the mismatch alone cannot establish a failure to convert retrieved causes into action constraints (W2). Can models first identify the event without the action, then predict from their own evidence (Q2)?
Thank you for your valuable suggestion. The ablation settings are shown below:
Action-free traceback: same marked history and current frame; we delete the reference action and only the wording that depends on it. Two-stage: the clip the model selected without the action, with markers removed, is given in a new call together with the current frame and the original goal, question and options. Gold clip: the same, with the gold clip. Association: traceback correctness vs. independent full-history prediction correctness, adjusted for item and model.
Table II
| Condition | Omni | Flash | Gemini | GPT | Q3.5 |
|---|---|---|---|---|---|
| C, with action (original) | 44.4 | 41.7 | 53.0 | 63.6 | 41.1 |
| C, without action | 28.5 | 35.1 | 55.0 | 52.3 | 33.1 |
| P, full history | 35.1 | 33.1 | 37.1 | 37.1 | 29.8 |
| P, self-selected clip | 27.2 | 27.8 | 36.4 | 37.1 | 26.5 |
| P, gold clip | 31.8 | 25.2 | 37.7 | 31.8 | 27.8 |
| Traceback | P acc., C correct | P acc., C wrong | Adjusted Δ (pp) |
|---|---|---|---|
| With action | 34.0% (125/368) | 34.9% (135/387) | −2.5 |
| Without action | 33.8% (104/308) | 34.9% (156/447) | −5.0 |
1.First, the mismatch does not depend on the cue. With action-free traceback, P accuracy is 33.8% when C is correct and 34.9% when it is not (adjusted −5.0 pp, not significant), in line with the action-cued condition (−2.5) and the paper's 16-model estimate (−3.5). Four of five models show no positive association; GPT shows a positive direction (39.6% vs 32.7%) but still pairs about 60% of its correct tracebacks with a wrong prediction.
2.Locating the event, or being handed it, does not improve the decision. Predictions from the self-selected clip (26.5–37.1%) or the gold clip (25.2–37.7%) do not exceed full-history prediction (29.8–37.1%); gold minus self-selected is −0.1 pp. Because a clip alone drops the surrounding history, we also highlighted the gold clip inside the full history, which adds only 1.2 pp (not significant).
3.Stating the event's consequence does improve it: describing the consequence in text raises prediction by 15.5–16.0 pp on average, significantly in every model (Response 3/4).
4.The action does help traceback itself. Removing it lowers traceback by a significant 7.9 pp on average, mainly in L3 (18.3 pp) and L4 (13.3 pp), where several candidates are anomalous. Gemini and GPT still find the event in 55.0% and 52.3% of items without the action.
As your notes and as Sec. 4.3 cautions, the correlation alone is not direct evidence. Results 2 and 3 add interventional evidence: giving the model the right event leaves its action unchanged, while giving it the event's consequence changes it. Thus, even when the correct past event is located or directly provided, models often fail to exploit its consequence when choosing the action, unless that consequence is stated. In the revision, we will present these results together with the cue effect and state the finding in these terms. Thank you again for your constructive suggestion!
[Author Response Section 3/4]
In Table 10 the consequence annotation adds a new physical outcome (the apple rolls onto the floor), so the gain supports explicit consequences but does not isolate causal extraction (W3). Would equally detailed state descriptions without causal labels give comparable gains, and how do logs and annotations perform without video (Q3)?
Thanks a lot for this suggestion. Here's ablation settings for your concern. S: a neutral rewrite of E that states the same events and outcomes as plain observations ("Observation 1: the arm hits the clean apple. Observation 2: the apple rolls onto the floor."), with no cause/consequence fields or causal connectives. S matches E in length (5,202 vs 5,225 words; every item within ±10%) and was screened for answer-revealing wording. S is placed inline with the log, or in a block at E's position. Renamed fields: E verbatim, with cause/consequence renamed to field_1/field_2. S2: an end-state sentence for every completed action, not only the relevant one. Text only: L, L+S, L+E.
Table III.
| Comparison | Omni | Flash | Gemini | GPT | Q3.5 | Mean |
|---|---|---|---|---|---|---|
| V+L − V (log) | 4.6 | 6 | 6.6 | 14.6 | 0.7 | 6.5 |
| V+L+S − V+L | 12.6 | 12.6 | 21.2 | 19.9 | 13.9 | 16 |
| V+L+E − V+L | 14.6 | 11.3 | 17.9 | 15.9 | 17.9 | 15.5 |
| E − renamed fields | 0.7 | −2.0 | n/a | −2.7 | 0.7 | none significant |
| Acc. %, L (no video) | 33.8 | 37.7 | 45 | 38.4 | 36.4 | |
| Acc. %, L+S (no video) | 57 | 58.3 | 74.8 | 69.5 | 57.6 | |
| Acc. %, L+E (no video) | 54.3 | 51 | 72.2 | 66.2 | 55.6 |
From results in Table III, we can see that E indeed adds outcome information beyond the log, and this is the information Contribution 2 refers to: what a past event changed. The controls show that this content, not the causal labelling, drives the gain: S gives +16.0 pp and E +15.5 pp over V+L, each significant in all five models. E vs renamed the fields changes accuracy by −2.7 to +0.7 pp; neither is significant.
More state information does not explain the gain by itself. S2 gives an end state for every completed action, including the relevant one, and so contains more state information than S, yet it helps less (+10.2 pp), a significant 5.8 pp below S. What helps most is knowing which past event matters and what it changed, the distinction EMBER-Bench is built around.
Without video, we can see that the log alone gives 33.8–45.0%, while L+S and L+E give 57.0–74.8% and 51.0–72.2%. Adding the video to L+E does not help (−5.0 pp). Humans reach 97.8% on the same video input, so the information is in the video, yet models use it only when it is stated in text.
On top of the log, consequences add 11.3–17.9 pp in every model, while the log itself adds 0.7–6.6 pp (not significant) in four models; GPT is the exception (+14.6 pp, significant). About three quarters of this gain comes from L2 (+29.3 pp), where the self-induced anomaly itself appears as a log entry: once the triggering event is written down, GPT infers its consequence and acts on it.
We will make precise that by causal information we mean what a decision-relevant past event changed and whether that change still holds, not causal labelling. Concretely, we will (a) describe E as explicit descriptions of past events and their persisting consequences; (b) add S, the renaming control and S2, and state that causal labels add no measurable gain; (c) clarify that "causal extraction" means extracting and using the persisting consequences of past events; and (d) report log gains per model. With this precision Contribution 2 stands: what changes decisions is the consequence of the relevant past event, not the action history. Thank you again for your constructive suggestion!
[Author Response Section 4/4]
Causal CoT is compared only with direct answering, without a generic reasoning prompt matched in output length or inference budget; only one of three models improves significantly, and repeated runs are needed (W4). Does causal CoT outperform generic CoT under matched budgets and repeated runs, and are its intermediate consequence assessments accurate (Q4)?
Thanks for pointing this out. Here's the setup. Direct answering, generic CoT and causal CoT were submitted interleaved in the same batch, in two runs, on 50 prediction questions (17,17,16 with L2/L3/L4 respectively) within 151 questions for better ablation, testing five models. Both CoT variants use the same two-line format (a short assessment, then the answer) and the same cap of two sentences or 60 words. The generic prompt asks only for a brief justification; the causal prompt is the paper's Prompt B.5. Visual input, reasoning setting and output limit are fixed within each model. Since the Qwen3.8-27B API we applied is unstable recently, we changed to different qwen models to elaborate.
Table IV:
| Model | Direct | Generic CoT | Causal CoT | Causal − generic (pp) |
|---|---|---|---|---|
| Omni | 35 | 38 | 33 | −5.0 |
| Flash | 33 | 41 | 43 | 2 |
| Gemini | 50 | 47 | 55 | 8 |
| GPT | 45 | 43 | 45 | 2 |
| Q3.5 | 29 | 31 | 37 | 6 |
| Mean | 38.4 | 40 | 42.6 | 2.6 |
From Table IV, we can compare the performance of causal CoT and generic CoT. Under a matched format and budget, causal CoT is higher than generic CoT in four of five models, by 2.0–8.0 pp (largest in Gemini, +8.0, and Q3.5, +6.0), and lower in Omni (−5.0).
Generic CoT changes accuracy by −3.0 to +8.0 pp relative to direct answering (mean +1.6) and lowers it in Gemini and GPT; causal CoT adds 4.2 pp on average. In Gemini and Q3.5, the two models with the largest causal advantage, generic CoT adds little or nothing (−3.0 and +2.0), so their advantage is not explained by extra reasoning. Flash is the case you anticipated: a generic justification already gives most of its gain (+8.0 of +10.0 over direct).
Model capability. The gain does not track model strength: causal − direct is +5.0 for Gemini and 0.0 for GPT, against +10.0 for Flash and +8.0 for Q3.5. We will remove the capability-dependent reading in Sec. 4.4.
We will restate the CoT finding as: causal CoT gives a small advantage over a matched generic CoT in most models (mean +2.6 pp) and we will report the generic control, both runs and per-model results in the paper. This was a secondary observation, and the main findings are unchanged.
Appreciate it for your patience and effort. Thank you once again for your valuable reviews. We are looking forward to hearing from you.
[Author Response Section 1/4]
We thank the reviewer for the detailed and constructive review. Our motivation is that past events keep constraining a physical agent's actions after they leave view. EMBER-Bench therefore tests whether models use such history to select the next action, not only whether they can retrieve it. Following your suggestions, we ran new controls on the 151 paired questions with five models: Qwen3.8 Omni Flash (Omni), Qwen3.8 Flash (Flash), Gemini 3.8 Flash (Gemini), GPT-6 Sol (GPT) and Qwen3.5 35B-A3B (Q3.5). Differences are in percentage points (pp). Because the API versions and prompts differ from the submission and between control groups, absolute accuracies differ by a few points from the paper and across tables; comparisons should be read within each table.
Thanks a lot for this suggestion. Experiment setup: Recent window: only the last 8 s or 30 s of the history, taken from the same frame sequence as the full history; goal, current frame, question and options unchanged. Only the earlier history is removed. Text / Event window: the history as text (action log plus neutral state descriptions, L+S), keeping only events that start and end inside the window.
Table I. Experiment Results with different lengths of history window
How many items depend on evidence outside the window. By annotation, the key event ends more than 8 s before the decision in about two thirds of the 151 pairs, and more than 30 s before it in about one third (Appendix E.3, Table 8). The windows therefore remove the key event from a large share of items, and from more items at 8 s than at 30 s.
When the history is given as text, models can actually use it (53.6–68.9% with full history). Keeping only the last 30 s lowers accuracy by 6.7–17.9 pp (mean 11.4). Keeping only the last 8 s lowers it by 17.8–31.8 pp (mean 21.5), to 35.8–37.1%, significant in every model. The drop roughly doubles from the 30 s to the 8 s window, as the share of items whose key event falls outside the window grows. This graded pattern is what we expect if answers depend on earlier events, and not if they depend on the most recent frames.
Video history. All eight model–window cells drop: by 3.3–10.6 pp at 8 s and 3.9–13.2 pp at 30 s (means 8.4 and 8.1 pp), to 19.9–30.5%, around chance (25%). Distractors are built as the actions a model would choose if it ignored or misread the earlier event (Sec. 3.3), and a recent window alone leads models to them. Because full-history video accuracy is already low (23.8–38.4%), there is little room for a graded difference between the two windows, and Q3.5, near chance even with full history, changes least. We therefore treat the text condition as the more sensitive measure of what the items require. The video condition shows that models also draw on earlier history when it is presented as video.
This control asks a different question from the windows: not whether the item needs earlier evidence, but whether a model’s answer currently depends on that specific clip. Given only the gold clip and the current frame, the prediction is 25.2–37.7%, which does not exceed full-history prediction (29.8–37.1%; Table II in our response to W2). This matches our responses to W2 and W3: handing models the gold clip does not improve their decisions, whereas stating its consequence in text raises accuracy by 15.5–16.0 pp in every model. Together, the controls separate two findings:
The second finding is the gap EMBER-Bench is designed to measure.
In summary, we agree that each question is answered in one stateless call with the full history, and this is deliberate. Full access is the most favorable condition for retention, so the failures we report concern the use of past events, not their storage. We will state this scope explicitly in Sec. 3 and the Limitations, and add Table I together with all controls above to the Appendix. Thank you again for this suggestion, which has made the distinction considerably sharper.