- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of causal attribution in videos. The task is to see if the outcome would have been the same if action was instead of , a process called counterfactual inference. Under well established causal assumptions (Consistency, Positivity, No Unmeasured Confounding) plus Conditional Independence of Potential Outcomes and a Sufficient Scene Representation, the authors identify via both Outcome Regression (Prop. 2) and Inverse Propensity Weighting (Prop. 3). They then derive the Efficient Influence Function (Prop. 4) and propose a cross-fitted, one-step doubly robust estimator that achieves asymptotic normality and semiparametric efficiency at fixed bandwidth (Thm. 5). TongSim-CF benchmark is introduced, which contains simulated household videos and is used for evaluating the proposed approach.
Critical strengths
- The motivation and the problem statement are clearly defined and the distinction between correlated VLM priors and true counterfactual attribution is an important direction for learning (physical) mechanisms of an environment.
- The authors go beyond main table metrics to test how their estimator behaves when key unverified assumptions fail. Specifically injecting an unmeasured confounder (violating Ass. 3) and coupling potential outcomes via a Gaussian copula (violating Ass. 4) in Section 5.4.
Critical weaknesses
- The requirement of 551-601 videos per environment, with various combinations of actions in the same environment, limits the zero-shot generalization to new scenes.
- Physics-101 dataset provides videos without confounding. All that matters in this dataset is a physical parameter, angle ( vs degrees). Thus there are no concurrent actions, , that might confound that final outcome .
- Even though the paper is about video causal attributions, the experiments are collapsed into a single frame which is the first frame before the action happens. If an action's causal effect depends on how the execution evolves over time (trajectory, timing, physical interactions), conditioning only on a single frame will miss it completely.
- In Setting 3 of TongSim-CF, where scene conditions confound action selection and outcomes, the paper highlights a significant error reduction over the strongest baseline (0.061 vs. 0.169 MAE for V-JEPA 2). However, looking closely at Table 5 (Appendix I.1) and Table 6 (Appendix I.2), the proposed estimator is given explicit discrete attributes ("Simulator preconditions", such as binary power or lock states read from the simulator, or extracted via Gemini-3-Flash), whereas the visual baselines (V-JEPA 2, Qwen2.5-VL, LLCP) only receive raw first-frame visual features and action indicators. Crucially, in the authors' own ablation in Table 8 (Appendix J.2), when is removed and the proposed estimator relies solely on continuous visual features , its Setting 3 MAE jumps nearly from 0.061 to 0.169, which is identical to V-JEPA 2 (0.169). This suggests that the performance gap in Setting 3 is largely driven by access to explicit discrete precondition labels (and exact indicator-kernel matching on ) rather than the doubly robust estimator alone.
Questions for authors
- In Table 1 (Setting 3), did the fine-tuned baselines (Qwen2.5-VL, V-JEPA 2) and LLCP also receive the explicit discrete precondition attributes as input features, or did they only receive the raw first frame? How do V-JEPA 2 and Qwen2.5-VL perform in Setting 3 if is concatenated to their action inputs, and conversely, how does your estimator perform when is extracted by Gemini-3-Flash (with its attendant extraction errors) rather than read from the simulator state?
3 comments
Response to Reviewer CeBS (2/2): Questions 1–2
About other technical details:
Question 1. "The requirement of 551-601 videos per environment … limits the zero-shot generalization to new scenes."
We do not claim zero-shot generalization. Our method learns each environment from videos recorded in it, and it does not need many such videos. To test this, we rerun our method and the finetuned and causal baselines with 50 to 350 random videos per environment. We report the overall MAE ± SEM over 31 variants. Results show that our method is the most accurate at every number of videos (Appendix K.3).
| Method | 50 | 100 | 200 | 350 | All |
|---|---|---|---|---|---|
| Ours | 0.135 ± 0.009 | 0.100 ± 0.010 | 0.077 ± 0.008 | 0.058 ± 0.006 | 0.046 ± 0.005 |
| V-JEPA 2 + head, with | 0.247 ± 0.011 | 0.215 ± 0.010 | 0.163 ± 0.011 | 0.125 ± 0.009 | 0.101 ± 0.011 |
| Qwen2.5-VL + LoRA | 0.166 ± 0.015 | 0.156 ± 0.017 | 0.160 ± 0.017 | 0.147 ± 0.016 | 0.137 ± 0.018 |
| LLCP | 0.294 ± 0.012 | 0.283 ± 0.013 | 0.257 ± 0.015 | 0.237 ± 0.016 | 0.216 ± 0.018 |
| Fire-Zhu | 0.209 ± 0.016 | 0.195 ± 0.017 | 0.188 ± 0.017 | 0.184 ± 0.017 | 0.182 ± 0.018 |
Question 2. "If an action's causal effect depends on how the execution evolves over time (trajectory, timing, physical interactions), conditioning only on a single frame will miss it completely."
We believe that conditioning on the first frame does not miss this effect. By consistency (Assumption 1), a trial with shows the outcome , which already includes how the action unfolds over time. We use the first frame only to adjust for confounders of the actions and the outcome. Conditioning on later frames could instead block part of the effect.
Follow-up Before the Discussion Deadline
Dear Reviewer CeBS,
With only a few hours remaining in the discussion period, we would like to kindly follow up on our response and manuscript revisions. We have added to the inputs of the baselines, tested read by Gemini-3-Flash, and added three real datasets with concurrent actions and confounding. We have also tested fewer videos per environment and clarified the role of the first frame. Details are in our response to Concerns 1–2 and Questions 1–2.
We would greatly appreciate your feedback on whether these results address your main concerns, particularly any issues that remain unresolved or require further clarification. Your feedback would help us address outstanding questions before the discussion closes.
Thank you again for your time and consideration.
Response to Reviewer CeBS (1/2): Concerns 1–2
We thank the reviewer for the careful review and for recognizing our clearly defined problem, the importance of separating correlated VLM priors from counterfactual attribution, and our tests under violated assumptions. The reviewer's concerns mainly relate to Zd, the lack of confounding in Physics 101, and other minor technical details.
Concern 1: The attributes Zd in Setting 3
(a) Baselines with and without Zd
Qwen2.5-VL, V-JEPA 2, and LLCP receive the first frame, in which Zd is visible. We now also add Zd explicitly to their inputs. We report the average MAE ± SEM over 31 variants (10 in Setting 3). Results show that all three baselines remain much less accurate than our method. We give the other baselines with Zd in Appendix K.7 of the revised manuscript.
(b) Zd read by Gemini-3-Flash
Our error barely changes. We read Zd from the first frame with Gemini-3-Flash. We report the average MAE ± SEM over 31 variants.
Concern 2: Confounding in real data
We add three real datasets with concurrent actions and confounding. They are robotic grasping [1], robotic scooping [2], and C. elegans [3]. Specifically, the confounder in robotic grasping is the item's weight. In robotic scooping, it is the terrain. In C. elegans, it is the worm's movement. We describe the datasets in Sec. 5.3 and Appendix J of the revised manuscript. We report the average MAE ± SEM over units (Table 2 of the revised manuscript). Results show that our method has the lowest error on all three datasets. This suggests that it attributes outcomes accurately when confounders exist.
∗We evaluate these methods on 500 random queries per repetition; our method obtains 0.128 ± 0.007 on them. ‡LLCP needs a frame after the action, which only 17 grasping units have. We evaluate LLCP on these units, where our method obtains 0.156 ± 0.016.
References
[1] Wang et al., "Multimodal grasp data set: A novel visual–tactile data set for robotic manipulation," International Journal of Advanced Robotic Systems 2019.
[2] Zhu et al., "Few-shot adaptation for manipulating granular materials under domain shift," RSS 2023.
[3] Liu et al., "A high-throughput method to deliver targeted optogenetic stimulation to moving C. elegans populations," PLoS Biology 2022.