acceptodds
Under review as a conference paper at ICLR 2027

Causal Attribution from Video via Counterfactual Estimation

Abstract

Causal attribution from video asks whether an outcome would have changed if one of the observed actions had been different. Such attribution is essential for embodied agents that must reason about past actions to act safely next time. It is difficult because several concurrent actions can explain the same outcome. Pretrained models may be misled by correlations under confounding, and counterfactual outcomes are rarely labelled. We formulate video causal attribution as counterfactual inference given an observed video. To evaluate methods on this problem, we build TongSim-CF, a household video benchmark with controlled confounding from concurrent actions and scene conditions. We identify the resulting attribution target from repeated videos of the same environment without counterfactual supervision. We then estimate a local average of the target around the observed scene. Our estimator combines outcome predictions with a weighted residual correction derived from the efficient influence function. In the hardest simulated and real settings, it reduces the error of the strongest baseline by and .

Primary area:
causal reasoning
Keywords:
Causal attribution, video understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CeBSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of causal attribution in videos. The task is to see if the outcome YY would have been the same if action was a′a' instead of aa, a process called counterfactual inference. Under well established causal assumptions (Consistency, Positivity, No Unmeasured Confounding) plus Conditional Independence of Potential Outcomes and a Sufficient Scene Representation, the authors identify ψh\psi_h via both Outcome Regression (Prop. 2) and Inverse Propensity Weighting (Prop. 3). They then derive the Efficient Influence Function (Prop. 4) and propose a cross-fitted, one-step doubly robust estimator that achieves asymptotic normality and semiparametric efficiency at fixed bandwidth hh (Thm. 5). TongSim-CF benchmark is introduced, which contains simulated household videos and is used for evaluating the proposed approach.

Critical strengths
  • The motivation and the problem statement are clearly defined and the distinction between correlated VLM priors and true counterfactual attribution is an important direction for learning (physical) mechanisms of an environment.
  • The authors go beyond main table metrics to test how their estimator behaves when key unverified assumptions fail. Specifically injecting an unmeasured confounder γ\gamma (violating Ass. 3) and coupling potential outcomes via a Gaussian copula ρ\rho (violating Ass. 4) in Section 5.4.
Critical weaknesses
  • The requirement of 551-601 videos per environment, with various combinations of actions in the same environment, limits the zero-shot generalization to new scenes.
  • Physics-101 dataset provides videos without confounding. All that matters in this dataset is a physical parameter, angle (10∘10^\circ vs 20∘20^\circ degrees). Thus there are no concurrent actions, Aˉ\bar{A}, that might confound that final outcome YY.
  • Even though the paper is about video causal attributions, the experiments are collapsed into a single frame VV which is the first frame before the action happens. If an action's causal effect depends on how the execution evolves over time (trajectory, timing, physical interactions), conditioning only on a single frame will miss it completely.
  • In Setting 3 of TongSim-CF, where scene conditions confound action selection and outcomes, the paper highlights a significant error reduction over the strongest baseline (0.061 vs. 0.169 MAE for V-JEPA 2). However, looking closely at Table 5 (Appendix I.1) and Table 6 (Appendix I.2), the proposed estimator is given explicit discrete attributes ZdZ^d ("Simulator preconditions", such as binary power or lock states read from the simulator, or extracted via Gemini-3-Flash), whereas the visual baselines (V-JEPA 2, Qwen2.5-VL, LLCP) only receive raw first-frame visual features and action indicators. Crucially, in the authors' own ablation in Table 8 (Appendix J.2), when ZdZ^d is removed and the proposed estimator relies solely on continuous visual features ZcZ^c, its Setting 3 MAE jumps nearly 3×3\times from 0.061 to 0.169, which is identical to V-JEPA 2 (0.169). This suggests that the performance gap in Setting 3 is largely driven by access to explicit discrete precondition labels ZdZ^d (and exact indicator-kernel matching on ZdZ^d) rather than the doubly robust estimator alone.
Questions for authors
  • In Table 1 (Setting 3), did the fine-tuned baselines (Qwen2.5-VL, V-JEPA 2) and LLCP also receive the explicit discrete precondition attributes ZdZ^d as input features, or did they only receive the raw first frame? How do V-JEPA 2 and Qwen2.5-VL perform in Setting 3 if ZdZ^d is concatenated to their action inputs, and conversely, how does your estimator perform when ZdZ^d is extracted by Gemini-3-Flash (with its attendant extraction errors) rather than read from the simulator state?
3 comments
AuthorsOpenReview ↗

Response to Reviewer CeBS (1/2): Concerns 1–2

We thank the reviewer for the careful review and for recognizing our clearly defined problem, the importance of separating correlated VLM priors from counterfactual attribution, and our tests under violated assumptions. The reviewer's concerns mainly relate to ZdZ^d, the lack of confounding in Physics 101, and other minor technical details.

Concern 1: The attributes ZdZ^d in Setting 3

(a) Baselines with and without ZdZ^d

"In Table 1 (Setting 3), did the fine-tuned baselines (Qwen2.5-VL, V-JEPA 2) and LLCP also receive the explicit discrete precondition attributes ZdZ^d as input features, or did they only receive the raw first frame?"

"How do V-JEPA 2 and Qwen2.5-VL perform in Setting 3 if ZdZ^d is concatenated to their action inputs …?"

Qwen2.5-VL, V-JEPA 2, and LLCP receive the first frame, in which ZdZ^d is visible. We now also add ZdZ^d explicitly to their inputs. We report the average MAE ± SEM over 31 variants (10 in Setting 3). Results show that all three baselines remain much less accurate than our method. We give the other baselines with ZdZ^d in Appendix K.7 of the revised manuscript.

MethodSetting 3, without ZdZ^dSetting 3, with ZdZ^dOverall, with ZdZ^d
Qwen2.5-VL + LoRA0.214 ± 0.0360.192 ± 0.0310.126 ± 0.017
V-JEPA 2 + head0.146 ± 0.0280.136 ± 0.0260.101 ± 0.011
LLCP0.227 ± 0.0340.186 ± 0.0250.198 ± 0.018
Ours0.173 ± 0.0370.065 ± 0.0100.046 ± 0.005

(b) ZdZ^d read by Gemini-3-Flash

"… how does your estimator perform when ZdZ^d is extracted by Gemini-3-Flash (with its attendant extraction errors) rather than read from the simulator state?"

Our error barely changes. We read ZdZ^d from the first frame with Gemini-3-Flash. We report the average MAE ± SEM over 31 variants.

ZdZ^dSetting 3Overall
Simulator0.065 ± 0.0100.046 ± 0.005
Gemini-3-Flash, from the first frame0.067 ± 0.0100.046 ± 0.005

Concern 2: Confounding in real data

"Physics-101 dataset provides videos without confounding … there are no concurrent actions, Aˉ\bar A, that might confound that final outcome YY."

We add three real datasets with concurrent actions and confounding. They are robotic grasping [1], robotic scooping [2], and C. elegans [3]. Specifically, the confounder in robotic grasping is the item's weight. In robotic scooping, it is the terrain. In C. elegans, it is the worm's movement. We describe the datasets in Sec. 5.3 and Appendix J of the revised manuscript. We report the average MAE ± SEM over units (Table 2 of the revised manuscript). Results show that our method has the lowest error on all three datasets. This suggests that it attributes outcomes accurately when confounders exist.

MethodGraspingScoopingC. elegans
Ours0.124 ± 0.0140.126 ± 0.0060.053 ± 0.002
GPT-5.6, five examples0.237 ± 0.0280.327 ± 0.009$^*$0.191 ± 0.006
Gemini-3-Flash, five examples0.271 ± 0.0310.296 ± 0.009$^*$0.207 ± 0.007
GPT-5.6 + retrieval0.227 ± 0.0260.189 ± 0.007$^*$0.113 ± 0.004
Gemini-3-Flash + retrieval0.250 ± 0.0180.146 ± 0.006$^*$0.129 ± 0.003
LLCP0.203 ± 0.029$^\ddagger$0.173 ± 0.0070.124 ± 0.003
Fire-Zhu0.346 ± 0.0290.218 ± 0.0070.181 ± 0.006
Qwen2.5-VL + LoRA0.362 ± 0.0270.143 ± 0.0070.146 ± 0.003
V-JEPA 2 + head0.331 ± 0.0480.149 ± 0.0070.151 ± 0.004
S-learner0.225 ± 0.0270.154 ± 0.0060.153 ± 0.004
T-learner0.234 ± 0.0290.158 ± 0.0060.155 ± 0.003
DR-learner0.259 ± 0.0300.177 ± 0.0050.151 ± 0.003

∗^*We evaluate these methods on 500 random queries per repetition; our method obtains 0.128 ± 0.007 on them. ‡^\ddaggerLLCP needs a frame after the action, which only 17 grasping units have. We evaluate LLCP on these units, where our method obtains 0.156 ± 0.016.

References

[1] Wang et al., "Multimodal grasp data set: A novel visual–tactile data set for robotic manipulation," International Journal of Advanced Robotic Systems 2019.
[2] Zhu et al., "Few-shot adaptation for manipulating granular materials under domain shift," RSS 2023.
[3] Liu et al., "A high-throughput method to deliver targeted optogenetic stimulation to moving C. elegans populations," PLoS Biology 2022.

AuthorsOpenReview ↗

Response to Reviewer CeBS (2/2): Questions 1–2

About other technical details:

Question 1. "The requirement of 551-601 videos per environment … limits the zero-shot generalization to new scenes."

We do not claim zero-shot generalization. Our method learns each environment from videos recorded in it, and it does not need many such videos. To test this, we rerun our method and the finetuned and causal baselines with 50 to 350 random videos per environment. We report the overall MAE ± SEM over 31 variants. Results show that our method is the most accurate at every number of videos (Appendix K.3).

Method50100200350All
Ours0.135 ± 0.0090.100 ± 0.0100.077 ± 0.0080.058 ± 0.0060.046 ± 0.005
V-JEPA 2 + head, with ZdZ^d0.247 ± 0.0110.215 ± 0.0100.163 ± 0.0110.125 ± 0.0090.101 ± 0.011
Qwen2.5-VL + LoRA0.166 ± 0.0150.156 ± 0.0170.160 ± 0.0170.147 ± 0.0160.137 ± 0.018
LLCP0.294 ± 0.0120.283 ± 0.0130.257 ± 0.0150.237 ± 0.0160.216 ± 0.018
Fire-Zhu0.209 ± 0.0160.195 ± 0.0170.188 ± 0.0170.184 ± 0.0170.182 ± 0.018

Question 2. "If an action's causal effect depends on how the execution evolves over time (trajectory, timing, physical interactions), conditioning only on a single frame will miss it completely."

We believe that conditioning on the first frame does not miss this effect. By consistency (Assumption 1), a trial with A=aA=a shows the outcome YaY_a, which already includes how the action unfolds over time. We use the first frame only to adjust for confounders of the actions and the outcome. Conditioning on later frames could instead block part of the effect.

AuthorsOpenReview ↗

Follow-up Before the Discussion Deadline

Dear Reviewer CeBS,

With only a few hours remaining in the discussion period, we would like to kindly follow up on our response and manuscript revisions. We have added ZdZ^d to the inputs of the baselines, tested ZdZ^d read by Gemini-3-Flash, and added three real datasets with concurrent actions and confounding. We have also tested fewer videos per environment and clarified the role of the first frame. Details are in our response to Concerns 1–2 and Questions 1–2.

We would greatly appreciate your feedback on whether these results address your main concerns, particularly any issues that remain unresolved or require further clarification. Your feedback would help us address outstanding questions before the discussion closes.

Thank you again for your time and consideration.

Reviewer_HScNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether repeated videos from the same environment can help identify which actions caused an observed outcome. The authors aim to distinguish an action’s effect from correlations caused by other actions or scene conditions. They propose estimating the probability of an outcome if one action were changed while the other actions and scene conditions were held fixed, using recorded examples without counterfactual training labels. The paper introduces TongSim-CF, develops a statistical estimator under explicit assumptions, and evaluates the approach on simulated household scenes and real object-sliding experiments.

Critical strengths
  1. The paper provides a useful benchmark for a meaningful causal reasoning problem. The question of which action caused an outcome matters for agents learning from video, especially when several actions occur together. The availability of counterfactual ground truth for evaluation makes this a useful resource for studying failures that ordinary video prediction benchmarks may not reveal.
  2. The method connects a clear causal target to principled estimation and informative experiments. The authors specify the counterfactual probability they want to estimate, state the assumptions needed to identify it from observational data, and derive an efficient estimator for a local average around the observed scene. The distinction between identification and estimation is useful, and the analysis makes clear what must hold for the statistical guarantees to apply.
Critical weaknesses

The concerns are mostly about the proposed method:

  1. The comparison does not clearly show how much the new estimator helps. The proposed method receives important scene information directly from the simulator, while the visual baselines must infer this information from images. The improvement may therefore partly reflect this difference in inputs. In the hardest simulated setting, removing the supplied state labels increases the proposed method’s MAE from 0.061 to 0.169, matching the strongest visual baseline (Table 8). Comparing the methods with the same scene information would clarify how much of the improvement comes from the estimator itself.
  2. The improvement on real data is modest. On Physics 101, MAE decreases from 0.153 to 0.146 on the nine objects with variable outcomes, while the overall MAE of 0.068 matches LLCP (Table 1).
Questions for authors
  1. Contribution of the estimator. Does the proposed estimator still outperform the baselines when all methods receive the same scene information? This would clarify how much of the improvement comes from the estimator itself.

  2. Evidence on real data. How consistent is the modest improvement on Physics 101 across objects? Given that this dataset contains no concurrent actions, is there additional evidence that the method can distinguish the effects of multiple actions in real videos?

2 comments
AuthorsOpenReview ↗

Response to Reviewer HScN: Concerns 1–2

We thank the reviewer for the positive assessment and for recognizing our useful benchmark with counterfactual ground truth, clear causal estimand, and principled estimation. The reviewer's concerns mainly relate to the comparison under the same scene information and the evidence on real data.

Concern 1: Same scene information for all methods

"Does the proposed estimator still outperform the baselines when all methods receive the same scene information?"

Yes. To give all methods the same scene information, we give ZdZ^d to all baselines that can take it as input. We report the average MAE ± SEM over 31 variants. Results show that our method has the best overall performance. This suggests that the improvement comes from our methodological design, not from the scene information.

Method (all with ZdZ^d)Setting 3Overall
Qwen2.5-VL + LoRA0.192 ± 0.0310.126 ± 0.017
V-JEPA 2 + head0.136 ± 0.0260.101 ± 0.011
GPT-5.6 + retrieval$^\dagger$0.100 ± 0.0150.076 ± 0.007
Gemini-3-Flash + retrieval$^\dagger$0.097 ± 0.0160.069 ± 0.007
S-learner0.062 ± 0.0060.052 ± 0.004
T-learner0.078 ± 0.0110.065 ± 0.007
X-learner0.093 ± 0.0050.097 ± 0.004
DR-learner0.058 ± 0.0060.055 ± 0.004
Ours0.065 ± 0.0100.046 ± 0.005

†^\daggerWe evaluate these methods on 1,762 random tasks, where our method obtains 0.069 (Setting 3) and 0.047 (overall).

Concern 2: Evidence on real data

(a) Consistency on Physics 101

"How consistent is the modest improvement on Physics 101 across objects?"

On Physics 101, the improvement varies across objects, because the dataset has a single action and no confounding. In this case, a strong outcome predictor is already accurate, and our method ties the strongest baseline overall (Appendix I.3). Our method is designed for concurrent actions under confounding, which we test on three new real datasets in (b).

(b) Multiple actions in real videos

"Given that this dataset contains no concurrent actions, is there additional evidence that the method can distinguish the effects of multiple actions in real videos?"

We add three real datasets in which each trial has two concurrent actions: robotic grasping [1], robotic scooping [2], and C. elegans [3] (Sec. 5.3 and Appendix J of the revised manuscript). To test whether a method distinguishes the effects of the two actions, we switch only one action in each query. We report the average MAE ± SEM over units (Table 2 of the revised manuscript). Results show that our method has the lowest error on all three datasets. This suggests that it distinguishes the effects of multiple actions in real videos.

MethodGraspingScoopingC. elegans
Ours0.124 ± 0.0140.126 ± 0.0060.053 ± 0.002
GPT-5.6, five examples0.237 ± 0.0280.327 ± 0.009$^*$0.191 ± 0.006
Gemini-3-Flash, five examples0.271 ± 0.0310.296 ± 0.009$^*$0.207 ± 0.007
GPT-5.6 + retrieval0.227 ± 0.0260.189 ± 0.007$^*$0.113 ± 0.004
Gemini-3-Flash + retrieval0.250 ± 0.0180.146 ± 0.006$^*$0.129 ± 0.003
LLCP0.203 ± 0.029$^\ddagger$0.173 ± 0.0070.124 ± 0.003
Fire-Zhu0.346 ± 0.0290.218 ± 0.0070.181 ± 0.006
Qwen2.5-VL + LoRA0.362 ± 0.0270.143 ± 0.0070.146 ± 0.003
V-JEPA 2 + head0.331 ± 0.0480.149 ± 0.0070.151 ± 0.004
S-learner0.225 ± 0.0270.154 ± 0.0060.153 ± 0.004
T-learner0.234 ± 0.0290.158 ± 0.0060.155 ± 0.003
DR-learner0.259 ± 0.0300.177 ± 0.0050.151 ± 0.003

∗^*We evaluate these methods on 500 random queries per repetition; our method obtains 0.128 ± 0.007 on them. ‡^\ddaggerLLCP needs a frame after the action, which only 17 grasping units have. We evaluate LLCP on these units, where our method obtains 0.156 ± 0.016.

References

[1] Wang et al., "Multimodal grasp data set: A novel visual–tactile data set for robotic manipulation," International Journal of Advanced Robotic Systems 2019.
[2] Zhu et al., "Few-shot adaptation for manipulating granular materials under domain shift," RSS 2023.
[3] Liu et al., "A high-throughput method to deliver targeted optogenetic stimulation to moving C. elegans populations," PLoS Biology 2022.

AuthorsOpenReview ↗

Follow-up Before the Discussion Deadline

Dear Reviewer HScN,

With only a few hours remaining in the discussion period, we would like to kindly follow up on our response and manuscript revisions. We have compared all methods under the same scene information ZdZ^d, and added three real datasets with concurrent actions. Details are in our response to Concerns 1–2.

We would greatly appreciate your feedback on whether these results address your questions, particularly any issues that remain unresolved or require further clarification. Your feedback would help us address outstanding questions before the discussion closes.

Thank you again for your time and consideration.

Reviewer_fYDjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates causal attribution from video as estimating E[Ya′∣Y=y,A=a,Aˉ=aˉ,V=v]E[Y_{a'}\mid Y=y,A=a,\bar A=\bar a,V=v]. Given the pre-action frame and the other concurrent actions, this is the probability that the outcome would have occurred had the target action been a′a'. To evaluate this task, the authors build TongSim-CF, a simulated household benchmark. It has controlled confounding from co-occurring actions and scene preconditions, and counterfactual ground truth computed by the simulator. Under some assumptions, they identify a kernel-localized version of the target from repeated videos of the same environment, without counterfactual labels. They also derive its efficient influence function and prove that a cross-fitted one-step estimator is asymptotically normal and efficient at fixed bandwidth. The implementation uses discrete preconditions recorded by the simulator, low-dimensional visual features and logistic nuisance models. It reaches an overall MAE of 0.045 on TongSim-CF, and 0.061 vs. 0.169 for the strongest baseline in the hardest setting. On the Physics 101 ramp subset it matches the best baselines overall and gains 5% on a 9-object stochastic subset. This paper is solid with theory but still have some concerns about experiment part, if I can see more experiments I would raise my score.

Critical strengths
  • A benchmark that makes confounding measurable. TongSim-CF provides what existing video-causality benchmarks lack:

    • overlapping videos under fixed mechanisms;

    • controlled confounding from both actions and scene conditions;

    • counterfactual ground truth.

  • Principled estimation.

    • As far as I could check, all the main theoretical results are correct: the identification results (Props. 2–3), the efficient influence function (Prop. 4), the asymptotic analysis (Thm. 5) and the variance formula (Eq. 8).
    • The authors state clearly what the guarantees do and do not cover.
    • The appendix diagnostics are thorough: misspecification, assumption violations, bandwidth and sample size.
Critical weaknesses
  1. The headline gains are not attributable to the estimator.
  • Privileged confounders. The method uses discrete preconditions ZdZ_d read from the simulator state (Sec. 5.1). These are exactly the Setting-3 confounders, and no baseline receives them (Tab. 6). Without ZdZ_d, the method’s Setting-3 MAE is 0.169 (Tab. 8), identical to the strongest baseline, so the 64% gain disappears. No results with pixel-derived ZdZ_d are reported.
  • No isolation of the estimator.
    • With correctly specified nuisances, OR, IPW and the full estimator perform the same (0.047/0.045/0.048, Tab. 7).
    • The gain from adding the correction to foundation-model predictors (Tab. 15) again relies on weights fitted with ZdZ_d.
    • No standard adjusted regression or CATE learner on the same features is compared.
  • Unequal protocols.
    • The method uses all videos, while the finetuned baselines use half.
    • VLMs are evaluated on only 53 tasks.
    • The bandwidth is justified on test tasks (Tab. 11).
  • Weak real-video evidence. Physics 101 has neither concurrent actions nor confounding, and the method only ties the baselines overall. The 5% gain rests on 9 objects and lies within one SEM.
  1. Belows are my concerns about theory part. The counterfactual framing and the efficiency result add less than claimed.
  • No dependence on the observed outcome. Under Ass. 1–5 the pointwise target equals E[Y∣A=a′,Aˉ=aˉ,Z=g(v)]E[Y\mid A=a',\bar A=\bar a,Z=g(v)], which does not depend on yy; the benchmark ground truth satisfies this by construction (Sec. H.2). The task is therefore conditional interventional (CATE-type) estimation. Yet CATE methods are neither discussed nor compared.
  • Efficiency is relative to a model that ignores Ass. 5. Prop. 4 and Thm. 5 work in the unrestricted observed-data model. However, Ass. 5 implies the testable restriction (Y,A,Aˉ)⊥ ⁣ ⁣ ⁣⊥V∣Z(Y,A,\bar A)\perp\!\!\!\perp V\mid Z (Sec. D.2).
    • Under this restriction the efficient influence function is ϕ−R/E[ρh]\phi-R/E[\rho_h], where R=(μ−ψh)I(Y=y,A=a,Aˉ=aˉ)−c(Z)Kh(V−v)−r(Z)R=(\mu-\psi_h){I(Y=y,A=a,\bar A=\bar a)-c(Z)}{K_h(V-v)-r(Z)} and c(Z)=Pr⁡(Y=y,A=a,Aˉ=aˉ∣Z)c(Z)=\Pr(Y=y,A=a,\bar A=\bar a\mid Z).
    • Eq. (7) therefore exceeds the bound by E[(μ−ψh)2c(1−c),Var(Kh(V−v)∣Z)]/E[ρh]2E[(\mu-\psi_h)^2c(1-c),\mathrm{Var}(K_h(V-v)\mid Z)]/E[\rho_h]^2.
    • It is efficient only if the kernel depends on VV through ZZ alone, and then rr is known exactly and needs no Monte-Carlo proxy.
  • Double robustness is weaker than suggested. The weight ff contains η=Pr⁡(Y=y∣A=a,Aˉ=aˉ,Z)\eta=\Pr(Y=y\mid A=a,\bar A=\bar a,Z), which is itself an outcome model, and μ\mu and η\eta are fitted with the same classifier (Sec. 4.3). A misspecified outcome classifier therefore biases both terms, so the IPW route (Sec. 4.1) does not protect against outcome misspecification.
  • Theory–practice gap. The guarantees concern a fixed-bandwidth kernel on raw frames. The implementation instead uses PCA features.
Questions for authors

See weekness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer fYDj (1/3): Concerns 1–2

We thank the reviewer for the thorough review and for recognizing our benchmark that makes confounding measurable, principled estimation, correct theoretical results, and thorough appendix diagnostics. The reviewer’s concerns mainly relate to the experimental validation, the interpretation of our estimand, and other minor technical details.

Concern 1: Does our improvement over the baselines come from the attributes ZdZ^d?

No. Our improvement remains when the baselines also receive ZdZ^d, and when we read ZdZ^d from pixels.

(a) Baselines given the same ZdZ^d

"These are exactly the Setting-3 confounders, and no baseline receives them (Tab. 6). Without ZdZ_d, the method’s Setting-3 MAE is 0.169 (Tab. 8), identical to the strongest baseline …"

To test whether our improvement comes from ZdZ^d, we give ZdZ^d to every baseline that can take it as input. We report the average MAE ± SEM over 31 variants (with 10 in Setting 3). Results show that our method still outperforms the baselines. This suggests that the improvement comes from our methodological design, not from ZdZ^d.

MethodSetting 3, without ZdZ^dSetting 3, with ZdZ^dOverall, with ZdZ^d
Qwen2.5-VL + LoRA0.214 ± 0.0360.192 ± 0.0310.126 ± 0.017
V-JEPA 2 + head0.146 ± 0.0280.136 ± 0.0260.101 ± 0.011
GPT-5.6 + retrieval$^\dagger$0.227 ± 0.0370.100 ± 0.0150.076 ± 0.007
Gemini-3-Flash + retrieval$^\dagger$0.211 ± 0.0380.097 ± 0.0160.069 ± 0.007
Ours0.173 ± 0.0370.065 ± 0.0100.046 ± 0.005

†^\daggerWe evaluate these methods on 1,762 random tasks, where our method obtains 0.069 (Setting 3) and 0.047 (overall).

(b) ZdZ^d read from pixels

"The method uses discrete preconditions ZdZ_d read from the simulator state (Sec. 5.1). … No results with pixel-derived ZdZ_d are reported."

To test whether our method relies on the simulator state, we read ZdZ^d from the first frame with Gemini-3-Flash. We report the average MAE ± SEM over 31 variants. Results show that our error barely changes, suggesting that we can obtain ZdZ^d from pixels in practice.

ZdZ^dSetting 3Overall
Simulator0.065 ± 0.0100.046 ± 0.005
Gemini-3-Flash, from the first frame0.067 ± 0.0100.046 ± 0.005

Concern 2: Does the estimator itself improve accuracy?

Yes. With the data and inputs held fixed, our estimator lowers the error in each comparison below.

(a) Comparison with OR and IPW

"With correctly specified nuisances, OR, IPW and the full estimator perform the same (0.047/0.045/0.048, Tab. 7)."

To test whether our estimator improves on OR and IPW, we compare the three at growing sample sizes. We simulate the data from TongSim-CF. We report the average RMSE ± SEM over 31 variants. Results show that our estimator achieves the lowest RMSE as the sample size grows, suggesting its efficiency advantage.

nnORIPWOurs
6000.059 ± 0.0040.079 ± 0.0070.058 ± 0.004
2,4000.032 ± 0.0020.038 ± 0.0030.027 ± 0.002
9,6000.017 ± 0.0010.018 ± 0.0020.013 ± 0.001

(b) The correction in Table 15

"The gain from adding the correction to foundation-model predictors (Tab. 15) again relies on weights fitted with ZdZ_d."

As shown in Concern 1, our improvement does not come from ZdZ^d. The same holds for this table (Table 18 of the revised manuscript). When we also give ZdZ^d to the outcome model, our correction still cuts its overall MAE by more than half (Appendix K.5).

(c) CATE learners on the same inputs

"No standard adjusted regression or CATE learner on the same features is compared."

To compare with CATE learners, we run them [1, 2] on exactly our inputs. The S-learner among them is the standard adjusted regression. Unlike our estimand, their estimand does not condition on the observed outcome yy:

CATE learners: E[Ya′∣A=a,Aˉ=aˉ,V=v],ours: E[Ya′∣Y=y,A=a,Aˉ=aˉ,V=v].\text{CATE learners: } E[Y_{a'}\mid A=a,\bar A=\bar a,V=v],\qquad \text{ours: } E[Y_{a'}\mid Y=y,A=a,\bar A=\bar a,V=v].

We report the average MAE ± SEM over 31 variants. Results show that our method has the lowest overall error. This suggests that its improvement comes from the estimator, not from the inputs.

MethodSetting 1Setting 2Setting 3Overall
Ours0.020 ± 0.0010.039 ± 0.0050.065 ± 0.0100.046 ± 0.005
S-learner0.020 ± 0.0050.051 ± 0.0060.062 ± 0.0060.052 ± 0.004
T-learner0.020 ± 0.0010.065 ± 0.0090.078 ± 0.0110.065 ± 0.007
X-learner0.086 ± 0.0030.100 ± 0.0060.093 ± 0.0050.097 ± 0.004
DR-learner0.024 ± 0.0020.058 ± 0.0050.058 ± 0.0060.055 ± 0.004

References

[1] Künzel et al., "Metalearners for estimating heterogeneous treatment effects using machine learning," PNAS 2019.
[2] Kennedy, "Towards optimal doubly robust estimation of heterogeneous causal effects," Electronic Journal of Statistics 2023.

AuthorsOpenReview ↗

Response to Reviewer fYDj (2/3): Concerns 3–4 and Questions 1–2

Concern 3: Real data with concurrent actions and confounding

"Physics 101 has neither concurrent actions nor confounding …"

We add three real datasets with concurrent actions and confounding. They are robotic grasping [3], robotic scooping [4], and C. elegans [5]. Specifically, the confounder in robotic grasping is the item's weight. In robotic scooping, it is the terrain. In C. elegans, it is the worm's movement. We describe the datasets in Sec. 5.3 and Appendix J of the revised manuscript. We report the average MAE ± SEM over units (Table 2 of the revised manuscript). Results show that our method has the lowest error on all three datasets. This suggests that it attributes outcomes accurately when confounders exist.

MethodGraspingScoopingC. elegans
Ours0.124 ± 0.0140.126 ± 0.0060.053 ± 0.002
GPT-5.6, five examples0.237 ± 0.0280.327 ± 0.009$^*$0.191 ± 0.006
Gemini-3-Flash, five examples0.271 ± 0.0310.296 ± 0.009$^*$0.207 ± 0.007
GPT-5.6 + retrieval0.227 ± 0.0260.189 ± 0.007$^*$0.113 ± 0.004
Gemini-3-Flash + retrieval0.250 ± 0.0180.146 ± 0.006$^*$0.129 ± 0.003
LLCP0.203 ± 0.029$^\ddagger$0.173 ± 0.0070.124 ± 0.003
Fire-Zhu0.346 ± 0.0290.218 ± 0.0070.181 ± 0.006
Qwen2.5-VL + LoRA0.362 ± 0.0270.143 ± 0.0070.146 ± 0.003
V-JEPA 2 + head0.331 ± 0.0480.149 ± 0.0070.151 ± 0.004
S-learner0.225 ± 0.0270.154 ± 0.0060.153 ± 0.004
T-learner0.234 ± 0.0290.158 ± 0.0060.155 ± 0.003
DR-learner0.259 ± 0.0300.177 ± 0.0050.151 ± 0.003

∗^*We evaluate these methods on 500 random queries per repetition; our method obtains 0.128 ± 0.007 on them. ‡^\ddaggerLLCP needs a frame after the action, which only 17 grasping units have. We evaluate LLCP on these units, where our method obtains 0.156 ± 0.016.

Concern 4: Does our estimand depend on the observed outcome yy?

Yes. Our estimand averages over the scene state ZZ with weights that depend on yy.

(a) Whether the estimand depends on yy

"Under Ass. 1–5 the pointwise target equals E[Y∣A=a′,Aˉ=aˉ,Z=g(v)]E[Y\mid A=a',\bar A=\bar a,Z=g(v)], which does not depend on yy … The task is therefore conditional interventional (CATE-type) estimation."

Even after conditioning on ZZ, the outcome event should also be included in the conditioning of the estimand. This is because we only assume ZZ to be a sufficient representation of the video (i.e., ((Yc)c,A,Aˉ)⊥ ⁣ ⁣ ⁣⊥V∣Z((Y_c)_c,A,\bar A)\perp\!\!\!\perp V\mid Z). The frame may therefore show ZZ only in part. We thus remove the notation Z=g(V)Z=g(V) in the revised manuscript (Sec. 2). Without this notation, the estimand becomes Ψ(a′;y,a,aˉ,v)=E[Ya′∣Y=y,A=a,Aˉ=aˉ,V=v]=∫μ(z;a′,aˉ) P(dz∣y,a,aˉ,v)\Psi(a';y,a,\bar a,v) = E[Y_{a'}\mid Y=y,A=a,\bar A=\bar a,V=v] = \int\mu(z;a',\bar a)\,P(dz\mid y,a,\bar a,v), where μ(z;a′,aˉ)=E[Y∣A=a′,Aˉ=aˉ,Z=z]\mu(z;a',\bar a)=E[Y\mid A=a',\bar A=\bar a,Z=z]. The weight P(dz∣y,a,aˉ,v)P(dz\mid y,a,\bar a,v) depends on yy.

(b) CATE methods

"Yet CATE methods are neither discussed nor compared."

We compare our method with CATE learners in Concern 2(c) above. Our method has the lowest overall error there. We describe their protocol in Appendix K.8 of the revised manuscript.

About other technical details:

Question 1. "The method uses all videos, while the finetuned baselines use half."

To conduct a fair comparison, we now let all methods train on all videos. We report the average MAE ± SEM over 31 variants. Results show that our method remains the most accurate. We report these results in Table 1 of the revised manuscript.

MethodTrained on half of the videosTrained on all videos
V-JEPA 2 + head0.145 ± 0.0120.104 ± 0.012
Qwen2.5-VL + LoRA0.142 ± 0.0170.137 ± 0.018
Ours–0.046 ± 0.005

Question 2. "VLMs are evaluated on only 53 tasks."

To obtain a more reliable comparison, we now evaluate the VLMs on 1,762 random tasks instead of 53. We report the results in Table 1 of the revised manuscript. Results show that the VLMs remain far less accurate than our method on the same tasks.

References

[3] Wang et al., "Multimodal grasp data set: A novel visual–tactile data set for robotic manipulation," International Journal of Advanced Robotic Systems 2019.
[4] Zhu et al., "Few-shot adaptation for manipulating granular materials under domain shift," RSS 2023.
[5] Liu et al., "A high-throughput method to deliver targeted optogenetic stimulation to moving C. elegans populations," PLoS Biology 2022.

AuthorsOpenReview ↗

Response to Reviewer fYDj (3/3): Questions 3–6

Question 3. "The bandwidth is justified on test tasks (Tab. 11)."

We would like to clarify that we fix the bandwidth rather than tune it on the test tasks. We only use this table (Table 14 of the revised manuscript) to report the sensitivity to hh. To check that our results do not rely on test labels, we also choose hh by cross-validation on the observed outcomes. This choice requires no counterfactual labels. Results show that it gives the same error as the fixed hh.

Choice of hhOverall MAE
Fixed0.046 ± 0.005
Cross-validation on observed outcomes0.045 ± 0.005

Question 4. "A misspecified outcome classifier therefore biases both terms, so the IPW route (Sec. 4.1) does not protect against outcome misspecification."

We apologize for the potential misunderstanding caused by the misspecification table. We did not intend to claim double robustness; our goal was to highlight the efficiency of the proposed estimator. Accordingly, we establish an efficiency result in Theorem 5. We have clarified this point in the revised manuscript (lines 260–262, 278–280 and 324–326). We have also replaced the misspecification table with a comparison table in Appendix K.1.

Question 5. "Efficiency is relative to a model that ignores Ass. 5. … It is efficient only if the kernel depends on VV through ZZ alone, and then rr is known exactly and needs no Monte-Carlo proxy."

We thank the reviewer for pointing this out. We agree. We have added the efficiency result under Assumption 5 to the revised manuscript (Prop. 6, Appendix F.1).

Question 6. "The guarantees concern a fixed-bandwidth kernel on raw frames. The implementation instead uses PCA features."

To close the gap between our theory and implementation, we now compute the kernel directly on the frame VV in all TongSim-CF experiments (Sec. 4.3). We report the new results in the revised manuscript. Results show that our errors barely change. This suggests that our conclusions hold with the kernel on the frame.

AuthorsOpenReview ↗

Follow-up Before the Discussion Deadline

Dear Reviewer fYDj,

With only a few hours remaining in the discussion period, we would like to kindly follow up on our response and manuscript revisions. As you suggested, we have added more experiments: baselines given the same ZdZ^d, comparisons with OR, IPW, and CATE learners, and three real datasets with concurrent actions and confounding. We have also added an efficiency result under Assumption 5 (Prop. 6). Details are in our response to Concerns 1–4 and Questions 1–6.

We would greatly appreciate your feedback on whether these results address your main concerns, particularly any issues that remain unresolved or require further clarification. Your feedback would help us address outstanding questions before the discussion closes.

Thank you again for your time and consideration.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading and constructive comments, and for recognizing our benchmark that makes confounding measurable (fYDj), principled estimation of a clear causal estimand (HScN), and the importance of separating correlated VLM priors from counterfactual attribution (CeBS).

The reviewers share three questions:

(i) Does our improvement over the baselines come from the scene attribute ZdZ^d?

(ii) How does our estimand differ from CATE estimation?

(iii) Does our method also work on real data with concurrent actions and confounding?

We give brief responses to these questions here. We address each point in detail in our reply to the corresponding reviewer.

(i) No. The improvement comes from our methodological design, rather than from scene attributes. (Details: fYDj Concern 1, HScN Concern 1, CeBS Concern 1.)

(ii) Our estimand conditions on the observed outcome in addition to the covariates, whereas the CATE conditions only on the covariates. This conditioning gives our estimand its attribution meaning. (Details: fYDj Concerns 4 and 2(c).)

(iii) Yes. Our method consistently outperforms the baselines across datasets. (Details: fYDj Concern 3, HScN Concern 2(b), CeBS Concern 2.)

We have uploaded a revised manuscript, with changes marked in blue. The main changes:

WhereWhat changedRaised by
Sec. 2; Appendix CZZ and its sufficiency are introduced before the assumptions; Assumptions 2–4 are stated given ZZ; the estimand as a weighted average over scene states, and when it depends on yyfYDj
Secs. 4.1–4.2; Appendix K.1No double-robustness claim; a bias and variance comparison replaces the misspecification tablefYDj
Sec. 4.3; Appendix F.1Efficiency under the restriction of Assumption 5 (new Prop. 6); the kernel now computed on the raw frame in all TongSim-CF experimentsfYDj
Table 1All methods that learn from factual outcomes use all videos; VLMs on 1,762 tasks; CATE learners addedall
Sec. 5.3; Fig. 2; Table 2; Appendix JReal data: Physics 101 and three datasets with concurrent actions and confoundingall
Appendix K.7Baselines given ZdZ^d; ZdZ^d read from pixels; general-purpose featuresall
Appendix K.8Protocol of the CATE learnersfYDj, HScN
Appendix K.3Bandwidth chosen without counterfactual labels; the number of videos per environmentfYDj, CeBS
Appendix K.5The correction with V-JEPA 2 + ZdZ^d as the outcome modelfYDj
Appendix I.3Physics 101 results and per-object comparisonfYDj, HScN

Due to the space limit, we moved the details of the sensitivity study from Sec. 5.5 to Appendix K.6. We also merged the limitations into Sec. 6.

AuthorsOpenReview ↗

Summary of Revisions and Additional Evidence

We thank the Area Chair and reviewers for their thoughtful feedback. We have conducted additional experiments and analyses to address the concerns regarding the source of our improvement, our estimand, and real-data evidence. We summarize our responses below.

Source of the improvement (Reviewers fYDj, HScN, CeBS). We gave the scene attributes ZdZ^d to all baselines that can take them, and we also read ZdZ^d from pixels. Our method remains the most accurate, so the improvement comes from our methodological design rather than from ZdZ^d.

Estimand and estimator (Reviewers fYDj, HScN). We clarified that our estimand conditions on the observed outcome, whereas the CATE does not. We added comparisons with CATE learners on the same inputs and with OR and IPW, where our estimator achieves the lowest error.

Real data with concurrent actions and confounding (Reviewers fYDj, HScN, CeBS). We added three real datasets: robotic grasping, robotic scooping, and C. elegans. Our method outperforms the baselines on all three.

Theory and evaluation details (Reviewers fYDj, CeBS). We added an efficiency result under Assumption 5 (Prop. 6) and computed the kernel on the raw frame. We also trained all methods on all videos, evaluated the VLMs on more tasks, and tested fewer videos per environment.

We have provided detailed point-by-point responses and uploaded a revised manuscript with changes marked in blue. We also followed up with each reviewer; however, we have not yet received further feedback or requests for clarification. We would welcome any additional questions during the remaining discussion period and hope the revised manuscript and additional evidence adequately address the concerns raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.