- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the out-of-distribution generalization limitations of Supervised Causal Learning (SCL). The authors show that existing SCL methods suffer significant performance degradation on real-world data, under distribution shifts, and on unseen combinations of causal components. To address these challenges, they propose TTT-SCL, which dynamically constructs training data tailored to each test dataset through score-guided graph search and causal mechanism fitting. Experiments demonstrate competitive performance across synthetic, pseudo-real, and real-world benchmarks.
Critical strengths
-
The paper provides an empirical analysis of the generalization limitations of existing SCL methods, including performance degradation under distribution shifts and failures on unseen combinations of causal components. These findings provide clear motivation for the task.
-
The proposed TTT-SCL framework offers a promising approach for addressing the out-of-distribution generalization challenges of existing SCL methods by dynamically constructing instance-specific training data at test time.
Critical weaknesses
-
The computation cost seems to be very high. It requires graph search, mechanism fitting, and retraining a supervised model for each test dataset, which raises concerns about its scalability and practical applicability to larger causal discovery problems.
-
The paper does not fully isolate the contribution of test-time supervised learning from the score-guided graph search. Although it compares TTT-SCL with graph voting baselines, these methods use different candidate graph sets. It doesn't isolate the contribution of the supervised learner. A comparison using identical candidate graphs would provide stronger evidence for the independent contribution of the supervised learner.
Questions for authors
-
Could the authors provide a comparison between TTT-SCL and graph voting baselines using identical candidate graph sets? This would help clarify how much of the performance improvement comes from the supervised learner itself.
-
Given that the synthetic training data are generated from mechanisms fitted to the observed test dataset, could the authors clarify why this process should improve OOD robustness? Since the generated data are still constrained by the information available in the original observations and the assumptions of the fitted mechanisms, it is unclear how the method can handle causal mechanisms or distributional variations that are not adequately captured by the observed data.
3 comments
Thank you for the clarification and additional experiments. The new ablation study addresses my concern about isolating the contribution of the supervised learner. However, my main concern regarding OOD robustness remains unanswered. While the method may help the model better utilize information from the observed test data, it is still unclear to me why this process should consistently improve OOD robustness. Since the synthetic training data are constructed from the original observations and candidate graph assumptions, how does the method ensure that the learned relationships reflect the underlying causal structure rather than environment-specific statistical patterns that may not generalize under distribution shifts? The current explanation may provide a reasonable intuition for test-time adaptation, but I believe stronger justification or more direct experimental evidence is needed to support the claimed OOD robustness. Therefore, I will maintain the current score.
We greatly appreciate the reviewer's further description of their confusion regarding how to resolve OOD, which gives us the opportunity to explain further. We previously misunderstood your question; the core issue is that there are actually two different OODs within this topic . We will now explain this in more detail to hopefully clear up your confusion.
(1) First, we want to clarify the meaning of OOD robustness in this paper. TTT-SCL constructs training data, trains a learner, and predicts its causal structure separately for each current test dataset; when faced with another dataset, we repeat this process. Therefore, we do not require the currently generated training data or trained learner to be suitable for other testing environments, which is precisely the meaning of instance-specific. Compared to static SCL, TTT-SCL changes the way of relying on a fixed pre-trained distribution to cover subsequent test tasks from a methodological design perspective.
(2) Secondly, we understand your concerns and the OOD in your comments: even when generating training data for the current test data, a mismatch may still exist between the generated distribution and the true test distribution. This is mainly affected by the quality and coverage of the candidate graph, the ability of SIM to fit mechanisms and noise, and the limitations of limited observation data.
We do not require the two distributions to be completely identical, nor do we believe that distribution consistency is sufficient to determine the true causal graph. The role of SIM is to construct training data generated from explicit candidate causal graphs while incorporating the statistical characteristics of the current test data, so that the mapping between the SCL data and the graph structure is not overly influenced by the distribution. What needs to be preserved here is the statistical information that helps distinguish causal structures, not just general distributional similarity. The existing experiments in Table 2 of our paper, showing that TTT-SCL outperforms the static SCL baseline AVICI on both Sachs and SynTReN, provide empirical support for this approach. We do not know the mechanisms of these two test data, and the generated training data has a certain distribution mismatch with the test data, but still achieves good practical performance.
Furthermore, the two main technical aspects causing the mismatch can be improved within the TTT-SCL framework: candidate graph construction can employ better initialization and search methods, and SIM can use more flexible mechanisms and noise models that are compatible with the causal assumptions used. These improvements provide ways to further reduce the mismatch between the generated training and test distributions.
We are very grateful for the reviewers' efforts, and the following are our responses to the questions.
Q1/W2.Isolating the contribution of the supervised learner.
We agree that if the voting baseline uses a different set of candidate graphs than Full TTT-SCL, performance differences may arise from both candidate selection and subsequent learning, thus failing to adequately account for additional contributions during the learning phase. To address this, we supplement this with a control experiment using a fixed set of candidate graphs: based on the same batch of candidate graphs actually used in Full TTT-SCL, predictions are obtained using the following three methods:
Voting: Equal-weighted averaging of the adjacency matrices within the same candidate set.
Score-weighted voting: Weighted averaging of the same candidate set based on graph scores to obtain continuous edge scores.
Full TTT-SCL: Using the same candidate set, a supervised learner is trained on training data constructed using SIM and used to predict on test data.
Table R2. Ablation experiments with the same candidate graph set.
In all five settings, Full TTT-SCL outperformed both voting baselines. Its AUROC improved by 1.2, 4.4, 1.7, 12.2, and 21.5 percentage points relative to the best-performing voting methods in each setting, with larger improvements on Sachs and SynTReN. The use of the same candidate set by all three methods supports the additional value of the supervised learning phase.
Q2.Why TTT-SCL improve OOD robustness
The potential benefit of TTT-SCL comes from adapting the learner’s training distribution to the current test dataset. A distinction is that useful statistical information may be present in the test dataset, while a model trained on a fixed distribution may not have learned how to use that information under the conditions encountered at test time.
Because reliable causal graph annotations are scarce, supervised causal learning relies heavily on synthetic training data. The synthetic generation process therefore determines which graph structures, mechanisms, and noise characteristics the learner encounters during training. When a test dataset differs from this training distribution, the learned mapping from observations to graphs may perform poorly.
TTT-SCL addresses this mismatch by using the current observations to construct the training tasks themselves. For each test dataset, we use its observations to guide candidate graph construction, fit mechanisms under these candidates, and generate corresponding graph-labeled datasets. We then train a newly initialized SCL model on these tasks and apply it to the observed dataset. TTT-SCL provides a dataset-specific training strategy within SCL, it reduces reliance on a single, fixed pretraining distribution anticipating each future task.