- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies causal treatment effect estimation on natural language data under covariate and posterior shift. The authors introduce the TRACE-Bench dataset, which adapts an existing semi-synthetic moral-judgement dataset for this task by assigning posts into topic-based source and target domains and simulating treatments. To estimate the ATE, the authors propose a four-stage causal pipeline consisting of (1) source learning, (2) shift diagnosis, (3) target adaptation, and (4) AIPW estimation. Two implementations of this pipeline are presented, termed TRACE-AGENT, which uses agentic LLM workflows, and TRACE-NN, which uses non-LLM architectures. Experiments on the proposed benchmark demonstrate that TRACE-AGENT consistently improves estimation accuracy over generic LLM agent baselines across six foundation models, while TRACE-NN is also competitive and outperforms existing statistical ML baselines. The authors lastly establish the double robustness property of the estimation procedure.
Critical strengths
Significance and methodological soundness. The paper addresses the important problem of causal effect estimation from natural language data and proposes a natural four-stage causal pipeline for handling distribution shifts. Most of the stages are sound and well-motivated (though I have some minor concerns discussed in the Weaknesses). The pipeline is also largely model-agnostic and the authors provide two realizations using an agentic LLM workflow and a non-LLM architecture that both perform well empirically.
Empirical gains. The empirical gains are consistent and substantial compared to most LLM baselines (tested across six foundation models) for TRACE-AGENT and statistical methods for TRACE-NN. The paper also demonstrates that source learning slightly improves performance for TRACE-AGENT compared to target-only context learning.
Critical weaknesses
Transfer learning. The experiments do not convincingly show the benefit of source-to-target transfer learning. Table 2 provides partial evidence by demonstrating target-only TRACE-AGENT underperforms the full pipeline, but this analysis is absent for TRACE-NN. The paper also does not consider a simple pooled-data baseline in which the model is trained jointly on source and target data.
Missing statistical baselines. Building on the previous concern, the statistical ML evaluation lacks several important comparisons. In particular, the paper should consider text-based causal methods (e.g., CausalBERT) under both target-only and pooled source-target training. It would also be informative to compare these methods against TRACE in a setting without distribution shift, to isolate the benefits of nuisance estimation with text covariates from those of the transfer stages. As it stands, most of the statistical ML baselines (with the exception of -TCL) appear to be variants of TRACE-NN.
Shift diagnosis procedure. Several components of the shift diagnosis step for TRACE-AGENT are unexplained and seem ad hoc. For example, this step only compares the source-learned model with a constant baseline, which is not a test for distribution shift, and any reused source predictions may still be biased for ATE estimation in the target distribution. The value of also seems to be chosen arbitrarily with no sensitivity analysis. The same critique applies to choosing for TRACE-NN.
I think the overall approach is reasonable, but my main concerns are that the experiments do not show whether the gains come from the source-to-target transfer, better nuisance modeling of the text covariates, or some combination of the two. I am also not fully convinced that the shift diagnosis and correction steps are principled enough to justify the transfer component.
Questions for authors
- Could the authors provide examples of learned TRACE-AGENT contexts during the source learning and target adaptation stages? How does the model perform if these are replaced by simple baseline contexts (possibly) containing topic-stratified base rates?
- TRACE-NN is only evaluated with bag-of-words features and text embeddings from Qwen3. Could the authors also test it using hidden-layer representations from the same LLMs used in TRACE-AGENT?
- Can the authors explain why several of the statistical ML baselines actually perform worse than the simple difference-in-means estimator in the target domain?
- Only RMSE of the ATE estimate is reported. Since paired potential outcomes are available, could the authors also report PEHE?
- The AIPW stage of the pipeline is not tested. Since AIPW can be sensitive to propensity weights, how do the models perform using an outcome-regression or IPW estimator instead?
- Table 2 states that the target-only model is trained with the same context optimization budget as TRACE-AGENT. Does this budget refer only to stage 3 of the pipeline, or stages 1 and 3 combined?
- See also weaknesses.
13 comments
Response to weakness 1
Comments and weaknesses
R3-W1: Evidence for source-to-target transfer
Reviewer comment:
Transfer learning. The experiments do not convincingly show the benefit of source-to-target transfer learning. Table 2 provides partial evidence by demonstrating target-only TRACE-AGENT underperforms the full pipeline, but this analysis is absent for TRACE-NN. The paper also does not consider a simple pooled-data baseline in which the model is trained jointly on source and target data.
Response:
Our original paper reports TRACE-NN’s source-only and target-only comparisons in Table 1. Appendix D.3 explicitly defines FIDDLE (source-only) and FIDDLE (target-only) as ablations using the architectures of the corresponding TRACE-NN stages. Using the same text representation, TRACE-NN achieves an RMSE of 0.0327, compared with 0.0457 and 0.0368 for these two variants, supporting the benefit of source-to-target transfer learning. We also report these comparisons with Qwen3 embeddings in Appendix E.2/Table 5, in simulation in Appendix E.3/Figure 5(b), and on the News benchmark in Appendix E.4/Table 6.
We additionally evaluate baselines trained jointly on labeled source and target records (i.e., simple pooled-data baselines). With bag-of-words features, pooled FIDDLE, TARNet, and DragonNet achieve RMSEs of 0.0352, 0.0494, and 0.0481, respectively. For Qwen3-4B, pooled context learning yields an RMSE of 0.0690, compared with 0.0381 for TRACE-AGENT. Beyond the lower RMSEs achieved by TRACE-NN and TRACE-AGENT in these comparisons, TRACE supports reusing learned source contexts or nuisance models when the labeled source training records are unavailable. Pooling requires access to those records during target estimation.
The evidence of transfer learning benefits on TRACE-Bench is summarized below (Lower is better):
| Method | Text representation | Training setting | ATE RMSE |
|---|---|---|---|
| FIDDLE | Bag-of-words | Source only | 0.0457 |
| FIDDLE | Bag-of-words | Target only | 0.0368 |
| FIDDLE | Bag-of-words | Pooled source and target | 0.0352 |
| TARNet | Bag-of-words | Pooled source and target | 0.0494 |
| DragonNet | Bag-of-words | Pooled source and target | 0.0481 |
| TRACE-NN | Bag-of-words | Source-to-target transfer | 0.0327 |
| TRACE-AGENT, Qwen3-4B | Raw text | Target only | 0.0493 |
| Pooled agent, Qwen3-4B | Raw text | Pooled source and target | 0.0690 |
| TRACE-AGENT, Qwen3-4B | Raw text | Source-to-target transfer | 0.0381 |
Response to weakness 2
R3-W2: Missing statistical and no-shift baselines
Reviewer comment:
Missing statistical baselines. Building on the previous concern, the statistical ML evaluation lacks several important comparisons. In particular, the paper should consider text-based causal methods (e.g., CausalBERT) under both target-only and pooled source-target training. It would also be informative to compare these methods against TRACE in a setting without distribution shift, to isolate the benefits of nuisance estimation with text covariates from those of the transfer stages. As it stands, most of the statistical ML baselines (with the exception of -TCL) appear to be variants of TRACE-NN.
Response:
CausalBERT comparison. We add CausalBERT under both target-only and pooled source–target training, following Veitch et al. (2020) [1], which adapts BERT to learn document embeddings that preserve information needed for causal effect estimation. This adds a text-based causal method developed independently of TRACE-NN. The table compares it with TRACE-NN using bag-of-words features and Qwen3 embeddings, and with TRACE-AGENT using raw text.
| Method | Text representation | Training setting | ATE RMSE |
|---|---|---|---|
| CausalBERT | Raw text | Target only | 0.0788 |
| CausalBERT | Raw text | Pooled source and target | 0.0515 |
| TRACE-NN | Bag-of-words | Source-to-target transfer | 0.0327 |
| TRACE-NN | Qwen3 embeddings | Source-to-target transfer | 0.0348 |
| TRACE-AGENT, six backbones | Raw text | Source-to-target transfer | [0.0246, 0.0381] |
Single-domain comparison without source transfer. To isolate nuisance estimation from cross-domain transfer, we compare methods trained only on the same target population within each task. We include CausalBERT, TARNet, DragonNet, FIDDLE (the single-domain counterpart of TRACE-NN), and TRACE-AGENT. Removing source information from every method allows this comparison to assess nuisance estimation independently of transfer. The results support the effectiveness of our nuisance-learning approaches even when cross-domain transfer is absent.
All methods in the following table are trained using target data only.
| Method | Text representation | ATE RMSE |
|---|---|---|
| CausalBERT | Raw text | 0.0788 |
| FIDDLE (target-only) | Bag-of-words | 0.0368 |
| TARNet target-only | Bag-of-words | 0.0392 |
| DragonNet target-only | Bag-of-words | 0.0594 |
| TRACE-AGENT target-only, DeepSeek-V4-Flash | Raw text | 0.0270 |
| TRACE-AGENT target-only, Qwen3.6-27B | Raw text | 0.0276 |
| TRACE-AGENT target-only, GPT-OSS-120B | Raw text | 0.0326 |
| TRACE-AGENT target-only, Gemma-4-12B | Raw text | 0.0362 |
| TRACE-AGENT target-only, GPT-OSS-20B | Raw text | 0.0363 |
| TRACE-AGENT target-only, Qwen3-4B | Raw text | 0.0493 |
In the revised paper, we have added CausalBERT implementation details in Appendix E.3, the target-only comparisons in Table 7, and the pooled comparisons in Table 9. Section 4.3 also points to these additions.
Key ideas of responses
Key ideas of responses
We thank the reviewer for recognizing TRACE as a unified causal estimation framework that accommodates both LLMs and neural networks: a largely model-agnostic, four-stage pipeline for causal effect estimation from natural language under distribution shift. TRACE-AGENT and TRACE-NN instantiate the same four-stage pipeline for handling distribution shift, demonstrating the framework's largely model-agnostic design. The reviewer also highlights consistent and substantial empirical gains from both implementations, with TRACE-AGENT outperforming most LLM baselines across six foundation models and TRACE-NN outperforming statistical methods.