acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Benchmarking Agentic Causal Estimation from Text under Covariate and Posterior Shift

Abstract

Average treatment effect (ATE) estimation is central to causal inference, but extending it to settings with open-ended text covariates is not straightforward. An estimator must connect information in individual narratives to a target-population effect, even when source and target populations differ in their covariates and causal mechanisms. Large language models (LLMs) can interpret text and perform mathematical reasoning, but those abilities alone do not ensure reliable effect estimates. To study this gap, we introduce TRACE-Bench, an end-to-end benchmark built from social narratives for evaluating ATE estimation across shifted source and target domains. Generic LLM agents, including direct estimation and planning-based approaches, are unreliable on this benchmark, while structured statistical estimators remain competitive. We therefore propose TRACE, a shift-adaptive framework that retains transferable source information and adapts the mechanisms that change. TRACE-AGENT learns and adapts natural-language contexts for causal estimation before doubly robust aggregation. TRACE-NN is an LLM-free, factor-augmented neural realization of the same framework. Across foundation models, TRACE-AGENT consistently improves ATE estimation over generic agent baselines, while TRACE-NN remains competitive with substantially larger LLMs. We finally prove that the proposed TARCE is doubly robust, which will give consistent estimation of ATE as long as either the propensity or outcome model is consistently estimated.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
causal inference, average treatment effect estimation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vR2qOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies causal treatment effect estimation on natural language data under covariate and posterior shift. The authors introduce the TRACE-Bench dataset, which adapts an existing semi-synthetic moral-judgement dataset for this task by assigning posts into topic-based source and target domains and simulating treatments. To estimate the ATE, the authors propose a four-stage causal pipeline consisting of (1) source learning, (2) shift diagnosis, (3) target adaptation, and (4) AIPW estimation. Two implementations of this pipeline are presented, termed TRACE-AGENT, which uses agentic LLM workflows, and TRACE-NN, which uses non-LLM architectures. Experiments on the proposed benchmark demonstrate that TRACE-AGENT consistently improves estimation accuracy over generic LLM agent baselines across six foundation models, while TRACE-NN is also competitive and outperforms existing statistical ML baselines. The authors lastly establish the double robustness property of the estimation procedure.

Critical strengths

Significance and methodological soundness. The paper addresses the important problem of causal effect estimation from natural language data and proposes a natural four-stage causal pipeline for handling distribution shifts. Most of the stages are sound and well-motivated (though I have some minor concerns discussed in the Weaknesses). The pipeline is also largely model-agnostic and the authors provide two realizations using an agentic LLM workflow and a non-LLM architecture that both perform well empirically.

Empirical gains. The empirical gains are consistent and substantial compared to most LLM baselines (tested across six foundation models) for TRACE-AGENT and statistical methods for TRACE-NN. The paper also demonstrates that source learning slightly improves performance for TRACE-AGENT compared to target-only context learning.

Critical weaknesses

Transfer learning. The experiments do not convincingly show the benefit of source-to-target transfer learning. Table 2 provides partial evidence by demonstrating target-only TRACE-AGENT underperforms the full pipeline, but this analysis is absent for TRACE-NN. The paper also does not consider a simple pooled-data baseline in which the model is trained jointly on source and target data.

Missing statistical baselines. Building on the previous concern, the statistical ML evaluation lacks several important comparisons. In particular, the paper should consider text-based causal methods (e.g., CausalBERT) under both target-only and pooled source-target training. It would also be informative to compare these methods against TRACE in a setting without distribution shift, to isolate the benefits of nuisance estimation with text covariates from those of the transfer stages. As it stands, most of the statistical ML baselines (with the exception of ℓ1\ell_1-TCL) appear to be variants of TRACE-NN.

Shift diagnosis procedure. Several components of the shift diagnosis step for TRACE-AGENT are unexplained and seem ad hoc. For example, this step only compares the source-learned model with a constant baseline, which is not a test for distribution shift, and any reused source predictions may still be biased for ATE estimation in the target distribution. The value of δ=0.01\delta = 0.01 also seems to be chosen arbitrarily with no sensitivity analysis. The same critique applies to choosing δx\delta_x for TRACE-NN.

I think the overall approach is reasonable, but my main concerns are that the experiments do not show whether the gains come from the source-to-target transfer, better nuisance modeling of the text covariates, or some combination of the two. I am also not fully convinced that the shift diagnosis and correction steps are principled enough to justify the transfer component.

Questions for authors
  1. Could the authors provide examples of learned TRACE-AGENT contexts during the source learning and target adaptation stages? How does the model perform if these are replaced by simple baseline contexts (possibly) containing topic-stratified base rates?
  2. TRACE-NN is only evaluated with bag-of-words features and text embeddings from Qwen3. Could the authors also test it using hidden-layer representations from the same LLMs used in TRACE-AGENT?
  3. Can the authors explain why several of the statistical ML baselines actually perform worse than the simple difference-in-means estimator in the target domain?
  4. Only RMSE of the ATE estimate is reported. Since paired potential outcomes are available, could the authors also report PEHE?
  5. The AIPW stage of the pipeline is not tested. Since AIPW can be sensitive to propensity weights, how do the models perform using an outcome-regression or IPW estimator instead?
  6. Table 2 states that the target-only model is trained with the same context optimization budget as TRACE-AGENT. Does this budget refer only to stage 3 of the pipeline, or stages 1 and 3 combined?
  7. See also weaknesses.
13 comments
AuthorsOpenReview ↗

Key ideas of responses

Key ideas of responses

We thank the reviewer for recognizing TRACE as a unified causal estimation framework that accommodates both LLMs and neural networks: a largely model-agnostic, four-stage pipeline for causal effect estimation from natural language under distribution shift. TRACE-AGENT and TRACE-NN instantiate the same four-stage pipeline for handling distribution shift, demonstrating the framework's largely model-agnostic design. The reviewer also highlights consistent and substantial empirical gains from both implementations, with TRACE-AGENT outperforming most LLM baselines across six foundation models and TRACE-NN outperforming statistical methods.

  • W1 — Transfer benefits. This analysis is in Table 1 of the original paper, where TRACE-NN achieves an RMSE of 0.0327 compared with 0.0457 for the source-only ablation and 0.0368 for the target-only ablation. The pattern repeats with Qwen3 embeddings (Table 5), in simulation (Figure 5b), and on News (Table 15). Added pooled-data comparisons further support target adaptation for both TRACE implementations (Table 9).
  • W2 — Additional baselines. We added CausalBERT under target-only and pooled training, with lower RMSE achieved by TRACE-NN and all six TRACE-AGENT backbones. The added target-only comparisons with CausalBERT, TARNet, and DragonNet also support the effectiveness of our nuisance learning approaches independently of transfer (Tables 7 and 9).
  • W3 — Shift diagnosis. The original paper specifies both rules in Section 3.1, Step 2 and Appendix B.3. We clarify how the agent gate assesses predictive usefulness on target data and how the covariance gate balances source borrowing against covariance distortion. Added sensitivity analyses show stable performance around the selected thresholds (Tables 11–12).
  • Q1 — Learned contexts. Appendix D.5.1 of the original paper provides the prediction templates, initial contexts, and reflection prompts. We added learned source and target contexts to illustrate the resulting corrections, alongside fixed base-rate comparisons that show a larger benefit for DeepSeek-V4-Flash and a modest benefit for Qwen3-4B (Table 8).
  • Q2 — Feature extraction versus nuisance prediction. TRACE-NN uses embeddings or extracted hidden states as numerical inputs to separately trained neural nuisance models. TRACE-AGENT uses the LLM itself to predict nuisance probabilities directly from raw text and learned contexts. The same pretrained LLM therefore serves different functions in the two procedures. To address the request, we added hidden-state extraction details and comparisons of TRACE-NN and the statistical baselines using all six LLMs in revised Appendix E.2, Table 6, with a pointer in Section 4.1. TRACE-NN achieves the lowest RMSE among statistical methods on five of the six representations.
  • Q3 — Benchmark value and the DiM comparison. TRACE-Bench generates both potential outcomes before text-dependent treatment assignment and shows estimators only the assigned outcome. Some adjusted baselines lose to DiM because nuisance-estimation errors can outweigh adjustment benefits. With fitted-counterfactual imputation, the evaluation reference can inherit model errors and favor estimators that resemble the imputation model. TRACE-Bench instead scores every method against the same known target-pair average, so the comparison is free of this imputation-induced evaluation bias and reveals whether adjustment actually improves ATE accuracy under population shift.
  • Q4 — PEHE. We computed PEHE using the same runs reported in the paper. TRACE-AGENT improves over its target-only counterpart across all six backbones, and TRACE-NN achieves the lowest PEHE among the evaluated statistical methods.
  • Q5 — Aggregation. AIPW is supported by Theorem 1’s double robustness guarantee, bounded benchmark propensities, and propensity clipping during estimation. We added an OR/IPW comparison using identical nuisance predictions: AIPW achieves the lowest RMSE for TRACE-NN and five of six TRACE-AGENT backbones (Table 10).
  • Q6 — Optimization budget. The matched budget covers stage-3 target context optimization only, so the comparison assesses the benefit of source information under the same target optimization budget. We make this explicit in the Table 2 caption and Appendix D.2.
AuthorsOpenReview ↗

Response to weakness 1

Comments and weaknesses

R3-W1: Evidence for source-to-target transfer

Reviewer comment:

Transfer learning. The experiments do not convincingly show the benefit of source-to-target transfer learning. Table 2 provides partial evidence by demonstrating target-only TRACE-AGENT underperforms the full pipeline, but this analysis is absent for TRACE-NN. The paper also does not consider a simple pooled-data baseline in which the model is trained jointly on source and target data.

Response:

Our original paper reports TRACE-NN’s source-only and target-only comparisons in Table 1. Appendix D.3 explicitly defines FIDDLE (source-only) and FIDDLE (target-only) as ablations using the architectures of the corresponding TRACE-NN stages. Using the same text representation, TRACE-NN achieves an RMSE of 0.0327, compared with 0.0457 and 0.0368 for these two variants, supporting the benefit of source-to-target transfer learning. We also report these comparisons with Qwen3 embeddings in Appendix E.2/Table 5, in simulation in Appendix E.3/Figure 5(b), and on the News benchmark in Appendix E.4/Table 6.

We additionally evaluate baselines trained jointly on labeled source and target records (i.e., simple pooled-data baselines). With bag-of-words features, pooled FIDDLE, TARNet, and DragonNet achieve RMSEs of 0.0352, 0.0494, and 0.0481, respectively. For Qwen3-4B, pooled context learning yields an RMSE of 0.0690, compared with 0.0381 for TRACE-AGENT. Beyond the lower RMSEs achieved by TRACE-NN and TRACE-AGENT in these comparisons, TRACE supports reusing learned source contexts or nuisance models when the labeled source training records are unavailable. Pooling requires access to those records during target estimation.

The evidence of transfer learning benefits on TRACE-Bench is summarized below (Lower is better):

MethodText representationTraining settingATE RMSE
FIDDLEBag-of-wordsSource only0.0457
FIDDLEBag-of-wordsTarget only0.0368
FIDDLEBag-of-wordsPooled source and target0.0352
TARNetBag-of-wordsPooled source and target0.0494
DragonNetBag-of-wordsPooled source and target0.0481
TRACE-NNBag-of-wordsSource-to-target transfer0.0327
TRACE-AGENT, Qwen3-4BRaw textTarget only0.0493
Pooled agent, Qwen3-4BRaw textPooled source and target0.0690
TRACE-AGENT, Qwen3-4BRaw textSource-to-target transfer0.0381
AuthorsOpenReview ↗

Response to weakness 2

R3-W2: Missing statistical and no-shift baselines

Reviewer comment:

Missing statistical baselines. Building on the previous concern, the statistical ML evaluation lacks several important comparisons. In particular, the paper should consider text-based causal methods (e.g., CausalBERT) under both target-only and pooled source-target training. It would also be informative to compare these methods against TRACE in a setting without distribution shift, to isolate the benefits of nuisance estimation with text covariates from those of the transfer stages. As it stands, most of the statistical ML baselines (with the exception of ℓ1\ell_1-TCL) appear to be variants of TRACE-NN.

Response:

CausalBERT comparison. We add CausalBERT under both target-only and pooled source–target training, following Veitch et al. (2020) [1], which adapts BERT to learn document embeddings that preserve information needed for causal effect estimation. This adds a text-based causal method developed independently of TRACE-NN. The table compares it with TRACE-NN using bag-of-words features and Qwen3 embeddings, and with TRACE-AGENT using raw text.

MethodText representationTraining settingATE RMSE
CausalBERTRaw textTarget only0.0788
CausalBERTRaw textPooled source and target0.0515
TRACE-NNBag-of-wordsSource-to-target transfer0.0327
TRACE-NNQwen3 embeddingsSource-to-target transfer0.0348
TRACE-AGENT, six backbonesRaw textSource-to-target transfer[0.0246, 0.0381]

Single-domain comparison without source transfer. To isolate nuisance estimation from cross-domain transfer, we compare methods trained only on the same target population within each task. We include CausalBERT, TARNet, DragonNet, FIDDLE (the single-domain counterpart of TRACE-NN), and TRACE-AGENT. Removing source information from every method allows this comparison to assess nuisance estimation independently of transfer. The results support the effectiveness of our nuisance-learning approaches even when cross-domain transfer is absent.

All methods in the following table are trained using target data only.

MethodText representationATE RMSE
CausalBERTRaw text0.0788
FIDDLE (target-only)Bag-of-words0.0368
TARNet target-onlyBag-of-words0.0392
DragonNet target-onlyBag-of-words0.0594
TRACE-AGENT target-only, DeepSeek-V4-FlashRaw text0.0270
TRACE-AGENT target-only, Qwen3.6-27BRaw text0.0276
TRACE-AGENT target-only, GPT-OSS-120BRaw text0.0326
TRACE-AGENT target-only, Gemma-4-12BRaw text0.0362
TRACE-AGENT target-only, GPT-OSS-20BRaw text0.0363
TRACE-AGENT target-only, Qwen3-4BRaw text0.0493

In the revised paper, we have added CausalBERT implementation details in Appendix E.3, the target-only comparisons in Table 7, and the pooled comparisons in Table 9. Section 4.3 also points to these additions.

Reviewer_FiPHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies average treatment effect (ATE) estimation with LLMs. They introduce a semi-synthetic benchmark, TRACE-Bench, to study ATE estimation from text under distribution shift. They propose a framework, called TRACE, that estimates nuisance functions through prompt optimization for LLMs or model training for neural networks, and then combines the predictions with observed data using a doubly robust estimator to estimate ATE. The experiments show improvement over LLM agent and statistical ML baselines.

Critical strengths

This paper explores using LLMs for ATE estimation by decomposing the task into fitting nusiance fucntion, which is well-motivated. The proposed methods show improvements over several LLM-agent and statistical ML baselines, with ablation studies on the effects of prompt optimization and transfer learning.

Critical weaknesses

My main concern is whether TRACE-AGENT’s improvements generalize to other causal estimation settings. The paper tests six LLMs, which provides useful evidence for the improvements across models, but all tasks come from the same Reddit moral-judgment setting with LLM-generated outcomes.

Questions for authors
  1. The paper motivates estimation under both covariate and posterior shift. Could the authors clarify how TRACE-Bench’s construction establishes posterior shift, separately from covariate shift?
  2. How does TRACE-AGENT compare with using an LLM to extract covariates, followed by conventional nuisance modeling and AIPW estimation?
7 comments
AuthorsOpenReview ↗

Key ideas of responses

Key ideas of responses

We thank the reviewer for recognizing our well-motivated approach to LLM-based ATE estimation through nuisance-function learning. The reviewer highlights empirical improvements over both LLM-agent and statistical ML baselines, together with ablation evidence on the contributions of prompt optimization and transfer learning.

  • W1 — Generalization. Table 1 of our original paper shows that TRACE-AGENT's gains hold across six foundation models. To test a second corpus, we added PeerRead experiments with outcomes generated analytically without an LLM (Appendix E.5), showing that TRACE-AGENT improves over LLM direct estimation.
  • Q1 — Posterior shift. - Treatment posterior shift is established by the topic-specific propensity offsets in the benchmark construction (Appendix D.1). We added calibration analyses for outcome posterior shift, showing topic-associated target miscalibration beyond held-out-source placebos.
  • Q2 — LLMs as feature extractors. We have evaluated LLM-extracted features through Qwen3 embeddings with statistical nuisance estimators and AIPW (Appendix E.2, Table 5). For this rebuttal, we added hidden-layer representations from all six TRACE-AGENT backbones (Table 6) as a further test.
AuthorsOpenReview ↗

Response to comments and weaknesses

Comments and weaknesses

R2-W1: Generalization beyond the current benchmark

Reviewer comment:

My main concern is whether TRACE-AGENT’s improvements generalize to other causal estimation settings. The paper tests six LLMs, which provides useful evidence for the improvements across models, but all tasks come from the same Reddit moral-judgment setting with LLM-generated outcomes.

Response:

Additional corpus and construction. To address the concern, we evaluate TRACE-AGENT on an additional corpus PeerRead [1], and adapt the semi-synthetic construction used by CausalBERT [2]. We use titles and abstracts from 11,778 scientific papers as covariates, with 9,159 pre-2017 papers as the source and 2,619 papers submitted in 2017 as the target. Treatment indicates whether the parsed full paper contains a theorem reference.

Outcomes are simulated analytically without an LLM, and do not use real acceptance decisions. The source outcome mean is μtP(X)=σ(0.25t+b(X))\mu_t^P(X)=\sigma(0.25t+b(X)), where σ\sigma is the logistic function and b(X)=5{πref(Z(X))−0.2}b(X)=5\{\pi_{\mathrm{ref}}(Z(X))-0.2\}. Here, Z(X)Z(X) indicates title buzzwords, and πref(z)\pi_{\mathrm{ref}}(z) is the corpus-wide treatment proportion within that stratum. We consider two target conditions:

  • Temporal shift: the target retains the source outcome function, μtQ(X)=μtP(X)\mu_t^Q(X)=\mu_t^P(X).
  • Temporal plus mechanism shift: the target uses μtQ(X)=σ(0.25t+b(X)+E(X)t)\mu_t^Q(X)=\sigma(0.25t+b(X)+E(X)t), where E(X)E(X) is a fixed rule-based indicator that the abstract reports an empirical evaluation. This changes the treated-arm outcome mechanism while preserving the control-arm mechanism.

We sample binary outcomes from these probabilities and compute the target ATE from the known conditional means.

Evaluation and results. We compare TRACE-AGENT with direct LLM estimation using DeepSeek-V4-Flash and Qwen3-4B on the same source and target records. We report ATE RMSE over five outcome replicates on the fixed temporal split, using per-replicate medians of three runs. Parentheses report bootstrap SEs across replicates.

MethodTemporal shiftTemporal + mechanism shift
TRACE-AGENT, DeepSeek-V4-Flash0.0271 (0.0082)0.0236 (0.0038)
Direct estimation, DeepSeek-V4-Flash0.0586 (0.0118)0.0266 (0.0077)
TRACE-AGENT, Qwen3-4B0.0401 (0.0055)0.0220 (0.0042)
Direct estimation, Qwen3-4B0.2511 (0.0657)0.2368 (0.0366)

TRACE-AGENT improves substantially on direct estimation for Qwen3-4B in both conditions and for DeepSeek-V4-Flash under temporal shift. Under the additional mechanism shift, TRACE-AGENT also achieves a modest RMSE reduction over direct estimation with DeepSeek-V4-Flash. These results support generalization beyond Reddit moral judgments and LLM-generated outcomes.

In the revised paper, we have added the PeerRead construction, evaluation protocol, and results in Appendix E.5, highlighted this evaluation in Section 4.2, and updated the scope and limitations discussion in Section 6.

AuthorsOpenReview ↗

Response to question 1

Questions for authors

R2-Q1: Posterior shift versus covariate shift

Reviewer question:

The paper motivates estimation under both covariate and posterior shift. Could the authors clarify how TRACE-Bench’s construction establishes posterior shift, separately from covariate shift?

Response:

Our original paper describes covariate shift through semantically distant source and target topic clusters in Appendix D.1. For posterior shift, our starting point is that we do not assume the conditional treatment and outcome relationships are identical across these topics. Section 2.1 distinguishes changes in these conditional relationships from changes in the narrative distribution alone. Appendix D.1 documents the topic-specific offsets in the treatment rule and the LLM-generated outcomes under both exposures. These construction details establish the setup; differences in topic composition alone do not establish posterior shift conditional on the full narrative.

To supplement the documented construction, we provide calibration diagnostics for this rebuttal. We fit nuisance models on the source using the 128-dimensional PCA representation of story embeddings and evaluate their calibration on the target, with held-out source observations as a placebo. Across the same 21 tasks in TRACE-Bench:

  • Treatment: the mean required intercept adjustment is 0.42 logit on the target, compared with 0.12 on the placebo. Calibration diagnostics exceed ∣z∣>2|z|>2 in 19/21 target tasks versus 3/21 placebo tasks. The estimated adjustments track the designed topic offsets, with correlations of 0.92–0.97 across two model classes.
  • Outcomes: the corresponding adjustments for Y(0)Y(0) and Y(1)Y(1) are 0.39–0.43 logit on the target, compared with 0.12–0.14 on the placebo. Calibration diagnostics exceed ∣z∣>2|z|>2 in 12–14/21 target tasks versus 0–2/21 placebo tasks.

These diagnostics show systematic failures of source model calibration on the target beyond those observed within the source. The outcome diagnostics reveal topic-associated miscalibration, with residual dispersion across topics 1.8–2.0 times that under random groupings. This is consistent with outcome posterior shift relative to the shared text representation and supports target-specific correction despite using the same LLM labeling procedure.

Reviewer_wfJDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies target average treatment effect estimation from unstructured text under distribution shifts between source and target populations. The authors introduce TRACE-Bench, an end-to-end benchmark based on social narratives for evaluating ATE estimation when both covariate distributions and causal mechanisms may shift across domains. The paper further proposes TRACE, a shift-adaptive framework that aims to retain transferable source information while adapting mechanisms that change. Two instantiations are presented: TRACE-AGENT, which uses language-model-based context learning and adaptation followed by doubly robust aggregation, and TRACE-NN, an LLM-free factor-augmented neural realization of the same framework.

Critical strengths
  • Estimating causal effects from text is increasingly relevant, and the paper addresses a practically important but underexplored setting: causal estimation under both covariate shift and posterior/mechanism shift. This is a meaningful extension beyond standard text-based causal inference settings.
  • TRACE-Bench is valuable because it provides an end-to-end evaluation setting for studying whether LLM agents can perform population-level causal estimation from text under distribution shift.
  • TRACE’s decomposition into transferable source information and adaptable mechanisms is conceptually well aligned with the problem setting. The two implementations (TRACE-AGENT and TRACE-NN) also make the framework more broadly interpretable: one explores LLM-based adaptation, while the other shows that an LLM-free model can remain competitive.
Critical weaknesses
  • The current evaluation appears to rely on semi-synthetic tasks from one text corpus and one family of topic shifts. This substantially limits the evidence that TRACE generalizes to broader real-world text domains, different causal structures, or more diverse forms of distribution shift.
  • The theoretical guarantees still require causal identification and nuisance convergence conditions. These assumptions are standard but strong, especially in text-based settings where unmeasured confounding, label ambiguity, and representation error are difficult to rule out. The paper would benefit from a clearer discussion of how realistic these assumptions are in the benchmark and in intended applications.
  • TRACE appears to combine ideas from representation learning, shift adaptation, and AIPW-style doubly robust estimation. While the combination is useful, the paper should more explicitly distinguish its methodological contribution from prior work on transportability, domain adaptation for causal inference, and doubly robust estimation under covariate shift.
  • The paper would be stronger with more detailed ablations isolating the contribution of each component: source information retention, mechanism adaptation, language-context learning, factor augmentation, and doubly robust aggregation. Without these, it is hard to know which part of TRACE is most responsible for the observed gains.
Questions for authors
  • How sensitive are the results to the specific construction of TRACE-Bench, especially the choice of text corpus and topic-shift family?
  • Can the authors provide experiments on additional corpora or more heterogeneous shift types, such as demographic shifts, temporal shifts, label-mechanism shifts, or adversarially induced shifts?
  • How are causal identification assumptions justified in TRACE-Bench? Are there hidden confounders in the original narratives that may not be captured by the semi-synthetic setup?
  • For TRACE-AGENT, how stable are the learned natural-language contexts across random seeds, prompts, and different foundation models?
  • How much of the performance gain comes from the doubly robust aggregation versus the shift-adaptive representation/context-learning components?
  • Does TRACE provide calibrated uncertainty estimates for ATE? If so, how well do confidence intervals behave under covariate and posterior shift?
8 comments
AuthorsOpenReview ↗

Key ideas of responses

Key ideas of responses

We thank the reviewer for the careful reading and constructive feedback. Yes, TRACE-Bench is the first end-to-end benchmark for evaluating population-level causal estimation from raw text under both covariate and posterior shift. Its paired potential outcomes provide known causal effects for evaluation without fitted counterfactual imputation. TRACE-AGENT is the first framework to learn and transfer natural-language contexts for propensity and outcome estimation from raw text in a source–target ATE setting. It retains useful source information and learns target-specific corrections while keeping LLM weights frozen, followed by doubly robust aggregation. Across all six evaluated foundation models, TRACE-AGENT improves ATE accuracy over generic LLM agents. TRACE-NN provides a compact, LLM-free implementation that outperforms the evaluated statistical baselines and remains competitive with much larger LLMs. Both implementations admit double robustness guarantees under the stated assumptions.

  • W1, Q1–Q2 — Generalization. Our original paper shows that TRACE-AGENT's gains hold across six foundation models (Table 1). We added PeerRead experiments on scientific text with analytically generated outcomes under temporal and outcome-mechanism shifts (Appendix E.5). The experiments show that TRACE-AGENT improves over LLM direct estimation, and TRACE-NN is the best among statistical methods compared.
  • W2, Q3 — Identification and assumptions. TRACE-Bench ensures unconfoundedness and overlap by construction. It generates both outcomes at the same time, so hidden factors in the original narratives do not confound its controlled treatment assignment, and ground truth requires no counterfactual imputation. Nuisance convergence is plausible given our model designs, with empirical support from the ablations and stability analyses.
  • W3 — Methodological contribution. Section 1 of our original paper has positioned TRACE against text-based causal estimation, causal transfer, and test-time context learning. TRACE-AGENT is the first framework to learn and transfer natural-language contexts for propensity and outcome estimation from raw text. Our response spells out how this differs from transportability theory, parameter transfer, and prior LLM-based estimators.
  • W4, Q5 — Component contributions. Our original experiments have isolated source transfer (Table 1 for TRACE-NN, Table 2 for TRACE-AGENT), context learning (Figure 4), and the factor-augmented architecture (Table 1, Finetune Vanilla-NN). We added unconditional source reuse, fixed base-rate contexts (Table 8), and an AIPW/OR/IPW comparison (Table 10), showing that each component contributes.
  • Q4 — Context stability. We added a stability analysis in Appendix D.5.1. Across seeds, learned source contexts, and foundation models, the wording differs but outcome cues recur and target calibration is consistent.
  • Q6 — ATE uncertainty. Every RMSE in Table 1 of the original paper comes with a bootstrap SE (Appendix D.4). We also evaluated 95% confidence intervals for individual ATE estimates, showing that empirical coverage is 98.4–100% for TRACE-AGENT and 100% for TRACE-NN on the shifted tasks.
AuthorsOpenReview ↗

Response to weakness 1 and 2

Comments and weaknesses

R1-W1: Benchmark scope and generalization

Reviewer comment:

The current evaluation appears to rely on semi-synthetic tasks from one text corpus and one family of topic shifts. This substantially limits the evidence that TRACE generalizes to broader real-world text domains, different causal structures, or more diverse forms of distribution shift.

Response: Our original paper evaluates TRACE-NN under controlled mechanism shifts (Appendix E.3) and on the News corpus (Appendix E.4). In the revision, we have added PeerRead experiments for both TRACE-AGENT and TRACE-NN under temporal and outcome-mechanism shifts, using analytically generated outcomes (new Appendix E.5). TRACE-AGENT improves on direct estimation, while TRACE-NN achieves the lowest observed RMSE among statistical methods in both conditions. These additions extend the evidence beyond Reddit, topic-based splits, and LLM-generated outcomes. We have also highlighted this evaluation in Section 4.2 and updated the scope discussion in Section 6. Q1–Q2 provide the experimental details.

R1-W2: Identification and convergence assumptions

Reviewer comment:

The theoretical guarantees still require causal identification and nuisance convergence conditions. These assumptions are standard but strong, especially in text-based settings where unmeasured confounding, label ambiguity, and representation error are difficult to rule out. The paper would benefit from a clearer discussion of how realistic these assumptions are in the benchmark and in intended applications.

Response: In our original paper, we state the causal identification and nuisance-convergence assumptions in Section 5 and acknowledge their role in Section 6. The benchmark construction supporting identification is documented in Appendix D.1, as explained in our response to Q3. For nuisance learning, TRACE-NN uses factor augmentation and sparsity regularization (Section 3.2), while TRACE-AGENT optimizes task-specific contexts using labeled data and validation loss within a frozen LLM (Section 3.1; Appendix D.2). These design choices support learning with limited target data. The existing context-learning ablation and the additional stability analysis discussed in Q5 and Q4 provide empirical evidence.

AuthorsOpenReview ↗

Response to weakness 3 and 4

R1-W3: Methodological novelty and related work

Reviewer comment:

TRACE appears to combine ideas from representation learning, shift adaptation, and AIPW-style doubly robust estimation. While the combination is useful, the paper should more explicitly distinguish its methodological contribution from prior work on transportability, domain adaptation for causal inference, and doubly robust estimation under covariate shift.

Response:

TRACE-AGENT is the first framework to learn and transfer natural-language contexts specifically for propensity and outcome estimation from raw narratives in a source–target ATE setting. Its contribution is a supervised learning procedure that uses observed-label losses to optimize a frozen LLM’s nuisance predictions, evaluates their usefulness on target data, and learns target-specific corrections before causal aggregation.

Our original paper describes this framework and its two implementations in Section 1. The Related Work paragraph discusses text-based causal estimation, causal transfer, and test-time context learning. The distinctions are as follows.

Context learning for causal estimation. NATURAL [1] uses LLM distributions over study variables to construct causal estimators; CausalDANN [2] learns text-based outcome predictors with domain adaptation; and a LLM-driven ATE estimation [3] address incomplete text confounders at inference time through a doubly robust learner. TRACE-AGENT instead optimizes natural-language contexts so the LLM itself estimates the nuisance functions from text. We build on reflective prompt optimization, adapting its feedback and validation objectives to propensity and outcome learning and their source-to-target correction.

Transfer across populations. Transportability theory [4, 5] establishes when causal information can identify effects across populations. TRACE complements this work with an estimation procedure for limited labeled target data. Unlike parameter adaptation in ℓ1\ell_1-TCL [6], TRACE-AGENT adapts contexts while keeping LLM weights fixed. Prediction-based adaptation is also developed in SMART [7], on which TRACE-NN builds. TRACE-AGENT brings that transfer principle into supervised context learning, using source nuisance predictions and target feedback to guide correction.

Doubly robust estimation. AIPW supplies the established aggregation principle. TRACE’s methodological development concerns how its nuisance inputs are learned and adapted from raw text, with cross-fitted target predictions in TRACE-AGENT. The comparisons under Q5 separately assess learning, transfer, and aggregation.

The transportability comparison above supplements the related-work discussion in our original paper; we will add that reference in the revision.

R1-W4: Component ablations

Reviewer comment:

The paper would be stronger with more detailed ablations isolating the contribution of each component: source information retention, mechanism adaptation, language-context learning, factor augmentation, and doubly robust aggregation. Without these, it is hard to know which part of TRACE is most responsible for the observed gains.

Response:

Our original paper reports component comparisons in Table 1 and Section 4.3, including Table 2 and Figure 4. These examine nuisance learning and source-to-target transfer. Our response to Q5 supplements these existing results with additional ablations:

  • Source information and target adaptation. Table 1 compares TRACE-NN with source-only and target-only FIDDLE, and Table 2 compares TRACE-AGENT with target-only context learning. We additionally compare TRACE-AGENT with unconditional reuse of source predictions in Q5. These comparisons together assess the benefit of retaining source information and adapting it to the target population.
  • Language-context learning. Section 4.3 and Figure 4 evaluate context optimization with AIPW unchanged and transfer excluded. The added fixed base-rate context comparison in Q5 examines whether learned contexts improve on informative instructions without context optimization.
  • Factor-augmented nuisance learning. Table 1 compares TRACE-NN with Finetune Vanilla-NN. Appendix D.3 specifies that this baseline replaces the factor-augmented sparse networks in both stages with plain ReLU networks of the same depth and width while retaining the estimation pipeline. This comparison supports the benefit of the factor-augmented sparse architecture for nuisance estimation.
  • Doubly robust aggregation. The added AIPW, OR, and IPW comparisons in Q5 use the same nuisance predictions and target observations, so differences reflect the aggregation step.

The additional comparisons in Q5 extend these existing ablations by testing unconditional source reuse, fixed base-rate contexts, and alternative aggregation rules.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.