- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper focuses on estimating the C-TACE, the conditional, treatment-averaged causal effect in situations, first described in Athey (2026). This is the expected contrast in potential outcomes under the realized and unrealized treatments, conditional on units where the feature of the realized treatment is greater than the feature of the unrealized treatment. In particular, the paper focuses on a situation of structural non-overlap, where units propensity to treatment is either 0 or 1. In that situation, the probability of observing the counterfactual treatment and outcome is zero, even when replay is available. The authors provide bounds for the estimand under replay that are able to be estimated from data and experiments demonstrating the performance of the method.
Critical strengths
Given the setup where the non-realized features and propensities are not logged, the paper cleverly builds up an identification argument beginning from 1) the easier overlap case ( with known propensities and no replay budget (so that ), 2) partial identification/bounds with known propensities, infinite replay and bounded potential outcomes ( is known under replay for units with overlap whereas the bounded potential outcomes are needed for the non-overlap group) and finally 3) when propensities are unknown with finite replay budget. The staging of this identification argument helps to provide intuition for how different parts of the final estimator/solution resolve different issues beginning from the easiest overlap, known propensities situation.
For the bounds provided for the structural non-overlap case, where propensities and non-realized features are unknown, the paper provides a proof that the estimator is asymptotically normal which is essentially for practical use of the method, e.g for constructing confidence intervals and hypothesis testing.
Critical weaknesses
The biggest weakness is the difference between the motivation (and original setup of the Athey (2026) work this paper extends) compared to the theory. For instance, as in Athey (2026) the motivation poses a personalized GenAI setup where many personalized treatments are generated. However, the theory in the paper focuses on a setup where there are only two potential treatments, with the replay mechanism stochastically choosing between the two. In Athey (2026), the replay only has to reproduce the feature class, here, it has to reproduce the same items. E.g Section 2.2 says the replay service retains both candidate treatments in order to reproduce the pair. The realism of this setup is questionable particularly when paired with the assumption that the non-realized feature is unknown - it is hard to think of a real world system where there are two possible options for treatment/personalization and that meta-data on those treatments are unavailable.
The fixed pair does allow for feature values that are continuous (since binary comparisons can be made) whereas the unbounded set of treatments requires discrete feature values or discretization. It may be worth highlighting this distinction.
The contrast with Athey (2026) is interesting since in that paper, they do not focus on a pair of treatments, only assuming that a single non-realized feature is available, only using replay to estimate inverse propensities. Under those assumptions, this paper largely redcues to using Susmann (2026) style bounds to handle structural non-overlap.
In short, the motivation needs rewriting given the existing theory or new theory needs to be written for the current motivation.
Questions for authors
Can you describe a concrete deployed system where (a) the treatment is a fixed pair, (b) replay re-selects within that pair, and (c) the analyst cannot access the non-realized feature? Section 2.2 says the replay service retains both candidates. Why can’t it return both features, or the selection probability?
In the generative-AI applications cited in the introduction, a fresh replay seed produces a new output rather than the other member of a fixed pair. How does Assumption 2 apply there?
In Athey et al.’s generator setting, the alternative is logged and replay only needs to reproduce the feature class. Wouldn’t deterministic feature assignment then produce tied pairs, which drop out of the estimand, rather than discordant units with hidden membership? In other words, does the non-overlap problem studied here arise outside the fixed-pair model?
Why not include the most natural baseline: the non-realized feature logged as in Athey et al., replay used only for the propensity, and worst-case bounds with known membership for deterministic units? The gap to your bound would show the cost of not logging the alternative.
In Table 4, the positive threshold widens the population bound (0.102 vs 0.037) yet narrows the confidence interval, and your own selection rule picks c = 0 in 464 of 500 trials. How should readers interpret this result?
5 comments
Q3. Does deterministic feature assignment create ties, and can hidden membership arise beyond fixed candidate contents?
Yes. If the realized and non-realized treatments are drawn from the same distribution whose feature is always , then
.
These tied records do not belong to the target population. We thank the reviewer for pointing out that this case should be distinguished from deterministic selection between alternatives with different features.
The distinction concerns how the non-realized treatment is defined. In our model it comes from the unselected candidate or, under the generation extension in Appendix G, the unselected conditional treatment distribution. If the algorithm always selects the 60-second duration class and the other class is 30 seconds, then even though replay only returns the 60-second class. An analyst observing that class alone cannot distinguish this case from an unselected 90-second class, which would give .
This uncertainty also arises when video contents change. The selected class can generate different videos on every call while the other class remains unobserved. In the KuaiRand structural non-overlap emulation in Appendix H, 40.684% of assignments are deterministic, and mean confidence width remains 0.617 at despite changing video identities. If the non-realized feature is logged, membership becomes known, and counterfactual responses under zero-probability assignments remain unobserved. Section 2.1 states the exclusion of ties, and the generated-advertisement example in Section 2.2 illustrates hidden membership under deterministic assignment.
Q4. What does the baseline with the non-realized feature logged show?
Thanks for your suggestion! We agree that logging the non-realized feature provides a natural comparison. We added a comparator that receives for every record, using the same observed responses, replay budget, and finite-replay scores as our method. This directly measures the information gain from knowing target membership.
Let . The identity identifies the retained contribution. Known membership then gives
, and .
In remark, the denominator is observable, non-target records contribute zero, and nonretained target records require bounds only on their missing response.
We also add a comparison uses , , , , nominal level 95%, and 500 paired evaluation replications per setting.
| Feature-only width | Logged width | Joint coverage, feature-only / logged | |
|---|---|---|---|
| 0 | 0.268 | 0.109 | 95.800% / 95.600% |
| .1 | 0.325 | 0.152 | 96.800% / 96.400% |
| .2 | 0.413 | 0.221 | 94.400% / 93.800% |
| .4 | 0.659 | 0.373 | 95.800% / 95.000% |
At , logging membership reduces mean confidence width by 43.426%. The exact unsmoothed population width decreases from 0.630 to 0.341, and the known-membership infinite-replay reference has width 0.333, which reflects uncertainty about counterfactual outcomes under deterministic assignment.
This implemented comparison uses our finite-replay moment identity to handle unlogged propensities, and the table reports the value of logging the alternative with that estimator. Appendix H of the revised paper describes the known-membership comparator's information requirements and reports its KuaiRand comparisons for fixed and changing video contents.
Q5. Why can positive trimming widen the population bound while narrowing the confidence interval?
The confidence interval contains both the population bound and an expansion for sampling uncertainty. Before intersecting it with , its width is
.
A positive threshold moves more records into the bounded contribution, widening the population interval, while limiting the correction . This can reduce sampling variability. At finite , the confidence interval can narrow if the decrease in the sampling expansion exceeds the increase in population width. Section 4.2 of the revised paper gives the confidence-interval construction, and Section 5.1 and Appendix E analyze this threshold-dependent tradeoff.
The sparse setting assigns the higher-feature treatment with probability in 5% of the population. At , only one lower-feature production assignment from this component is expected, and the untrimmed correction can be large at . To better illustrate the implication, we added an experiment using 5,000 paired replications, , , , nominal level 95%, and 200,000 replay calls per method, with no pilot calls.
| Method | Exact unsmoothed population width | Mean confidence width | Joint endpoint coverage |
|---|---|---|---|
| Original, | 0.037 | 0.230 | 87.660% |
| Original, | 0.102 | 0.220 | 95.360% |
| Discovered membership, | 0.097 | 0.214 | 95.260% |
| Balanced weights, | 0.092 | 0.210 | 95.160% |
All methods target the same , and each method's joint coverage concerns its own smoothed population endpoints. The membership refinement uses non-realized features discovered by replay even when the record is not retained. Balanced weights additionally bound the outcome correction by while preserving the finite-replay identity. In this evaluation, they give narrower intervals with joint endpoint coverage near 95% at the same replay budget. The revised paper reports the original and membership-aware validation results in Section 6.3 and derives the membership refinement in Appendix C.
We greatly appreciate your recognition of the identification argument and of the asymptotic-normality result and its practical value for confidence intervals. We also thank you for the constructive and insightful suggestions, which help us clarify the generative setting, explain the role of the non-realized feature, and improve the experimental comparisons. We address these concerns below.
Q1. Can we identify a deployed system with a saved pair, pairwise replay, and restricted feature access?
The updated framework allows fresh treatment contents on every replay, so retaining the same candidate pair is no longer required. For each unit, the provider preserves the context, generator configuration, two conditional treatment distributions, their distinct feature values, and the assignment law. Production and replay use independent generation and assignment randomness. A fixed candidate pair is the special case in which both conditional distributions are point masses. Appendix G gives the extension.
For example, a GenAI advertising system can generate different videos at two preserved duration settings on every call. The provider retains the settings and configuration needed to regenerate treatments, while an external evaluation interface returns only the selected feature. The analyst observes (Yi,Xi,Vi,Vi(1),…,Vi(Mi)). Thus, replay can be supported without storing every generated video or releasing the non-realized feature. Section 2.2 gives this example and specifies the analyst's information.
We also add KuaiRand-Pure application evaluating both fixed video pairs and changing video contents using actual randomized production exposures and observed responses. Independent replays are reconstructed from the specified assignment laws. Section 6.4 and Appendix H report the protocols, independent references, and comparisons with known membership.
Q2. How does the framework apply to fresh GenAI outputs?
The binary assignment indexes two candidates from the large treatment space A. The submitted Assumption 2 keeps their contents fixed. We have extended the formulation to allow fresh treatment contents on each replay, as described in Section 2.2 and Appendix G of the revised paper. The conditional treatment distributions, feature values, and assignment law remain fixed within each unit.
Let Pi0,Pi1 be two conditional treatment distributions with distinct feature values Vi0<Vi1. Production and replay independently repeat the following procedure at the same context and algorithm configuration:
Wi(j)∣Gi∼Bernoulli(ei),
Ai(j)∣Wi(j)=w,Gi∼Piw,
Vi(j)=f(Ai(j))=Viwalmost surely when Wi(j)=w.
Here Gi contains the context, configuration, conditional treatment distributions, and potential outcomes, and assignment is independent of potential outcomes given the context and configuration. Only production, j=0, yields an observed response. For example, a response generator can select between two response-length classes and generate different wording within the selected class. The treatment contents change while the conditional distributions, feature values, and assignment law remain fixed.
With μi(w)=EA∼Piw[Yi(A)], the same finite-replay argument gives
E[Ti∣Gi,Mi]=E[Di∣Gi,Mi]{μi(1)−μi(0)}, and τP=E[ei]E[ei{μi(1)−μi(0)}].
The non-realized treatment is defined by a draw from the other conditional distribution. Thus, the target averages contrasts between generated treatments, and the original endpoint equations apply when E[Di]>0. Appendix G gives the identity, endpoint construction, proof, inference, and replay costs.