acceptodds
Under review as a conference paper at ICLR 2027

Explore Overlap Boundary: Causal Effect Bounds via Assignment Replay from Stochastic Algorithm under Structural Non-Overlap

Abstract

Stochastic algorithms assign personalized treatments such as videos or contexts whose features and outcomes vary across users. We study the conditional treatment-averaged causal effect of the treatment feature (C-TACE), with treatments generated from a stochastic algorithm. Under structural non-overlap, this estimand is unidentified as the non-realized treatment feature is concealed. We derive replay-aware causal bounds on C-TACE under non-overlap. When propensities are not logged, repeated replays are applied to estimate the propensity together with the non-realized feature. We show that the boundaries are in the form of function roots and that the bound is sharp under conditions, and we construct boundary estimators at finite replay counts. Rigorous theory has been developed on its asymptotic coverage. We also develop a replay-budget allocation method to improve bound efficiency. The method is comprehensively evaluated on semi-synthetic KuaiRec data, providing an applicable solution to non-overlap problem on causal effect under a stochastic algorithm.

Primary area:
causal reasoning
Keywords:
Non-overlap, Causal Effect Bounds, Stochastic Assignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ezJBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on estimating the C-TACE, the conditional, treatment-averaged causal effect in situations, first described in Athey (2026). This is the expected contrast in potential outcomes under the realized and unrealized treatments, conditional on units where the feature of the realized treatment is greater than the feature of the unrealized treatment. In particular, the paper focuses on a situation of structural non-overlap, where units propensity to treatment is either 0 or 1. In that situation, the probability of observing the counterfactual treatment and outcome is zero, even when replay is available. The authors provide bounds for the estimand under replay that are able to be estimated from data and experiments demonstrating the performance of the method.

Critical strengths

Given the setup where the non-realized features and propensities are not logged, the paper cleverly builds up an identification argument beginning from 1) the easier overlap case (ei∈(0,1)e_i \in (0, 1) with known propensities and no replay budget (so that ViNV_i^N), 2) partial identification/bounds with known propensities, infinite replay and bounded potential outcomes (ViNV_i^N is known under replay for units with overlap whereas the bounded potential outcomes are needed for the non-overlap group) and finally 3) when propensities are unknown with finite replay budget. The staging of this identification argument helps to provide intuition for how different parts of the final estimator/solution resolve different issues beginning from the easiest overlap, known propensities situation.

For the bounds provided for the structural non-overlap case, where propensities and non-realized features are unknown, the paper provides a proof that the estimator is asymptotically normal which is essentially for practical use of the method, e.g for constructing confidence intervals and hypothesis testing.

Critical weaknesses

The biggest weakness is the difference between the motivation (and original setup of the Athey (2026) work this paper extends) compared to the theory. For instance, as in Athey (2026) the motivation poses a personalized GenAI setup where many personalized treatments are generated. However, the theory in the paper focuses on a setup where there are only two potential treatments, with the replay mechanism stochastically choosing between the two. In Athey (2026), the replay only has to reproduce the feature class, here, it has to reproduce the same items. E.g Section 2.2 says the replay service retains both candidate treatments in order to reproduce the pair. The realism of this setup is questionable particularly when paired with the assumption that the non-realized feature is unknown - it is hard to think of a real world system where there are two possible options for treatment/personalization and that meta-data on those treatments are unavailable.

The fixed pair does allow for feature values that are continuous (since binary comparisons can be made) whereas the unbounded set of treatments requires discrete feature values or discretization. It may be worth highlighting this distinction.

The contrast with Athey (2026) is interesting since in that paper, they do not focus on a pair of treatments, only assuming that a single non-realized feature is available, only using replay to estimate inverse propensities. Under those assumptions, this paper largely redcues to using Susmann (2026) style bounds to handle structural non-overlap.

In short, the motivation needs rewriting given the existing theory or new theory needs to be written for the current motivation.

Questions for authors

Can you describe a concrete deployed system where (a) the treatment is a fixed pair, (b) replay re-selects within that pair, and (c) the analyst cannot access the non-realized feature? Section 2.2 says the replay service retains both candidates. Why can’t it return both features, or the selection probability?

In the generative-AI applications cited in the introduction, a fresh replay seed produces a new output rather than the other member of a fixed pair. How does Assumption 2 apply there?

In Athey et al.’s generator setting, the alternative is logged and replay only needs to reproduce the feature class. Wouldn’t deterministic feature assignment then produce tied pairs, which drop out of the estimand, rather than discordant units with hidden membership? In other words, does the non-overlap problem studied here arise outside the fixed-pair model?

Why not include the most natural baseline: the non-realized feature logged as in Athey et al., replay used only for the propensity, and worst-case bounds with known membership for deterministic units? The gap to your bound would show the cost of not logging the alternative.

In Table 4, the positive threshold widens the population bound (0.102 vs 0.037) yet narrows the confidence interval, and your own selection rule picks c = 0 in 464 of 500 trials. How should readers interpret this result?

5 comments

We greatly appreciate your recognition of the identification argument and of the asymptotic-normality result and its practical value for confidence intervals. We also thank you for the constructive and insightful suggestions, which help us clarify the generative setting, explain the role of the non-realized feature, and improve the experimental comparisons. We address these concerns below.

Q1. Can we identify a deployed system with a saved pair, pairwise replay, and restricted feature access?

The updated framework allows fresh treatment contents on every replay, so retaining the same candidate pair is no longer required. For each unit, the provider preserves the context, generator configuration, two conditional treatment distributions, their distinct feature values, and the assignment law. Production and replay use independent generation and assignment randomness. A fixed candidate pair is the special case in which both conditional distributions are point masses. Appendix G gives the extension.

For example, a GenAI advertising system can generate different videos at two preserved duration settings on every call. The provider retains the settings and configuration needed to regenerate treatments, while an external evaluation interface returns only the selected feature. The analyst observes (Yi,Xi,Vi,Vi(1),…,Vi(Mi))(Y_i,X_i,V_i,V_i^{(1)},\ldots,V_i^{(M_i)}). Thus, replay can be supported without storing every generated video or releasing the non-realized feature. Section 2.2 gives this example and specifies the analyst's information.

We also add KuaiRand-Pure application evaluating both fixed video pairs and changing video contents using actual randomized production exposures and observed responses. Independent replays are reconstructed from the specified assignment laws. Section 6.4 and Appendix H report the protocols, independent references, and comparisons with known membership.

Q2. How does the framework apply to fresh GenAI outputs?

The binary assignment indexes two candidates from the large treatment space A\mathcal A. The submitted Assumption 2 keeps their contents fixed. We have extended the formulation to allow fresh treatment contents on each replay, as described in Section 2.2 and Appendix G of the revised paper. The conditional treatment distributions, feature values, and assignment law remain fixed within each unit.

Let Pi0,Pi1\mathcal P_{i0},\mathcal P_{i1} be two conditional treatment distributions with distinct feature values Vi0<Vi1V_i^0<V_i^1. Production and replay independently repeat the following procedure at the same context and algorithm configuration:

Wi(j)∣Gi∼Bernoulli⁡(ei)W_i^{(j)}\mid\mathcal G_i\sim\operatorname{Bernoulli}(e_i),

Ai(j)∣Wi(j)=w,Gi∼PiwA_i^{(j)}\mid W_i^{(j)}=w,\mathcal G_i\sim\mathcal P_{iw},

Vi(j)=f(Ai(j))=Viwalmost surely when Wi(j)=wV_i^{(j)}=f(A_i^{(j)})=V_i^w\quad\text{almost surely when }W_i^{(j)}=w.

Here Gi\mathcal G_i contains the context, configuration, conditional treatment distributions, and potential outcomes, and assignment is independent of potential outcomes given the context and configuration. Only production, j=0j=0, yields an observed response. For example, a response generator can select between two response-length classes and generate different wording within the selected class. The treatment contents change while the conditional distributions, feature values, and assignment law remain fixed.

With μi(w)=EA∼Piw[Yi(A)]\mu_i(w)=\mathbb E_{A\sim\mathcal P_{iw}}[Y_i(A)], the same finite-replay argument gives

E[Ti∣Gi,Mi]=E[Di∣Gi,Mi]{μi(1)−μi(0)}\mathbb E[T_i\mid\mathcal G_i,M_i]=\mathbb E[D_i\mid\mathcal G_i,M_i]\{\mu_i(1)-\mu_i(0)\}, and τP=E[ei{μi(1)−μi(0)}]E[ei]\tau_{\mathcal P}=\frac{\mathbb E[e_i\{\mu_i(1)-\mu_i(0)\}]}{\mathbb E[e_i]}.

The non-realized treatment is defined by a draw from the other conditional distribution. Thus, the target averages contrasts between generated treatments, and the original endpoint equations apply when E[Di]>0\mathbb E[D_i]>0. Appendix G gives the identity, endpoint construction, proof, inference, and replay costs.

Q3. Does deterministic feature assignment create ties, and can hidden membership arise beyond fixed candidate contents?

Yes. If the realized and non-realized treatments are drawn from the same distribution whose feature is always v0v_0, then

Vi=ViN=v0,Zi=1Vi>ViN=0V_i=V_i^N=v_0,\qquad Z_i=1\\{V_i>V_i^N\\}=0.

These tied records do not belong to the target population. We thank the reviewer for pointing out that this case should be distinguished from deterministic selection between alternatives with different features.

The distinction concerns how the non-realized treatment is defined. In our model it comes from the unselected candidate or, under the generation extension in Appendix G, the unselected conditional treatment distribution. If the algorithm always selects the 60-second duration class and the other class is 30 seconds, then Zi=1Z_i=1 even though replay only returns the 60-second class. An analyst observing that class alone cannot distinguish this case from an unselected 90-second class, which would give Zi=0Z_i=0.

This uncertainty also arises when video contents change. The selected class can generate different videos on every call while the other class remains unobserved. In the KuaiRand structural non-overlap emulation in Appendix H, 40.684% of assignments are deterministic, and mean confidence width remains 0.617 at M=20M=20 despite changing video identities. If the non-realized feature is logged, membership becomes known, and counterfactual responses under zero-probability assignments remain unobserved. Section 2.1 states the exclusion of ties, and the generated-advertisement example in Section 2.2 illustrates hidden membership under deterministic assignment.

Q4. What does the baseline with the non-realized feature logged show?

Thanks for your suggestion! We agree that logging the non-realized feature provides a natural comparison. We added a comparator that receives ZiZ_i for every record, using the same observed responses, replay budget, and finite-replay scores as our method. This directly measures the information gain from knowing target membership.

Let Bi=1−RiB_i=1-R_i. The identity E[Ti]=E[RiZi(Yi−YiN)]\mathbb E[T_i]=\mathbb E[R_iZ_i(Y_i-Y_i^N)] identifies the retained contribution. Known membership then gives

Llog,M=E[Ti+BiZi(Yi−1)]E[Zi]L_{\mathrm{log},M}=\frac{\mathbb E[T_i+B_iZ_i(Y_i-1)]}{\mathbb E[Z_i]}, and Ulog,M=E[Ti+BiZiYi]E[Zi]U_{\mathrm{log},M}=\frac{\mathbb E[T_i+B_iZ_iY_i]}{\mathbb E[Z_i]}.

In remark, the denominator is observable, non-target records contribute zero, and nonretained target records require bounds only on their missing response.

We also add a comparison uses n=50,000n=50{,}000, M=20M=20, c=0c=0, ϵ=.01\epsilon=.01, nominal level 95%, and 500 paired evaluation replications per setting.

ρ\rhoFeature-only widthLogged widthJoint coverage, feature-only / logged
00.2680.10995.800% / 95.600%
.10.3250.15296.800% / 96.400%
.20.4130.22194.400% / 93.800%
.40.6590.37395.800% / 95.000%

At ρ=.4\rho=.4, logging membership reduces mean confidence width by 43.426%. The exact unsmoothed population width decreases from 0.630 to 0.341, and the known-membership infinite-replay reference has width 0.333, which reflects uncertainty about counterfactual outcomes under deterministic assignment.

This implemented comparison uses our finite-replay moment identity to handle unlogged propensities, and the table reports the value of logging the alternative with that estimator. Appendix H of the revised paper describes the known-membership comparator's information requirements and reports its KuaiRand comparisons for fixed and changing video contents.

Q5. Why can positive trimming widen the population bound while narrowing the confidence interval?

The confidence interval contains both the population bound and an expansion for sampling uncertainty. Before intersecting it with [−1,1][-1,1], its width is

Wn(c)=Uϵ(c)−Lϵ(c)+z.975n{σ−(c)+σ+(c)}W_n(c)=U_\epsilon(c)-L_\epsilon(c)+\frac{z_{.975}}{\sqrt n}\{\sigma_-(c)+\sigma_+(c)\}.

A positive threshold moves more records into the bounded contribution, widening the population interval, while limiting the correction Hi/(Ni−Hi)H_i/(N_i-H_i). This can reduce sampling variability. At finite nn, the confidence interval can narrow if the decrease in the sampling expansion exceeds the increase in population width. Section 4.2 of the revised paper gives the confidence-interval construction, and Section 5.1 and Appendix E analyze this threshold-dependent tradeoff.

The sparse setting assigns the higher-feature treatment with probability .99.99 in 5% of the population. At n=2,000n=2{,}000, only one lower-feature production assignment from this component is expected, and the untrimmed correction can be large at M=100M=100. To better illustrate the implication, we added an experiment using 5,000 paired replications, n=2,000n=2{,}000, M=100M=100, ϵ=.01\epsilon=.01, nominal level 95%, and 200,000 replay calls per method, with no pilot calls.

MethodExact unsmoothed population widthMean confidence widthJoint endpoint coverage
Original, c=0c=00.0370.23087.660%
Original, c=.05c=.050.1020.22095.360%
Discovered membership, c=.05c=.050.0970.21495.260%
Balanced weights, κ=5\kappa=50.0920.21095.160%

All methods target the same τ=−0.440\tau=-0.440, and each method's joint coverage concerns its own smoothed population endpoints. The membership refinement uses non-realized features discovered by replay even when the record is not retained. Balanced weights additionally bound the outcome correction by κ\kappa while preserving the finite-replay identity. In this evaluation, they give narrower intervals with joint endpoint coverage near 95% at the same replay budget. The revised paper reports the original and membership-aware validation results in Section 6.3 and derives the membership refinement in Appendix C.

Reviewer_hcPyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies causal effect estimation under stochastic treatment assignment when some units have no overlap and the feature of the unselected treatment is not observed. The authors focus on C-TACE, which measures the treatment effect conditional on the realized treatment having a larger feature value than the alternative. They derive lower and upper bounds for this effect and show that the bounds can be sharp under certain conditions. The paper then uses repeated runs of the assignment algorithm to construct bounds when assignment probabilities are not logged, along with asymptotic confidence intervals. The authors also propose methods for choosing a trimming threshold and allocating replay calls under a limited budget. They evaluate these methods on semi-synthetic KuaiRec data, comparing interval width and coverage with several baselines.

Critical strengths

The paper addresses an interesting problem where both the counterfactual outcome and the unselected treatment feature may be unavailable under non-overlap. The proposed bounds for full-population C-TACE are useful, and the authors establish sharpness under certain conditions. I think this is a meaningful extension of existing causal inference work. Another strength is the replay-based estimation method, which does not require directly observed assignment probabilities. The authors also provide theoretical guarantees for confidence interval coverage, which adds support to the proposed approach.

Critical weaknesses

My main concern is that the proposed methods for improving replay efficiency do not show convincing practical gains. In Appendix F, Table 6, adaptive allocation produces wider confidence intervals than uniform replay (U18) across four settings when the total budget includes the pilot calls. The threshold selection results are also limited. In Section 6.3, the pilot always selects c=0c=0 in the main ablation, so threshold selection provides no improvement there. In the sparse-overlap experiment, a positive threshold improves the interval width, but this threshold is fixed before evaluation rather than selected by the proposed procedure (Appendix F, Table 4). Since improving replay efficiency is one of the paper's contributions, I am not fully convinced that the additional complexity and pilot cost are justified by the results.

My second concern is about the practical scope of Assumption 2 in Section 2.2. The method requires the same two candidate treatments and assignment mechanism to remain fixed across independent replays. This is reasonable for some recommendation systems, but it is less clear how it applies to more general (e.g. generative) systems where running the algorithm again may produce different candidates. The paper describes a possible replay service, but the experiments use simulated assignments and replays rather than testing such a service in practice. I would have liked to see a clearer explanation of which real-world systems support this setting and how restrictive the assumption is. These concerns make me lean toward weak rejection, although I find the theoretical contributions useful.

Questions for authors

I have a few questions based on the weaknesses I identified in the paper. I am willing to reconsider my assessment if the authors can satisfactorily address these concerns.

  1. In Appendix F, Table 6, uniform replay with 18 calls per unit (U18) produces narrower confidence intervals than adaptive allocation across four settings under the same total replay budget. Under what conditions would the proposed allocation method outperform uniform replay after accounting for pilot costs?

  2. In Section 6.3, the threshold-selection procedure always chooses c=0c=0 in the main ablation. Appendix F, Table 4 shows that a fixed threshold of c=0.05c=0.05 improves confidence interval width in the sparse-overlap setting. However, a separate pilot experiment reported in Appendix F selects c=0c=0 in 464 of 500 replications. Was c=0.05c=0.05 included among the candidate thresholds, and could the authors explain why the selection procedure usually fails to identify the improvement?

  3. Assumption 2 requires repeated independent assignments using the same fixed candidate pair and assignment mechanism. This seems reasonable for some recommendation systems, but I am unsure how it applies to systems where rerunning the algorithm may generate different candidates. Could the authors clarify which practical applications satisfy this assumption?

  4. In Section 2.2, the replay service is assumed to retain the original candidate pair but only return the selected treatment feature to the analyst. Could the authors explain when this type of restricted access would arise in practice and why the unselected treatment feature could not be obtained directly?

5 comments

We greatly appreciate your recognition of the paper's strengths on the usefulness of C-TACE bounds, the sharpness result and replay-based estimation, and the theoretical guarantee for endpoint inference. We also thank you for the constructive suggestions, which greatly helpa us clarify the framework and strengthen the empirical comparison. The detailed answer is provided below.

Q1. When can proposed allocation improve efficiency after including pilot costs?

In our framework, pilot calls select the allocation, while inference calls estimate the endpoints. Pilot calls rerun the assignment algorithm on independent preliminary records to estimate feature-discovery rates and choose the replay allocation. Once the allocation is fixed, inference calls replay fresh analysis records, whose feature counts and observed production outcomes construct the endpoint estimates and standard errors. Pilot calls and observations are used only to select the allocation and are not reused in inference scores or covariance estimation.

Table 6 shows that our allocation outperforms uniform allocation at the same number of inference calls. With 100,000 inference calls for each method, proposed allocation reduces mean width relative to U10 by 4.3%–7.6%. On the other hand , with our allocation using 80,000 pilot calls. When uniform allocation instead uses all 180,000 calls for inference, uniform 18 gives narrower intervals. This is reasonable because converting the pilot budget into inference calls produces an 80% increase in replay runs, which greatly improves the endpoint inference.

We therefore added an experiment reducing the pilot to 1,000 or 2,000 calls. These correspond to 100 or 200 independent pilot records with 10 replays each, followed by 100,000 inference calls on 10,000 records. The uniform comparator uses all 101,000 or 102,000 calls for inference, with no pilot calls. To spend these budgets exactly, 10% or 20% of records receive 11 replays and the remainder receive 10, with this choice fixed before assignment. We use 500 paired evaluation replications per setting, c=0c=0, ϵ=.01\epsilon=.01, and nominal level 95%.

Pilot callsρ\rhoAllocated widthUniform widthUniform minus allocated [95% MC interval]
1,000.20.5170.5400.024 [0.023, 0.025]
1,000.40.7200.7400.020 [0.019, 0.022]
1,000.61.0271.0580.032 [0.031, 0.032]
2,000.20.5110.5390.028 [0.028, 0.028]
2,000.40.7130.7390.026 [0.026, 0.027]
2,000.61.0231.0570.034 [0.034, 0.035]

Our allocation reduces mean confidence width by 2.8%–5.2% in all six settings, even when the uniform comparator uses the entire pilot budget for additional inference calls. These results further demonstrate the advantage of our method in providing narrower confidence intervals with small total replay budgets, despite using fewer inference replay calls than uniform allocation. The full protocol and extended results are in Appendix F.

Q2. Was c=.05c=.05 included in the pilot grid, and why did the pilot often choose c=0c=0?

Thanks for your suggestion. We replace the earlier sparse-pilot selection count with an evaluation using the specified grid {0,.05}\{0,.05\}. An independent pilot minimizes the projected confidence width for ninf=2,000n_{\mathrm{inf}}=2{,}000, after intersecting the interval with [−1,1][-1,1]. Both candidates use M=100M=100 and ϵ=.01\epsilon=.01, and the chosen threshold is fixed before inference. Across 500 replications at each pilot size, .05.05 is selected in 41/500, 166/500, and 343/500 trials with 400, 2,000, and 10,000 pilot records, respectively. The results imply that it is more likely to select a nonzero threshold cc with more accurate endpoint inference followed by increasing amount of pilot calls.

In this sparse setting, the higher-feature treatment has assignment probability .99.99 in 5% of the population. A 400-record pilot therefore expects only 400×.05×.01=.2400\times.05\times.01=.2 lower-feature assignments from this component. A pilot with few such observations may underestimate sampling variability at c=0c=0. Since positive trimming widens the population bound, the projected confidence width can then favor c=0c=0. The increased frequency of selecting .05.05 with larger pilots is consistent with this insight. We specify the candidate grid and pilot size alongside the selection results in the revision. Thanks again for your valuable comment!

Q3. How can replay allow different candidates and a larger treatment space, and what are the practical applications?

Thank you for this question. We have extended the model to allow fresh treatment contents on every replay. Each unit's two feature values, context, conditional treatment distributions, and assignment law remain fixed, with independent generation and assignment randomness. The feature values can vary continuously across units. For example, a GenAI advertising system can generate different videos at two preserved durations and estimate the purchase effect among users shown the longer candidate. Section 2.2 of the updated paper gives this application, and Appendix G defines the treatment-averaged C-TACE and proves that the finite-replay identity and endpoint construction still apply under the stated positivity conditions.

For a larger treatment space, following Athey et al.'s pairwise approach [1], we compare KK conditional treatment distributions through binary reductions. Each pair uses its corresponding production records and replays from the same conditional assignment law. The resulting bounds also bound a prespecified average of pairwise effects with nonnegative weights. Appendix G provides the formulation, proof, simultaneous endpoint inference, and replay-cost accounting.

We also add a KuaiRand-Pure application [2] which provides another empirical example. Its changing-video evaluation uses 320 pairs of video sets containing 2,402 videos and 200 reconstructed replay streams per scenario, conditional on fixed production samples. At M=20M=20, the fraction of records receiving multiple distinct replay videos with the same duration-class feature rounds to 100.000%. Mean confidence widths are 0.012 under randomized exposure and 0.617 under an emulation with 40.684% deterministic assignments. Exposures and responses come from real logs; replays are reconstructed and non-overlap is emulated. Section 6.4 and Appendix H give the fixed-video and changing-video protocols, independent references, and results.

References

  • [1] Athey, S., Imbens, G., and Ji, Z. (2026). Estimating Causal Effects from Data Generated by Stochastic Algorithms, Sections 8–11.

  • [2] Gao et al. (2022). KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos.

Q4. Why can the provider retain candidates while the analyst lacks their features, and is non-overlap a practical problem?

Following Q3's example, the provider retains each unit's duration settings, generator configuration, and assignment mechanism, while regenerating the video contents on each call. An external analyst receives an evaluation interface that returns only the selected treatment feature. The analyst cannot inspect the unselected generated videos or the internal candidate settings, and cannot override the assignment rule. Using the main paper's notation, the observed data are

(Yi,Xi,Vi,Vi(1),…,Vi(Mi)).\bigl(Y_i,X_i,V_i,V_i^{(1)},\ldots,V_i^{(M_i)}\bigr).

YiY_i is collected only from the production exposure. Each replay uses the same context and mechanism and returns Vi(j)V_i^{(j)}, without a new user response. The released covariates XiX_i contain the evaluation context, including the advertising placement, but not the provider's two candidate duration settings. This is an example of restricted analyst access: retaining information internally for replay does not make it available through the evaluation interface. Retaining every generated video is unnecessary when the conditional treatment distributions and feature values are preserved.

Structural non-overlap can arise from the advertising placement. Suppose a reserved slot requires a 30-second video. The generator constructs both candidates before the assignment algorithm applies this restriction. The following table describes two possible units using the same slot:

UnitCandidate features, fixed across replaysProduction and replay featureeie_iActual ZiZ_i
User 1Vi0=15, Vi1=30V_i^0=15,\ V_i^1=30Always 30 seconds11
User 2Vi0=30, Vi1=60V_i^0=30,\ V_i^1=60Always 30 seconds00

These candidate features are known to the provider, but the analyst receives only the selected 30-second feature. For User 1, the realized video is longer than the 15-second alternative, so Zi=1Z_i=1. For User 2, the realized video is shorter than the 60-second alternative, so Zi=0Z_i=0. Both assignment probabilities are deterministic, even though the generated video contents change. Replay preserves the placement restriction. Consequently, for either unit,

Vi=Vi(j)=30,j=1,…,MiV_i=V_i^{(j)}=30,\qquad j=1,\ldots,M_i.

No replay reveals ViNV_i^N, regardless of the number of calls. A fresh random seed changes content but cannot make the assignment algorithm display a 15- or 60-second video in the reserved 30-second slot. Production data likewise contain no response to the unselected duration for these restricted contexts. Other placements accepting both candidate durations can assign either with positive probability, giving an overlap population and a positive retained-target mass.

The selected feature alone does not reveal target membership. Observing 30 seconds repeatedly cannot determine whether the unselected feature is 15 seconds or 60 seconds. Knowledge that the slot requires 30 seconds fixes the selected feature, but does not fix the unselected feature. This is why both the counterfactual outcome and membership can remain unknown when the duration pair differs across units and is not released.

The paper's bounds account for both unknown membership and counterfactual outcomes through

min⁡(0,Yi−1−t)≤Zi(Yi−YiN−t)≤max⁡(0,Yi−t)\min(0,Y_i-1-t)\le Z_i(Y_i-Y_i^N-t)\le\max(0,Y_i-t).

If the non-realized feature is released, membership becomes known and the residual bounds can be tightened, while uncertainty about the counterfactual outcome under deterministic assignment remains. In the KuaiRand example, releasing membership reduces mean confidence width from 0.617 to 0.424 at the same replay budget. Appendix H gives the experiment and logged-membership comparison.

Reviewer_sFRiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies partial identification and inference for the conditional treatment-averaged causal effect (C-TACE) under structural non-overlap in stochastic treatment assignment. The main challenge is that non-overlap may conceal both counterfactual outcomes and the feature-defined membership of the target population. The authors derive population bounds characterized by roots of monotone functions, propose a finite-replay estimator that avoids explicitly estimating propensity scores, and establish asymptotic inference for smoothed bound estimators. They further investigate trimming threshold selection and replay budget allocation to improve efficiency, with semi-synthetic experiments on KuaiRec.

Critical strengths
  1. The paper addresses non-overlap when both counterfactual outcomes and target-population membership may be unobserved.

  2. The finite-replay identity enables estimation without explicitly estimating individual propensity scores.

  3. The paper integrates identification, inference, and replay budget allocation.

Critical weaknesses
  1. Much of the framework relies on standard trimming, partial identification, smoothing, and influence-function methods, with insufficient differentiation from prior work.

  2. The allocation method does not consistently outperform uniform replay under equal total budgets (Table 6).

  3. Experiments mainly report scalar-effect coverage rather than the endpoint coverage guaranteed by Theorem 3.

  4. Some nonretained observations already reveal target membership, yet the proposed bounds appear to ignore this information.

Questions for authors
  1. What are the key methodological advances beyond existing non-overlap bounds and replay-based estimators?

  2. How restrictive is the feature-support condition required for sharpness in Theorem 1, and can it be assessed from observed data?

  3. How sensitive is the finite-replay identity to violations of conditional independence across replay draws?

  4. What happens when the retained-target probability (d) is very close to zero, even if the theoretical positivity condition holds?

7 comments

Thank you for recognizing the importance of addressing non-overlap when both counterfactual outcomes and target-population membership may be unobserved, and the finite-replay mechanism as well as methodlogy of the paper. We really appreciate the valuable questions, and we are glad to respond them point by point as below.

W1 / Q1. What are the methodological advances beyond existing non-overlap bounds and replay estimators?

This work develops causal bounds under structural non-overlap when target membership can be partially observed under finite replays. With ViV_i and ViNV_i^N denote the realized and non-realized features respectively, the target population is defined by Zi=1Vi>ViNZ_i=1\\{V_i>V_i^N\\}, whose membership can remain hidden when replay never reveals the unselected feature. The setting considered in this paper leads to unique challenges in identification problem, the estimation, and the endpoint inference.

Compared with standard non-overlap bounds, such as Susmann et al. [2], our paper features three key distinctions:

  • Difference in causal target and population of interest. Standard ATE bounds, including [2], average potential outcome differeneces EY(1)−EY(0)EY(1)-EY(0) over the entire population, where membership in the target population is observed. This paper instead targets τ=E[Zi(Yi−YiN)]/E[Zi]\tau=E[Z_i(Y_i-Y_i^N)]/E[Z_i], averaging difference between realized and non-realized outcomes among units whose realized feature exceeds the non-realized feature, Zi=1Vi>ViN=1Z_i=1\\{V_i>V_i^N\\}=1. Because ViNV_i^N is not logged, membership in this population can itself be unobserved. This lead to substantially different expression and theorem on the partial identification bound.

  • Difference in endpoint expression. Unknown target membership requires an adaptation of the classical bound formulation. With known target membership, one can establish causal bound by bounding counterfactual outcome contributions. However, when the membership is unknown, E[Zi]E[Z_i] is not an observed normalizing constant, and bounding the missing outcome alone does not resolve the target ratio. This leads to a key challenge that the endpoint calculation must jointly account for both membership and outcome uncertainty. To address this challenge, this paper proposes a root formulation to handle this joint uncertainty. For E[Zi]>0E[Z_i]>0, the target ratio is equivalently characterized by $ \tau=t\Longleftrightarrow E[Z_i(Y_i-Y_i^N-t)]=0, $ which converts the ratio into the root of a function. With RiR_i selecting replay-count configurations, DiD_i their retained-target indicator, and TiT_i the observable outcome score, the exact finite-replay identity identifies the retained contribution as E[Ti−tDi]E[T_i-tD_i], and this paper proposes a new way to determine the endpoints under replay mechanism via solving endpoint equations F−(t)=E[Ti−tDi+(1−Ri)min⁡(0,Yi−1−t)],F+(t)=E[Ti−tDi+(1−Ri)max⁡(0,Yi−t)].F_-(t)=E[T_i-tD_i+(1-R_i)\min(0,Y_i-1-t)], \quad F_+(t)=E[T_i-tD_i+(1-R_i)\max(0,Y_i-t)]. This is a novel way to estimate the endpoint adaptive to the causal target under stochastic replay mechanism.

  • Endpoint inference incorporates the finite replay mechanism. First, the endpoints are implicit roots, so we derive their influence functions by linearizing the smoothed equations and dividing each score by its negative expected derivative. Classical ATE bound influence functions cannot be directly reused. Second, the scores vary with both sampled units and within-unit replay randomization, which remains under fixed replay budgets. The joint asymptotic analysis must account for both sources and the covariance between endpoints. In Theorem 3, we establishe joint asymptotic normality, consistent covariance estimation, and asymptotic containment of both smoothed endpoints. The theoretical results are fundamentally different from those without replay mechanism in that the result is built under a prespecified or independently selected replay rule, bounded replay counts, and positive retained mass and endpoint variances.

Compared with the stochastic-algorithm framework of Athey et al. [1], this paper also features two additional distinctions:

  • Replay supports full-population bounds with hidden membership. Athey et al.'s replayable-mechanism regime logs the alternative feature and uses inverse sampling on a trimmed overlap population. We instead use feature-only replays and retain deterministic records whose target membership may be unknown. The challenge is to identify an observable contribution without individual propensities and bound the remaining numerator and denominator jointly. Our exact finite-replay identity, E[Ti]=E[Di(Yi−YiN)],E[T_i]=E[D_i(Y_i-Y_i^N)], combined with membership-and-outcome residual bounds, yields the full-population bound, for E[Di]>0E[D_i]>0, τ∈[L,U],E[Ti−LDi+(1−Ri)min⁡(0,Yi−1−L)]=0,E[Ti−UDi+(1−Ri)max⁡(0,Yi−U)]=0.\tau\in[L,U],\quad E[T_i-LD_i+(1-R_i)\min(0,Y_i-1-L)]=0,\quad E[T_i-UD_i+(1-R_i)\max(0,Y_i-U)]=0. This addresses this challenge for the original full-population target.
AuthorsOpenReview ↗

Reply to Q1/W1, continue

  • Replay-budget constraints and allocation. Stepping forward from [1], which only considers a uniform replay allocation and does not formulate this gallocation problem for causal bounds, we study how to distribute a fixed total replay budget across units. Section 5.2 derives an allocation rule under ∑iMi=B\sum_iM_i=B, minimizing a bound on the probability of unresolved feature comparisons. An independent pilot fixes the allocation before inference, linking resource allocation to valid endpoint estimation. Further reports on the cost-adjusted comparison with uniform replay, including all pilot calls, and be seen in W2 below.

The following table contrasts standard ATE bounds [2], replay-aware estimator [1], and our method.

WorkMembershipTarget / boundsEstimationInferenceReplay roleBudget
Non-overlap [2]Fixed populationATE boundsOutcome modelsNo——
SGT [1]KnownOverlap effectInverse weightingPartiallyAssignmentUniform allocation
Our workPartially hiddenFull population boundsFunction SolvingYesAssignment + Membership discoveryTotal budget allocation

References

  • [1] Athey, S., Imbens, G., and Ji, Z. (2026). Estimating Causal Effects from Data Generated by Stochastic Algorithms, Sections 8–11.
  • [2] Susmann, H. P., McClean, A., and Díaz, I. (2026). Non-overlap Average Treatment Effect Bounds, version 2.

W2. Further explanation on Table 6 and the comparison on replay allocation mechanism.

In the inference method, pilot calls select the design while inference calls estimate the endpoints. Pilot calls rerun the assignment algorithm on independent preliminary records to estimate feature-discovery rates and choose the replay allocation. After that allocation is fixed, inference calls replay fresh analysis records, and their feature counts and logged production outcomes construct the endpoint estimates and standard errors. Pilot calls and pilot observations are not reused in inference scores or covariance estimation. Their role is to select the allocation rule independently of the inference sample.

Table 6 separates gains at equal inference cost from gains at equal total cost. With 100,000 inference calls for each method, allocation reduces mean width relative to uniform 10 by 4.3%–7.6%. Uniform allocation itself requires no pilot, and therefore Table 6's U10 evaluates the allocation gain at equal inference cost. When uniform allocation instead spends all 180,000 calls on inference, U18 outperforms allocation in reported configurationa, which is reasonable since there is a substantial 80% increase in replay rounds.

We also conduct an additional experiment to reduce the pilot to 1,000 or 2,000 total calls. These correspond to 100 or 200 independent pilot records with 10 replays each, followed by 10,000 inference records receiving 100,000 allocated calls. The uniform comparator uses all 101,000 or 102,000 calls for inference, with zero pilot calls. To spend these budgets exactly, 10% or 20% of records receive 11 replays and the remainder receive 10, giving means of 10.1 or 10.2.

We keep the existing allocation rule and use 500 paired replications per cell, c=0c=0, smoothing ϵ=.01\epsilon=.01, and nominal level 95%. The result is shown as follows:

Pilot callsρ\rhoAllocated widthUniform-all widthUniform minus allocated [95% MC interval]
1,000.20.5170.5400.024 [0.023, 0.025]
1,000.40.7200.7400.020 [0.019, 0.022]
1,000.61.0271.0580.032 [0.031, 0.032]
2,000.20.5110.5390.028 [0.028, 0.028]
2,000.40.7130.7390.026 [0.026, 0.027]
2,000.61.0231.0570.034 [0.034, 0.035]

Our allocation improves mean width in all six settings, by 2.8%–5.2%, even after every pilot call is charged. Thes result demonstrates the capability of our allocation method giving narrower bounds, even under stricter replay-budget constraint than uniform-allocation baseline.

W3. Endpoint coverage in experiment.

To assess the endpoint guarantee in Theorem 3 directly, we report 5,000 paired replications in the sparse KuaiRec setting, with n=2,000n=2{,}000, M=100M=100, nominal level 95%, and the same 200,000-call budget for every method. Completion outcomes are observed; production assignments and replays are simulated.

Scalar coverage measures containment of the true causal effect. Lower and upper endpoint coverage measure whether the confidence interval extends beyond the corresponding endpoints of each method's own smoothed, clipped population bound. Joint endpoint coverage requires both events and is the guarantee studied in Theorem 3. The bracketed Wilson intervals quantify Monte Carlo uncertainty in the reported joint coverage.

MethodMean widthScalarLower endpointUpper endpointJoint endpoints [95% Wilson MC interval]
Original, c=0c=00.23095.560%87.800%99.860%87.660% [86.719%, 88.543%]
Original, c=.05c=.050.220100.000%97.160%98.200%95.360% [94.741%, 95.909%]
Discovered membership, c=.05c=.050.214100.000%97.040%98.220%95.260% [94.635%, 95.815%]
Balanced weights, κ=5\kappa=50.21099.980%97.020%98.140%95.160% [94.530%, 95.721%]

The results supports the advantage of our method in producing tighter confidence intervals with joint endpoint coverage close to the nominal 95% level at the same replay budget. In this setting, the balanced-weight version achieves the lowest mean width, 0.210, with 95.160% joint endpoint coverage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.