acceptodds
Under review as a conference paper at ICLR 2027

Finite-Sample Certification of Capacity-Constrained Treatment Allocations in Prespecified Cohorts

Abstract

Capacity-constrained treatment allocation uses experimental evidence to select a fixed number of individuals from a prespecified cohort for treatment. We study a complementary problem: certifying the welfare loss of a selected allocation relative to the best feasible alternative. Under a prespecified treatment effect model linking the randomized trial and target cohort, we characterize which allocation welfare differences are identified. These differences can remain identifiable even when some individual target treatment effects are not. We then construct simultaneous finite-sample upper bounds directly for the welfare differences that determine regret. The resulting regret certificates remain valid when the same trial is used for both allocation selection and certification. The construction applies to a prespecified candidate set and extends to all exact- allocations through exchange bounds and maximum-weight partial matching. We further bound the excess of the certificate over true cohort-specific regret and show that, under a common confidence region, direct comparison gives no larger uncertainty radius than propagating separate treatment effect or allocation welfare intervals. Experimental results confirm post-selection coverage and show smaller mean certificates from direct comparison in the evaluated three-feature settings.

Primary area:
causal reasoning
Keywords:
Causal inference, Treatment allocation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UfK2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines finite-sample certification of welfare loss for treatment allocations to a prespecified cohort relative to the best allocation. The approach bounds the difference in welfare between allocations. The theoretical guarantees hold when the same randomized trial is used for allocation, selection, and certification. Experimental results show that smaller mean regret certificates are obtained compared to baselines.

Critical strengths
  • The paper addresses a relevant problem in treatment allocation. The focus is on certifying the quality of a treatment allocation, which could be useful in practice.
  • The introduction and related work are fairly well written. The related work contains a good discussion of related methods in CATE and policy learning with resource constraints.
  • The experiments show promising results in reducing certificate size in some representative settings. These show the benefits of directly bounding welfare differences.
Critical weaknesses
  • There is an omission to a relevant literature, namely, decision-focused learning. A central idea of that field is that decision quality rather than prediction accuracy should guide model training for matching and resource allocation. Connecting to that broader literature would be a good addition to the related work. There are also closely related works in bandits and resource/treatment allocation via matching with predictions (e.g., for transplantation).
  • After the introduction and related work, I found the writing quality dropped and the paper was substantially harder to follow. The technical sections rely heavily on dense notation and jargon, with limited explanatory prose. Important details are often deferred to the appendix and there is little discussion of the results, their implications, importance, and intuition. I would encourage the authors to simplify the notation and include more intuition and even some proof sketches in the main body. For example, Proposition 1 is introduced with limited motivation. Figure 1 provides some intuition, but the main text should explain more directly what the goal and importance is. The idea that a welfare difference can be identified when the components left unidentified by the trial cancel is much simpler than the notation suggests.
  • The role of the certificates in choosing an allocation is underdeveloped. Does certificate-based selection improve welfare or reduce regret compared with maximizing estimated welfare? Addressing these points and adding algorithmic implications would greatly strengthen the paper’s contributions.
  • Overall, I do not find the technical contributions particularly compelling as presented. A clearer explanation of the challenges specific to this setting and which results require new arguments rather than adaptations of established techniques would benefit the paper and highlight the contribution. The presentation issues discussed above make this distinction harder to assess and may obscure the actually novel parts of the work.
Questions for authors
  1. How do these results connect to prior work in decision-focused learning, bandits, and matching? Which results follow from adapting existing techniques, and which require new technical arguments specific to this setting?
  2. What algorithmic contributions can be obtained from the welfare certificates? Does certificate-based selection improve welfare or reduce regret compared with maximizing estimated welfare?
  3. Could the finite-sample guarantees extend to data generated via nonrandom treatment assignments?
4 comments
AuthorsOpenReview ↗

Response to Reviewer UfK2 (1/3)

Thank you for recognizing the relevance of the certification problem, the discussion of related work, and the benefits of directly bounding welfare differences! We address the connections to decision-focused learning, bandits, and matching, the technical contributions, allocation selection, nonrandom treatment assignment, and readability below.

We have simplified the notation, added intuition and a short proof sketch before Proposition 1, and stated the conditions together with the guarantee in Theorem 2. Equation (4) now writes out the gap between the best feasible allocation and the best candidate without a separate symbol. As you point out, a common unidentified effect component can cancel between allocations treating the same number of people. Under the common effect model, components unidentified by the trial can cancel in an allocation welfare difference, making it identifiable even when individual effects are not.

[W1, W4, Q1] Connections to prior work and technical contributions

The learning methods you mention help choose an allocation. Our method uses the trial to bound its welfare loss. We account for specified model error in both the trial and target cohort and compute the bound over all feasible allocations. Thank you for helping us make this connection explicit.

1. The cited decision-focused learning and matching work connects prediction to decision quality, while the bandit work guides sequential sampling. Our certification bounds the loss of a selected allocation using a given randomized trial.

Wilder et al. [1] train models using downstream decision objectives, and El Balghiti et al. [2] study generalization bounds for decision loss. Fiez et al. [3] guide sampling using reward differences. Xu et al. [4] learn counterfactual outcomes for unobserved donor–recipient pairs from observational data.

Each trial member provides only the outcome under the treatment received, so treatment effects are not directly observed. The common treatment effect model and identification conditions establish which target allocation welfare differences the trial identifies. Our method, Pairwise, bounds these differences simultaneously. Consequently, a learning method can propose an allocation, and certification checks whether the evidence supports a specified loss requirement after selection using the same trial.

2. Our additional arguments correct joint trial and target bias and establish a computable full-allocation certificate with bounded excess over true regret.

Linear model identification criteria, concentration inequalities, and matching solvers provide the established tools.

(a) Trial model bias can invalidate certification despite randomization. Correcting the combined trial and target bias preserves validity when the specified error range contains the actual deviations.

Our counterexample shows this failure even with an exact target model, and increasing sample size does not remove the bias. We derive the combined trial and target bias and bound it over a prespecified class. When one function describes both deviations, this correction also preserves their possible cancellation.

(b) With fixed treatment capacity, feasible alternatives correspond to partial matchings. Given simultaneous exchange bounds, this yields a polynomial-time certificate with bounded excess over true regret.

Each alternative can be represented by exchanges in which no member is used twice, so maximum-weight partial matching aggregates their bounds without enumerating all feasible allocations. Under the paper's conditions, at the prespecified confidence level, simultaneously for all feasible treated sets I\mathcal I,

0≤C(I)−R(I)≤2min⁡k,m−k,rmax⁡.0 \le C(\mathcal I)-R(\mathcal I) \le 2\min\\{k,m-k\\}\\,r_{\max}.

Here, C(I)C(\mathcal I) is the full-allocation Pairwise certificate, R(I)R(\mathcal I) is true total welfare loss against all feasible allocations, mm is cohort size, kk is treatment capacity, and rmax⁡r_{\max} is the largest sum of sampling radius and bias bound for an exchange. Each alternative requires at most min⁡k,m−k\min\\{k,m-k\\} exchanges. Because the bounds hold for all allocations at once, they also cover the allocation selected using the same trial.

Consequently, a learned allocation can carry a computable finite-sample loss guarantee supported by the trial, under the stated identification and model approximation conditions. Section 1 states the contributions specific to our setting, and Section 2 and Appendix F explain their relationship to established methods.

AuthorsOpenReview ↗

Response to Reviewer UfK2 (2/3)

[W3, Q2] How do certificates support allocation selection?

You can keep the allocation chosen by empirical welfare maximization and use the same trial to bound its loss under our assumptions. If this bound is at most your chosen loss threshold, the allocation is certified. Minimizing the certificate is optional and need not improve welfare.

1. In all three settings, certificate-based selection lowers mean certificates but has higher mean true regret than empirical welfare maximization.

Certificate-based selection minimizes the certificate, while the baseline, empirical welfare maximization, maximizes estimated welfare. Each setting has 1,000 paired trials using the same observations, cohort, candidates, effect estimator, and calibration. Neither rule uses true effects for fitting or selection. True effects are used only to evaluate regret, for which smaller values are better.

Table A. Comparison of selection rules.

SettingSelection ruleMean true regretMean certificate
Raw-featureCertificate-based selection0.068520.50083
Raw-featureEmpirical welfare maximization0.006960.55025
Nonlinear-featureCertificate-based selection0.138211.83616
Nonlinear-featureEmpirical welfare maximization0.071421.84839
SUPPORT2Certificate-based selection0.128960.90733
SUPPORT2Empirical welfare maximization0.023691.00375

2. The loss guarantee applies to empirical welfare maximization. Under certificate-based selection, small comparison errors imply welfare close to that of the best candidate.

For a prespecified candidate set S\mathcal S, let R(s)R(s) be allocation ss's total expected welfare loss relative to the best candidate and C(s)C(s) its certificate. Under the paper's conditions, R(s)≤C(s)R(s) \le C(s) holds simultaneously at the prespecified confidence level, including for a candidate chosen by empirical welfare maximization using the same trial. Certificate-based selection chooses

s^C∈arg⁡min⁡s∈SC(s).\widehat s_C \in \arg\min_{s \in \mathcal S}C(s).

Appendix E gives, on the same confidence event,

R(s^C)≤C(s^C)≤min⁡s∈SR(s)+2as,as=max⁡r∈Sars,R(\widehat s_C) \le C(\widehat s_C) \le \min_{s \in \mathcal S}\\{R(s)+2a_s\\}, \qquad a_s = \max_{r \in \mathcal S}a_{rs},

where arsa_{rs} is the sampling radius for comparing rr and ss. The candidate with highest true welfare, s⋆s^\star, has R(s⋆)=0R(s^\star) = 0, giving

R(s^C)≤2as⋆.R(\widehat s_C) \le 2a_{s^\star}.

With model approximation error, replace arsa_{rs} by ars+brsa_{rs}+b_{rs}, where brsb_{rs} bounds the combined trial and target bias over a specified class containing the actual deviations.

A smaller certificate need not mean higher welfare because it also accounts for uncertainty. You can choose a candidate by empirical welfare maximization and still certify its loss against the best candidate using the same trial.

AuthorsOpenReview ↗

Response to Reviewer UfK2 (3/3)

[Q3] Can the finite-sample guarantees extend to nonrandom treatment assignment?

Yes, with known treatment probabilities, conditional independence of treatment and potential outcomes given covariates, and the remaining model and sampling conditions, the certification argument can be adapted. Estimated probabilities require additional finite-sample error control. Thank you for raising this extension.

1. Known treatment probabilities give unbiased comparisons under overlap, conditional independence, and consistency.

Let XX denote pretreatment covariates, A∈0,1A \in \\{0,1\\} treatment, YY the observed outcome, and τ(X)\tau(X) the conditional average treatment effect. Assume treatment is independent of potential outcomes conditional on XX, and consistency holds, so the observed outcome equals the potential outcome under the treatment received. For the true treatment probability e(X)=Pr⁡(A=1∣X)e(X) = \Pr(A = 1\mid X),

E![AYe(X)−(1−A)Y1−e(X);|;X]=τ(X).\mathbb E\\!\left[ \frac{AY}{e(X)} -\frac{(1-A)Y}{1-e(X)} \\;\middle|\\;X \right] = \tau(X).

This inverse-probability weighting identity is discussed in Chapters 2–3 of [5]. Requiring η≤e(X)≤1−η\eta \le e(X) \le 1-\eta for known η>0\eta > 0 bounds the weights. Independent units, bounded outcomes, a prespecified cohort, representation, and comparisons, and the common effect model and identification conditions linking sample and target then permit finite-sample control of comparison errors. Simultaneous bounds support certification after selection and the matching construction.

2. Estimated treatment probabilities require their estimation errors to be included in the comparison bounds.

One possible extension would fit probabilities on independent data and incorporate finite-sample bounds on those errors under suitable model conditions. Thus, with known probabilities, the same certification construction applies under the stated causal, model, and sampling conditions. Estimated probabilities require a further finite-sample error analysis.

Thank you again for these helpful questions! We have improved the presentation, clarified the related work, and added the comparison of selection rules. We hope the revised manuscript and these responses answer your questions.

References

[1] Wilder, B., Dilkina, B., & Tambe, M. (2019). Melding the data-decisions pipeline: Decision-focused learning for combinatorial optimization. AAAI.

[2] El Balghiti, O., Elmachtoub, A. N., Grigas, P., & Tewari, A. (2019). Generalization bounds in the predict-then-optimize framework. NeurIPS.

[3] Fiez, T., Jain, L., Jamieson, K. G., & Ratliff, L. (2019). Sequential experimental design for transductive linear bandits. NeurIPS.

[4] Xu, C., Alaa, A., Bica, I., Ershoff, B., Cannesson, M., & van der Schaar, M. (2021). Learning matching representations for individualized organ transplantation allocation. AISTATS.

[5] Hernán, M. A., & Robins, J. M. (2020). Causal Inference: What If. Chapman & Hall/CRC.

I thank the authors for their response to my questions. My assessment of the paper remains the same.

Reviewer_Jc3AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies treatment allocation under capacity constraints, a problem consisting in selecting exactly kk individuals from a cohort, for treatment. The paper focus is on determining the gap in terms of welfare loss (akin to the 'value' observed) between a given allocation and the best feasible one. The paper first derives identifiability conditions, before presenting regret bounds. Results are valid evn when the same trial is used for both allocation selection and certification. The bounds are finite-sample, and results show how the obtained certificates are suboptimal. Certificates are then evaluated experimentally.

Critical strengths
  • S1) The setup and overall problem are very interesting, well formulated, and can yield useful practical applications in several domains.
  • S2) There has to be assumptions in order to have identification of the welfarre or pairwise welfare differences in the target cohort. Assumption 1 seems to be this assumption, and it is a parametric assumption on the CATE, which can thus be transported.
Critical weaknesses
  • The paper is very difficult to follow. It seems that there are a lot of unecesasry notations introduced. For instance, to introduce data and the target cohort, it takes 2 pages with a lot of notations. The main result (Theorem 2) is very far from being self contained, it would be best if assumptions and notations were clearly stated closer to it. The same goes for the whole section 5.
  • Related to S2) in the Strengths part, although it is necessary, this assumption is both untestable and unverifiable. The representation will in practice be mispecified. Thus, it seems reasonable to use orthogonal assumptions, as done for instance in [1] (for a different but related problem). In [1], parametric assumptions are made, but also weighted methods. Both can then be combined for double robustness in transportability.
  • In Equation 5, why center with ζ\zeta

[1] Causal Inference Methods for Combining Randomized Trials and Observational Studies: A Review, Colnet et al., Statistical Science, 2024.

Overall, because of how difficult the paper is to follow, I am leaning towards rejectance. The problem is interesting, but significant work is required for the approch to be conveyed in a clearer way. Additionally, the second bullet point above (different approaches for identifiability) should be addressed.

Questions for authors

See above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Jc3A (1/3)

Thank you for recognizing the relevance, formulation, and potential applications of the treatment allocation problem! We address the notation and presentation of the main results, the common treatment effect model and its relation to weighting and doubly robust methods, and the choice of center below.

[W1] Improving the paper's readability

We have simplified the notation, introduced our Pairwise method through three steps, and stated the relevant conditions alongside the certification guarantees in Section 5. Thank you for your concrete suggestions on readability.

1. We now explain Pairwise in three steps: estimate welfare differences, add error bounds, and take the maximum over alternatives.

Section 4 introduces these steps. Section 3 writes out the gap between the best feasible allocation and the best candidate in Equation (4), without a separate symbol.

2. In Section 5.1, Theorem 2 now states the trial assumptions, common effect model, and comparison identification condition together with its guarantee.

In words:

Fix the cohort, representation, and candidate set before observing trial assignments and outcomes. Assume consistency, no interference, independent trial units conditional on the fixed design, known randomization probabilities satisfying overlap, and bounded outcomes. Under the common effect model, require candidate feature differences to be linear combinations of trial features. Then, at the prespecified confidence level, all computed upper bounds on candidate welfare differences hold simultaneously. For any selected candidate, the maximum of its comparison bounds covers its loss relative to the best candidate, even when selection uses the same trial.

Section 5 also states the exchange identification and residual conditions in the full-allocation theorem.

Thus, the revised main text explains the construction with fewer symbols and places the relevant conditions alongside the certification guarantees.

AuthorsOpenReview ↗

Response to Reviewer Jc3A (2/3)

[W2] The common treatment effect model and misspecification

We agree that the common treatment effect model may be misspecified. We use it to obtain fixed comparison weights for the prespecified cohort and construct simultaneous finite-sample bounds. Our correction accounts for deviations in both the trial and target cohort, but the guarantee still depends on these deviations lying within the specified error bounds.

1. For comparisons identified by the trial, the common model gives fixed weights without requiring separate models for the two outcome means.

Trial and target treatment effects use the same feature functions and coefficient vector, while effects can vary across individuals. When an allocation feature difference is a linear combination of trial features,

Δrs=∑i=1nwrs,i,τ(Xi).\Delta_{rs} = \sum_{i = 1}^{n} w_{rs,i}\\,\tau(X_i).

Here, Δrs\Delta_{rs} is allocation rr's total expected treatment benefit minus that of ss, nn is the trial size, XiX_i is member ii's feature vector, and τ(Xi)\tau(X_i) is the conditional average treatment effect. The weights wrs,iw_{rs,i} depend on the features and allocations and are fixed before certification assignments and outcomes. Known randomization probabilities give an unbiased estimate. With bounded outcomes, simultaneous error control then provides a loss guarantee after selection using the same trial.

2. The doubly robust transport results we cite concern consistent estimation of a target population's average effect. This consistency guarantee alone does not give the simultaneous finite-sample bounds needed for welfare comparisons in our fixed cohort.

Under the transportability and overlap conditions in Colnet et al. [1], consistency holds when either the model of who enters the trial or the outcome models are correct. Our fixed comparison weights do not require fitting such a participation model. We use these weights with known randomization probabilities for finite-sample calibration, then correct the trial and target model deviations below.

3. Bounding both trial and target deviations preserves certification when the specified range contains the actual deviations.

Trial deviations affect the estimated welfare difference, while target deviations affect its true value. Accounting for both gives

Urs=Δ^rs+ars+brs.U_{rs} = \widehat{\Delta}_{rs} + a_{rs} + b_{rs}.

Here, UrsU_{rs} bounds Δrs\Delta_{rs}, Δ^rs\widehat{\Delta}_{rs} is its estimate, arsa_{rs} controls sampling fluctuations, and brsb_{rs} bounds the absolute bias over the specified joint range of trial and target deviations. Appendix B.3 gives simultaneous coverage, including after selection, when this range contains the actual deviations and the remaining trial and identification conditions hold. The deviations may be nonzero and need not vanish with sample size. An allocation is certified only if its corrected certificate meets the loss threshold. Randomization does not verify the effect model or the assumed deviation bounds.

The common model gives us fixed weights for comparing allocations. Correcting trial and target bias then keeps the guarantee valid when their actual deviations lie within the specified bounds. Section 4 states the conditions for the bias correction, and Appendix D.7 reports the Hillstrom sensitivity results. Appendix F compares our guarantee with those of doubly robust transport methods.

AuthorsOpenReview ↗

Response to Reviewer Jc3A (3/3)

[W3] Why subtract the center ζ\zeta in Equation (5)?

Centering at the midpoint of the known outcome range preserves the mean of the inverse-probability weighted (IPW) score and reduces its sampling radius. Thank you for asking us to explain this choice. For assignment Ai∈0,1A_i \in \\{0,1\\}, known treatment probability eie_i, outcome Yi∈[0,1]Y_i \in [0,1], and prespecified center ζ\zeta, the score is

Ti(ζ)=Ai(Yi−ζ)ei−(1−Ai)(Yi−ζ)1−ei.T_i^{(\zeta)} = \frac{A_i(Y_i-\zeta)}{e_i} - \frac{(1-A_i)(Y_i-\zeta)}{1-e_i}.

Under randomization, the same ζ\zeta is subtracted from both conditional outcome means, so their difference is unchanged. For balanced assignment, ei=1/2e_i = 1/2,

Ti(0)∈[−2,2],Ti(1/2)∈[−1,1].T_i^{(0)} \in [-2,2], \qquad T_i^{(1/2)} \in [-1,1].

Thus, at ζ=1/2\zeta = 1/2, the range length and Hoeffding sampling radius are halved with balanced assignment and the same weights and confidence level. The midpoint also minimizes range length for general known treatment probabilities, without fitting the center from outcomes. Section 4.1 explains the unchanged mean and midpoint choice next to Equation (5). Appendices B.2 and D.3 give the range comparison and the halved Hoeffding radius under balanced assignment.

Thank you for your careful reading! The revised manuscript uses fewer symbols, states the conditions near the main results, and explains model misspecification and centering. We hope this addresses your concerns.

References

[1] Colnet, B., Mayer, I., Chen, G., Dieng, A., Li, R., Varoquaux, G., Vert, J.-P., Josse, J., & Yang, S. (2024). Causal inference methods for combining randomized trials and observational studies: A review. Statistical Science, 39(1), 165–191.

I thank the authors for their very detailed answers, but maintain my score and overall assessment of the paper.

Reviewer_WjewOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies finite sample regret certification for kk treatment allocation in a prespecified target cohort using evidence from randomized trial. The method characterizes identifiable allocation contrasts and constructs simultaneous confidence bounds. It also extends to all feasible allocations through maximum weight partial matching.

Critical strengths
  • The theoretical results provide some interesting insights. The authors show that allocation comparisons can remain identifiable even when individual target effects are unidentified. Also, the results clarify which information is actually needed for certification.
  • The simultaneous finite-sample guarantees accommodate allocation selection using the same trial and the exchange-matching construction avoid enumerating all allocations. Such methodology design and theoretical insights provide practical guidance.
Critical weaknesses
  • The methodological novelty appears to be on the weaker side. The novelty beyond existing decision-certifcation and direct-constrast methods are somewhat incremental. It seems the matching components is the new algorithmic addition, but its loss of tightness relative to direct joint-region certification is not studied in the experiments.
  • The main experiment use three feature representation, while the main Hilstrom certificates assume zero model residual without empirical validation. The generalizability of the reported results to settings with approximation error or richer representations remain unclear. Also, the paper did not provide any supplementary code files.
Questions for authors
  • Could the authors clarify the major methodological novelty beyond applying established simultaneous contrast bounds and confidence-set projection to exact k allocation?
  • For the matching construction, could the authors quantify its rightness and runtime relative to direct joint-region regret certification in the existing experiments?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Wjew

Thank you for recognizing the identification results for allocation welfare differences, the simultaneous finite-sample guarantees, and the matching construction that avoids enumerating all allocations! We address the methodological contribution, the tightness and runtime of matching, and certification with additional features and model approximation error below.

[W1, Q1] Our specific technical contribution relative to existing methods

We derive a joint correction for trial and target model error and a matching construction that covers all feasible allocations. We prove the resulting finite-sample guarantee and bound how far the certificate can exceed true regret. Pairwise uses standard maximum-weight partial matching to aggregate exchange bounds. Our new argument establishes why this construction gives a valid full-allocation certificate and bounds its excess over true regret.

Certified Decisions [2] provides the general framework for loss certification, and Jeunen and Ustimenko [1] estimate policy value differences directly. Applying these ideas to a different target cohort first requires determining which welfare differences the trial identifies. Proposition 1 applies the standard linear model criterion and constructs the comparison weights. A shared unidentified effect component can cancel between allocations treating the same number of people.

1. Trial model bias can invalidate certification despite randomization. Correcting the combined trial and target bias preserves validity when the specified error range contains the actual deviations.

Trial deviations affect the estimate through its weights, while target deviations affect the true welfare difference. We derive their combined bias and correct it over a prespecified joint range. Theorem 4 then gives simultaneous candidate comparison bounds, including after selection using the same trial. Appendix B.3 shows why controlling sampling error alone is insufficient: even with randomized assignment and zero target error, omitting trial bias can make the bound fall below the true difference with probability one.

2. With fixed treatment capacity, feasible alternatives correspond to partial matchings. Given simultaneous exchange bounds, this yields a polynomial-time certificate with bounded excess over true regret.

Direct joint-region certification maximizes loss over a parameter confidence region and feasible alternatives. Our construction represents each alternative by exchanges in which no member is used twice. Maximizing the sum of simultaneous exchange bounds over partial matchings therefore covers all feasible alternatives.

Theorem 3 establishes why its output is valid simultaneously for all allocations, including one selected using the same trial. Corollary 5 bounds its excess over true loss using the maximum number of exchanges and their error bounds. Given the exchange bounds, computation is polynomial in cohort size.

Together, these arguments connect trial evidence to regret in the target cohort and provide a computable loss bound that remains valid after selection under explicit identification and model approximation conditions. Section 1 states the contributions specific to our setting, and Section 2 explains their relationship to established tools.

AuthorsOpenReview ↗

Response to Reviewer Wjew (2/3)

[W1, Q2] Tightness and runtime of the matching construction

Full-allocation Pairwise gives more conservative bounds in the paired comparisons. For one fixed SUPPORT2 trial, it provides the initial bound quickly, while further joint optimization does not converge within the budget of 600 seconds at eight and eleven features. Thank you for suggesting this comparison. The matching construction you refer to is the full-allocation version of our Pairwise method. We compare it with direct joint-region regret certification.

Both methods evaluate the same selected allocation against the best of all feasible allocations, using the same trial observations, target cohort, fitted model, and parameter confidence region. The experiments use 6,000 trial participants, a cohort of 500, and allocations treating exactly 20 people.

1. The mean full-allocation Pairwise bound is 0.5468–0.9851 higher in the three settings with converged paired joint-region results.

Pairwise bounds each one-person exchange separately and aggregates these bounds through maximum-weight partial matching. The joint-region baseline uses one common parameter value for all exchanges. Consequently, allowing different maximizing values in the separate exchange bounds makes Pairwise more conservative.

Table A. Mean welfare loss bounds against all feasible allocations, with 100 paired instances per setting and three features. The difference is the Pairwise bound minus the direct joint-region bound.

SettingPairwise (ours)Direct joint-region certificationDifference
Raw-feature simulation2.489431.734450.7550
Nonlinear-feature simulation4.171313.186210.9851
SUPPORT2 semisynthetic2.858482.311650.5468

The additional converged five-feature instance gives the same ordering: full-allocation Pairwise and direct joint-region bounds are 5.5617 and 2.9068, respectively.

2. For one fixed SUPPORT2 trial, the median time to the initial Pairwise bound is below 0.2 seconds at three, five, eight, and eleven features. The joint solver converges at three and five features but remains unconverged within 600 seconds at eight and eleven features.

Table B uses the same trial, target cohort, and selected allocation across dimensions, with zero model residual and a common 95% parameter confidence ellipsoid within each dimension. Each entry is the median of three repetitions. Both columns measure time from the start of the same joint run, first to the initial Pairwise bound and then to the end of joint optimization. The three-feature timings come from an earlier batch of completed runs, and the remaining dimensions use a budget of 600 seconds. All three- and five-feature runs converge. Entries of 600 indicate that all three repetitions stop without convergence under that budget.

Table B. Elapsed times within the same joint certification runs, in seconds. The Pairwise column reports time to the initial bound, and the joint-region column reports total time or the time limit.

Feature dimensionPairwise (ours)Direct joint-region certification
30.1130.301
50.11727.134
80.140Not converged (600 s limit)
110.191Not converged (600 s limit)

Thus, full-allocation Pairwise quickly gives an upper bound, while direct joint-region optimization can tighten it through further computation. Section 6 summarizes this comparison. Appendix G.1 reports the three-feature tightness results and the timing comparison at three, five, eight, and eleven features within the same joint runs.

AuthorsOpenReview ↗

Response to Reviewer Wjew (3/3)

[W2] Certification with additional features and model deviations in Hillstrom

In the tested SUPPORT2 feature expansion, Pairwise gives tighter certificates over 19 candidates under Hoeffding calibration. The simulations with nonzero model deviations retain coverage. The Hillstrom analysis shows how much model error the two certificates can tolerate. We will make the code publicly available upon acceptance of the paper.

1. With 11 model features and unchanged true effects, Pairwise certifies 58.7% of instances at loss threshold 2 among the 19 candidates, while neither baseline certifies any.

We compare 3, 5, 8, and 11 features in SUPPORT2 with 6,000 trial participants, 500 cohort members, and allocations treating 20 people. Each of five prespecified sets of trial covariates, target cohorts, and candidates has 200 treatment and outcome realizations, giving 1,000 trials per dimension. Across dimensions, the observations, true effects, 19 candidates, and allocation selected using the original representation with three features stay fixed, isolating the cost of estimating additional parameters.

Pairwise directly bounds candidate welfare differences. The baselines Effect and Welfare aggregate individual treatment effect intervals and allocation welfare interval endpoints, respectively. All three use the same data and effect estimator with their own 95% Hoeffding calibration. Each bounds total welfare loss against the best of the 19 candidates.

Table C reports certification rates at loss threshold 5, with regret measured against the best of 19 candidates. With 11 features at threshold 2: Pairwise 58.7%, Effect 0%, Welfare 0%.

Feature dimensionMethodMean certificateCertification (%)
3Pairwise (ours)1.004100
3Effect3.075100
3Welfare2.810100
5Pairwise (ours)1.222100
5Effect3.98795.6
5Welfare3.008100
8Pairwise (ours)1.608100
8Effect5.58620.1
8Welfare3.303100
11Pairwise (ours)1.932100
11Effect7.1010
11Welfare3.67199.5

The gain comes from tighter bounds.

2. The finite-sample guarantee also covers nonzero model deviations when the prespecified error range contains the actual deviations.

(a) With the specified shared-coefficient residual class, full-allocation Pairwise has 100% observed coverage in all four mechanisms. At threshold 5, model deviations only in the trial and partial cancellation permit certification in 24.5% and 98.8% of instances, respectively.

In these simulations, one common coefficient describes trial and target deviations, and its specified range contains the true value. Table D reports Pairwise against all feasible allocations with 6,000 trial participants and 95% Hoeffding calibration. Coverage is the proportion of bounds covering true welfare loss. Certification is the proportion at most 5.

Table D. Coverage and certification rates under nonzero model deviations, in percent.

Model deviationsCoverageCertification
Trial only100.024.5
Target only100.00.0
Partial cancellation100.098.8
Reinforcement100.00.0

Thus, some settings with nonzero model error still permit certification at threshold 5. With separate bounds on each individual's deviation, coverage is also 100%, but certification is 0% in all four settings.

(b) Sensitivity analysis shows that one Hillstrom visit split and one conversion split remain certified with allowed absolute model deviations up to 0.659% and 0.614%, respectively.

We fix the data and selected allocation, then increase the allowed absolute treatment effect model error for each trial and target member. Table E gives the error limits on the probability scale that keep the loss bound over all feasible allocations at most 5.

Table E. Hillstrom splits certified at zero model error and the deviation limits under which they remain certified.

OutcomeCertified splitsDeviation limit
Visit1 / 30.659%
Conversion1 / 30.614%

Together, these results demonstrate useful candidate-set certification in the tested feature expansions and valid certification under explicit nonzero deviation bounds. See Section 6.2 and Appendix D.1 (Table 2) for model error simulations, Appendix D.7 for Hillstrom sensitivity, and Appendix G.2 for added features.

Thank you again for these suggestions! We have clarified the contributions and added the comparisons in Appendix G. We hope these changes address your concerns.

References

[1] Jeunen, O., & Ustimenko, A. (2024). Δ-OPE: Off-policy estimation with pairs of policies. ACM Conference on Recommender Systems.

[2] Andrews, I., & Chen, J. (2025). Certified decisions. arXiv preprint arXiv:2502.17830.

Thank the authors for the detailed responses to my questions. I have raised my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.