- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
From our understanding, the authors use outcomes from previous cohorts to compare candidate randomization schemes in terms of efficiency of ATE estimation. Since it is not possible to (universally) unbiasedly estimate the risk associated with one design from a single cohort without restrictions on the potential outcomes, the authors use an independently randomized cohort to estimate risk difference, and then use this result to guide selection of the randomization scheme for subsequent cohorts.
Critical strengths
The theoretical results appear to be mathematically sound.
Critical weaknesses
First, Theorem 3.1, which is presented as one of the main contributions of the paper, is a special case of Proposition 1 in Harshaw, Middleton, and Savje (2026). Although the submission does cite Harshaw, Middleton, and Savje (2026), it does not explicitly identify Theorem 3.1 as this specialization of Proposition 1. Second, we have found numerous issues in the experiments. Most importantly, the submission's procedure is often found to be worse than the comparators (sometimes significantly so). First, the improvement over vanilla CRL is generally small, especially when compared with the gap to the polynomial plug-in. In Table 6, the contextual risk ratios relative to Bernoulli assignment are 0.222 for vanilla CRL, 0.212 for PA-CRL with a "polynomial hint", and 0.079 for the polynomial plug-in. PA-CRL also does worse than the polynomial plug-in in all 12 pxT cells (Figure 7b) and on the WDBC and Bike datasets (Table 6). Importantly, the polynomial plug-in is not an oracle, since it uses current covariates and previously observed outcomes, without access to current potential outcomes or risks (Appendices J.2 and K). One might then ask, why not simply use the polynomial plug-in in these settings? We understand that PA-CRL provides a robust regret guarantee, but the additional guarantee does not seem to justify the trade-off with the loss in precision. Further, several experiments simply verify algebraic identities or that simulated regret lies below a loose upper bound. These do not really establish that the method chooses better designs. For reference, in Appendix L (Figure 10), the reported regret is about 0.2 under a bound of about 1.7 at T=256. Ultimately, these experiments do not add anything substantive besides simple checks. Moreover, Figure 3b highlights the favorable comparisons, while graying out the unfavorable ones. On the other hand, Table 8 shows losses as large as 190.8% relative to the plug-in, which are instead omitted from Figure 3b. These values should instead be shown together with the favorable ones in Figure 3b.
Generally, the submission is very hard to read, and the lack of clarity significantly obscures the paper's contributions, if any. We did not identify mistakes in the derivations we checked, although the theory currently reads as a central observation (which is a direct specialization of an existing result) embedded into an overextended framework. Finally, the experiments leave a lot to be desired.
Questions for authors
-
Considering that polynomial plug-in significantly outperforms PA-CRL in many settings, why should a practitioner choose PA-CRL? Does the robust regret guarantee practically justify such a substantial loss in efficiency?
-
Could the authors clarify what Figure 10 tells us about practical design-selection performance, beyond validating the implementation and regret decomposition? In particular, how do the observed regret and theoretical bound compare with the regret of simple baselines at the same horizons, and does the bound actually certify an improvement over any of these baselines?
4 comments
Thanks for the review and here is the kind response(cont'd)
k when only one potential outcome is observed.
W2a(ii) / W2c: Complete signed comparison under misspecification.
The matched linear-predictor experiment measures the benefit of correcting structural ranking errors. Its complete grid is below. Entries are : positive favors PA-CRL; negative favors the raw linear plug-in.
| Interaction weight | ||||
|---|---|---|---|---|
| 0 | −190.83 | −164.53 | −148.85 | −137.43 |
| .125 | −177.72 | −147.24 | −130.30 | −117.82 |
| .250 | −115.14 | −92.44 | −79.17 | −69.72 |
| .375 | −57.07 | −45.17 | −39.21 | −34.25 |
| .500 | −19.91 | −11.33 | −6.66 | −1.49 |
| .625 | +3.36 | +13.37 | +20.69 | +26.28 |
| .750 | +17.41 | +27.51 | +33.76 | +38.25 |
| .875 | +22.19 | +32.45 | +37.98 | +42.25 |
| 1 | +17.19 | +24.54 | +29.82 | +31.55 |
This uses every one of the 73,728 method/history records: eight populations and 128 histories per population, cell, and method. There are 16 favorable and 20 unfavorable point estimates; pointwise paired intervals exclude zero in all 16 favorable and 19 unfavorable cells. Figure 3b now shows every signed value on one symmetric color scale, including the 190.8% loss. Figure 3e includes all nine interaction-weight trajectories. The favorable region therefore has an explicit boundary: correction pays off under sufficiently strong interaction misspecification, while accurate ranking makes direct modeling more efficient.
W2a(iii) / Q1: Independent comparisons against the strong polynomial.
The new mechanism study fixes p = 8, T = 512, n = 12, 24, 48, one or three observed contexts, and three coefficient spectra. Each of six methods selects one global configuration from eight candidates using risk/Bernoulli on six simulator-calibration settings, two populations and 32 histories, then freezes it before the complete 36-cell test on five fresh populations and 128 histories each. Random and fixed coefficients share the same coefficient norm, expected uniform-X response RMS and outcome bound. All learners receive the same covariates/context and actual law information, and only their own completed outcomes.
In the prespecified random18 family, PA minus strong-polynomial risk/Bernoulli is −.078532 [−.095458, −.061605]; each population first averages all 18 cells equally, followed by a paired interval across populations. PA has lower risk in 18/18 cells, with 14 pointwise intervals below zero. The polynomial plug-in leads in all 18 fixed controls; the combined all36 difference is +.000175 [−.008402, .008751]. Every-cohort pair-IPW and the full PSD second-moment plug-in have lower risk than PA in all 36 cells. This mean-zero family has tied mean rankings; the unique-ranking, nonzero-mean certificate in N.6 is a separate analytical construction.
In the earlier independent ten-setting study, calibrated PA-CRL improves on separately calibrated vanilla CRL in 10/10 terminal point estimates, with nine paired intervals favoring PA; the polynomial plug-in leads in 9/10 settings (Appendix N.5). The original fixed ε = .1, η = 4 study remains in Appendix N.3.
A separate frozen-parameter transfer reused ε = .01 and η = 256 with the original matched linear hint on four fresh populations and retained all 36 cells. PA-CRL had lower risk than the matched linear plug-in in 18/36 cells, with 17 pointwise paired 95% intervals favoring PA-CRL; the strong archived polynomial plug-in had lower risk in all 36 (Appendix N.5). This additional test is distinct from the original eight-population grid above.
3. Q1: Quantifying the efficiency cost of reference cohorts.
Let be PA-CRL's total risk ratio, its non-reference policy's risk ratio along the recorded history, and the polynomial plug-in ratio. With ,
The first term is the direct reference surcharge; the second is the selection gap. The original three-context campaign gives:
| PA-CRL | Polynomial | Non-reference | Reference surcharge | Selection gap | |
|---|---|---|---|---|---|
| 512 | .254402 | .081974 | .171558 | .082844 | .089584 |
| 1024 | .212033 | .079389 | .124481 | .087552 | .045092 |
At , direct reference cost accounts for 66.0% of the .132644 gap. This exact accounting holds the learned policy path fixed; changing the reference rate also changes subsequent feedback and learning. For a practitioner, the measured comparison is therefore between the observed benefit of correcting wrong rankings and this explicit precision cost.
4. W2b / Q2: Same-horizon baselines and the Figure 10 certificate.
We evaluated the simple baselines on exactly Figure 10's frozen two-policy instance, with , , , , and the exact contrast as the predictable hint. Entries are cumulative normalized-risk regret against the pair design; each OMD estimate uses 30,000 paths
Thanks for the review and here is the kind response(cont'd)
.
| PA-CRL | Zero-hint OMD | Theorem bound | Independent | Uniform | Pair / correct-ranking plug-in | |
|---|---|---|---|---|---|---|
| 16 | .047117 | .057085 | .162918 | .139375 | .069687 | 0 |
| 32 | .065112 | .090447 | .268073 | .278750 | .139375 | 0 |
| 64 | .081955 | .125468 | .478384 | .557500 | .278750 | 0 |
| 128 | .109953 | .160541 | .899007 | 1.115000 | .557500 | 0 |
| 256 | .165703 | .216781 | 1.740251 | 2.230000 | 1.115000 | 0 |
The bound certifies improvement over always-independent assignment at . It exceeds uniform-selection regret and correct-ranking plug-in regret at every displayed horizon. Observed PA-CRL regret improves on zero-hint OMD and uniform selection; a correctly ranking plug-in chooses the best design immediately and has zero regret. At , PA-CRL's .165703 regret separates into .111500 reference excess and .054203 exploitation excess. Figure 10 thus identifies the source of the guarantee and its looseness; the learned-predictor experiments establish practical design-selection performance under the declared model specifications.
5. W3: Readability and presentation.
Section 2 adds the operational sequence before the formal notation: predict relative risks, select a law, collect occasional reference feedback, and correct the next policy update. The revision places the HMS specialization immediately beside Theorem 3.1 and explains feedback computation, policy learning, and inference by their operational roles. Section 7 now begins with competitive design-selection comparisons; the new appendix puts the complete signed grid, reference-cost decomposition, and matched-horizon baselines together. This gives a direct path from each claim to its assumptions and numerical evidence.
The blue revision makes the attribution explicit in Sections 1, 3–5, shows the complete Figure 3 comparison in Section 7, and reports protocols, full results, and reference-cost accounting in Appendix N.
We are eager to know whether we have addressed your questions, and we would be happy to discuss further at any time to assist with your re-evaluation.
Just like Reviewer LJfs, I have the same impression that the response is purely AI-generated. It is effectively unreadable. I will not change my score.
Thanks for the review and here is the kind response
Design precision depends on the conditional second moment, including cross-unit covariance Ω. A polynomial model may estimate u = E[μ|G] consistently while the risk ℓₕ = (uᵀΣₕu + tr(ΣₕΩ))/(4n²) favors a different law. Appendix N.6 gives a general criterion: with a unique mean-greedy ranking, an expected γ-per-cohort gap to a fixed admissible policy, and mean-selection error probabilities pₜ, PA-CRL’s expected cumulative advantage is at least γT − Σₜpₜ − B_T. Thus γT > Σₜpₜ + B_T certifies lower risk. An exact n = 4 construction has a nonzero linear mean and unique wrong mean-greedy choice. At T = 512, ε = .05 and η = .411325, the theorem bounds PA risk by .030746, below the mean-oracle plug-in’s .070313 and independence/uniform’s .039063. Both methods receive the correct mean; the certificate uses dense reference scores and analytical η. A fixed-parameter reproduction with five fresh sign paths and 4,096 histories each gives learned-hint PA risk .015034 versus learned polynomial’s .069511: 78.37% [78.18%, 78.56%] lower. The .030746 bound covers the exact-mean-hint variant (Appendix N.6).
The separate mean-zero latent-cohort test finds lower PA-CRL risk than the strong polynomial in 18/18 random-coefficient settings, with 14 pointwise paired 95% intervals favoring PA. The mean cell-specific reduction is 15.04% [11.34%, 18.75%]. All 18 fixed-coefficient controls and both enhanced baselines remain in the complete 36-cell report (Appendix N.7).
The original matched-linear experiment identifies a complementary robustness regime: PA-CRL reduces risk by 3.4–42.3% in all 16 cells with interaction weight β≥.625, with every pointwise seed-level 95% interval excluding zero. We added the complete signed comparison, quantified reference cost, and evaluated the requested same-instance baselines for Figure 10. The strong polynomial plug-in appears alongside PA-CRL throughout.
For navigation, W1–W3 denote the review's theory, experiment, and presentation concerns; W2a–c separate baseline performance, diagnostic evidence, and signed-grid presentation. Q1–Q2 retain the order of Questions for Authors.
1. W1: Theorem 3.1 and the contribution beyond estimability.
Theorem 3.1 is precisely the independent-reference specialization of Proposition 1 in Harshaw, Middleton, and Sävje (2026). We now state this relationship immediately before the theorem and in the contribution statement. Writing μ=a+b, the quadratic form expands as
μ⊤Aμ=a⊤Aa+2a⊤Ab+b⊤Ab.
Under independent fair assignment, the incompatible potential-outcome pairs are exactly (ai,bi), whose cross-product coefficients are 2Aii. Thus the HMS criterion becomes diag(A)=0. Every centered-design difference A=Σh−Σg satisfies this condition automatically.
The scientific question is how to use that comparison information to choose the next cohort's assignment law while preserving treatment-effect inference. Three steps make this operational. First, ghost assignments compute unbiased contrasts from sampling access to the candidate laws, avoiding dense covariance construction. Second, predictable contrast forecasts are corrected with randomized reference feedback; the resulting variance calculation identifies both reference noise and forecast error. Third, the corrected feedback gives fixed-policy regret and, through cohort-size weighting, excess-MSE control, while fresh centered deployment preserves unbiased effect estimation. The two-unit reversal in Proposition 3.3 explains why feedback identification matters: deployment-correlated scoring changes a true contrast of +1/32 into an expected score of −1/4, reversing the preferred design. We explicitly attribute the augmented inverse-propensity correction and optimistic mirror descent to their established sources. The contribution is their constructive use for outcome-relevant assignment-law learning, with the feedback and inference conditions stated together.
2. W2a / Q1: Competitive design selection and the operating regime.
W2a(i): A feasible strong baseline.
The complete evidence distinguishes two questions. Adding polynomial forecasts improves the audited learner over vanilla CRL in all 17 original controlled synthetic cells, by 1.83–10.28% (median 5.66%). Comparing against the strong polynomial plug-in yields a different ordering: that plug-in leads in all 12 phase-grid cells and all five contextual horizons. At T=1024, the three-context risk ratios are 0.222 for vanilla CRL, 0.212 for PA-CRL, and 0.079 for the polynomial plug-in. These values now appear directly with the main comparison. The polynomial selector uses current covariates and its own completed observed outcomes. In the original zero-effect replays, PA-CRL/polynomial risk ratios are .438/.356 on WDBC and .810/.656 on Bike (Section 7). The added heterogeneous-effect replays retain that feasible comparator and test reference feedbac