- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper considers sequential prediction sets that influence treatment and thereby change the distribution of later outcomes. SCCP separates the threshold that determines treatment from the cutoff used to evaluate coverage, identifies stagewise deployment coverage using historical-policy trajectory-prefix importance weights, and then evaluates the diagonal where the two thresholds coincide. It recursively selects a threshold at each stage by minimizing estimated prediction-set size subject to an estimated coverage constraint. The theory establishes asymptotic stagewise marginal coverage for fixed finite grids and a fixed horizon under overlap, bounded cumulative likelihood ratios, accurate historical-policy estimation, and availability of a complete feasible schedule. Experiments use a synthetic environment and four controlled semi-synthetic environments constructed from clinical datasets.
Critical strengths
S1. The problem formulation and correction account for the full relevant treatment prefix. Equations (7)–(11) distinguish coverage evaluation from policy induction, and the likelihood ratio includes both the current action and preceding actions. This is important: correcting only the last treatment does not generally recover a state distribution shaped by earlier decisions. The recursive construction follows this temporal structure, so later choices do not invalidate earlier-stage coverage constraints.
S2. The paper makes several important theoretical and empirical qualifications explicit. Appendix A controls selection over all finite candidate prefixes rather than relying on pointwise consistency for a data-selected threshold. It also separates schedule availability from the validity of a returned schedule. The independent pilot grid, calibration-size and horizon sensitivity experiments, and effective-sample-size diagnostics are useful. The study therefore goes beyond merely reporting average coverage in one feedback setting.
Critical weaknesses
W1. The experiments largely test calibration with a known simulated treatment mechanism, leaving the central historical-data difficulty insufficiently evaluated. Appendix D.2 states that after fitting the clinical components, all pilot, calibration, and evaluation trajectories are generated from a fixed controlled environment. In particular, the fitted historical policy generates the simulated actions and is also available when constructing importance weights. This does not test estimation of an unknown logging policy from a finite observational sample, although the theory requires accurate estimated prefix weights. Figure A8 and the horizon experiments do address weight concentration, but not systematic propensity-estimation error, which can accumulate across stages even when individual propensity errors are modest.
The feedback family is also specified directly as a known function of the scalar threshold: in the clinical environments, it interpolates between the historical policy and a fixed alternative policy (Appendix D.2, pp. 23–24). It does not learn clinicians' responses to the content of a patient's prediction set. This is a legitimate controlled experiment, but it makes offline identification substantially easier than the motivating clinical use. The invariant transition kernel additionally requires an adequate observed state/history representation and excludes unobserved treatment–outcome confounding; the importance-weight identity does not establish these conditions from clinical records.
The current evidence consequently supports self-consistent calibration under a specified simulator and known response family. To support the broader historical-data claim, I would expect at least an experiment that hides and re-estimates the logging policy on separate simulated data, together with controlled misspecification of the threshold-to-treatment response. This can be done within the existing benchmarks and need not require prospective patient data.
W2. The baseline configurations and uncertainty summaries leave the incremental advantage unresolved. The MFCS adaptation uses a history depth of three, cumulative-weight clipping at 40, and a single scale multiplying a fixed stage profile, whereas SCCP uses full prefixes and stage-specific thresholds without clipping (Appendix D.2, p. 22). These choices differ precisely along the dimensions the ablations identify as consequential. The resulting comparison does not isolate how much of SCCP's advantage comes from self-consistency, fuller history, or a more flexible threshold schedule. This is a concern about the implemented comparator, not a claim that the original MFCS theory directly solves the present setting.
The closest policy-coupled work is discussed, including Zheng and Jin (2026), but the experiments do not establish a direct one-stage comparison or identify which of its construction's requirements prevent a matched comparison. Historical-policy transport is also an established starting point, as illustrated by Zhang et al. (2023). The new contribution is therefore most persuasive as the longitudinal self-consistent construction; the off-diagonal representation alone does not establish a new estimation principle.
Finally, classifying a method as “WSC-qualified” using mean empirical worst-stage coverage at or just above 90% is stronger than the uncertainty warrants. Table 1 reports SCCP values such as 90.0 ± 0.2% and 90.2 ± 0.7%, not a guarantee that each calibrated schedule reaches 90%. The Appendix A.2 bound is useful, but depends on a potentially large cumulative-weight bound and all candidate prefixes, and is not evaluated numerically. The statement that SCCP yields the smallest width among methods reaching nominal coverage should be treated as a comparison of reported point estimates unless supported by coverage uncertainty or a risk–width frontier at matched reliability. Stagewise marginal coverage also does not establish better clinical treatment decisions, which the title and conclusion should continue to distinguish.
Questions for authors
Q1 — W1, essential robustness evidence. In the current semi-synthetic protocol, is the policy used in the weights exactly the policy that generates calibration actions? If so, can you hide that mechanism and re-estimate it from an independent finite logging sample, reporting propensity error, stagewise effective sample size, and coverage as the treatment horizon grows?
Q2 — W1, scope and mechanism clarification. What information about the threshold-induced clinical response must be known before calibration? How sensitive is SCCP to modest perturbations of that response during deployment? Please also specify whether the state used in the theoretical factorization represents the complete observed history or an assumed sufficient summary. These answers determine the interpretation of offline identification from clinical trajectories.
Q3 — W2, fair comparison. Can you separate the effects of MFCS history depth, clipping, and threshold-profile restrictions by varying them under a shared prediction model, grid, and calibration budget? Where a matched adaptation is not theoretically justified, please explain the limitation rather than interpreting the configured baseline's result as a general inferiority result.
Q4 — W2, novelty and qualification of claims. What does SCCP add in the T=1 special case relative to the policy-coupled construction already cited? For the sequential experiments, can you report the frequency and uncertainty of genuine population-coverage shortfalls across fitted schedules, and compare widths at matched coverage rather than qualifying methods solely by a mean WSC point estimate?
Thank you for the additional experiments and responses. However, the issues of dependence on a specified treatment-response model and undercoverage under modest misspecification remain, so I have decided to keep my original score.