acceptodds
Under review as a conference paper at ICLR 2027

Self-Consistent Conformal Prediction: Reliable Decisions under Performative Feedback

Abstract

Conformal prediction affords finite-sample marginal coverage under exchangeability, making it attractive for medical decision support. Yet in sequential care, when a prediction set contains clinically concerning outcomes, clinicians may be more likely to initiate preventive treatment or intensify an existing intervention, changing the very outcome the model aims to predict. This can break the exchangeability assumption underlying the coverage guarantee. A seemingly simple workaround is to ignore this feedback and calibrate under the historical treatment process. However, ignoring treatment feedback can lead to systematic coverage errors at deployment, so the nominal coverage level may no longer provide a reliable measure for clinical decision making. For all that, we show that this feedback can nevertheless be handled before deployment. We propose Self-Consistent Conformal Prediction (SCCP), a method for reliable uncertainty quantification in sequential care under performative feedback. The central difficulty is that the deployment distribution itself depends on the threshold being calibrated. We formulate this coupling as a self-consistent calibration problem, show that its stagewise coverage can be identified from historical trajectories, and use this result to construct a threshold schedule for the treatment process induced by its own use. We prove asymptotic stagewise marginal coverage without requiring post-deployment outcomes. Experiments on one synthetic benchmark and semi-synthetic treatment settings built from four clinical datasets show that SCCP maintains coverage close to the nominal level while retaining informative prediction sets.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Sequential care, decision making, healthcare, conformal prediction, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_Gh15OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper considers sequential prediction sets that influence treatment and thereby change the distribution of later outcomes. SCCP separates the threshold that determines treatment from the cutoff used to evaluate coverage, identifies stagewise deployment coverage using historical-policy trajectory-prefix importance weights, and then evaluates the diagonal where the two thresholds coincide. It recursively selects a threshold at each stage by minimizing estimated prediction-set size subject to an estimated coverage constraint. The theory establishes asymptotic stagewise marginal coverage for fixed finite grids and a fixed horizon under overlap, bounded cumulative likelihood ratios, accurate historical-policy estimation, and availability of a complete feasible schedule. Experiments use a synthetic environment and four controlled semi-synthetic environments constructed from clinical datasets.

Critical strengths

S1. The problem formulation and correction account for the full relevant treatment prefix. Equations (7)–(11) distinguish coverage evaluation from policy induction, and the likelihood ratio includes both the current action and preceding actions. This is important: correcting only the last treatment does not generally recover a state distribution shaped by earlier decisions. The recursive construction follows this temporal structure, so later choices do not invalidate earlier-stage coverage constraints.

S2. The paper makes several important theoretical and empirical qualifications explicit. Appendix A controls selection over all finite candidate prefixes rather than relying on pointwise consistency for a data-selected threshold. It also separates schedule availability from the validity of a returned schedule. The independent pilot grid, calibration-size and horizon sensitivity experiments, and effective-sample-size diagnostics are useful. The study therefore goes beyond merely reporting average coverage in one feedback setting.

Critical weaknesses

W1. The experiments largely test calibration with a known simulated treatment mechanism, leaving the central historical-data difficulty insufficiently evaluated. Appendix D.2 states that after fitting the clinical components, all pilot, calibration, and evaluation trajectories are generated from a fixed controlled environment. In particular, the fitted historical policy generates the simulated actions and is also available when constructing importance weights. This does not test estimation of an unknown logging policy from a finite observational sample, although the theory requires accurate estimated prefix weights. Figure A8 and the horizon experiments do address weight concentration, but not systematic propensity-estimation error, which can accumulate across stages even when individual propensity errors are modest.

The feedback family is also specified directly as a known function of the scalar threshold: in the clinical environments, it interpolates between the historical policy and a fixed alternative policy (Appendix D.2, pp. 23–24). It does not learn clinicians' responses to the content of a patient's prediction set. This is a legitimate controlled experiment, but it makes offline identification substantially easier than the motivating clinical use. The invariant transition kernel additionally requires an adequate observed state/history representation and excludes unobserved treatment–outcome confounding; the importance-weight identity does not establish these conditions from clinical records.

The current evidence consequently supports self-consistent calibration under a specified simulator and known response family. To support the broader historical-data claim, I would expect at least an experiment that hides and re-estimates the logging policy on separate simulated data, together with controlled misspecification of the threshold-to-treatment response. This can be done within the existing benchmarks and need not require prospective patient data.

W2. The baseline configurations and uncertainty summaries leave the incremental advantage unresolved. The MFCS adaptation uses a history depth of three, cumulative-weight clipping at 40, and a single scale multiplying a fixed stage profile, whereas SCCP uses full prefixes and stage-specific thresholds without clipping (Appendix D.2, p. 22). These choices differ precisely along the dimensions the ablations identify as consequential. The resulting comparison does not isolate how much of SCCP's advantage comes from self-consistency, fuller history, or a more flexible threshold schedule. This is a concern about the implemented comparator, not a claim that the original MFCS theory directly solves the present setting.

The closest policy-coupled work is discussed, including Zheng and Jin (2026), but the experiments do not establish a direct one-stage comparison or identify which of its construction's requirements prevent a matched comparison. Historical-policy transport is also an established starting point, as illustrated by Zhang et al. (2023). The new contribution is therefore most persuasive as the longitudinal self-consistent construction; the off-diagonal representation alone does not establish a new estimation principle.

Finally, classifying a method as “WSC-qualified” using mean empirical worst-stage coverage at or just above 90% is stronger than the uncertainty warrants. Table 1 reports SCCP values such as 90.0 ± 0.2% and 90.2 ± 0.7%, not a guarantee that each calibrated schedule reaches 90%. The Appendix A.2 bound is useful, but depends on a potentially large cumulative-weight bound and all candidate prefixes, and is not evaluated numerically. The statement that SCCP yields the smallest width among methods reaching nominal coverage should be treated as a comparison of reported point estimates unless supported by coverage uncertainty or a risk–width frontier at matched reliability. Stagewise marginal coverage also does not establish better clinical treatment decisions, which the title and conclusion should continue to distinguish.

Questions for authors

Q1 — W1, essential robustness evidence. In the current semi-synthetic protocol, is the policy used in the weights exactly the policy that generates calibration actions? If so, can you hide that mechanism and re-estimate it from an independent finite logging sample, reporting propensity error, stagewise effective sample size, and coverage as the treatment horizon grows?

Q2 — W1, scope and mechanism clarification. What information about the threshold-induced clinical response must be known before calibration? How sensitive is SCCP to modest perturbations of that response during deployment? Please also specify whether the state used in the theoretical factorization represents the complete observed history or an assumed sufficient summary. These answers determine the interpretation of offline identification from clinical trajectories.

Q3 — W2, fair comparison. Can you separate the effects of MFCS history depth, clipping, and threshold-profile restrictions by varying them under a shared prediction model, grid, and calibration budget? Where a matched adaptation is not theoretically justified, please explain the limitation rather than interpreting the configured baseline's result as a general inferiority result.

Q4 — W2, novelty and qualification of claims. What does SCCP add in the T=1 special case relative to the policy-coupled construction already cited? For the sequential experiments, can you report the frequency and uncertainty of genuine population-coverage shortfalls across fitted schedules, and compare widths at matched coverage rather than qualifying methods solely by a mean WSC point estimate?

1 comment

Thank you for the additional experiments and responses. However, the issues of dependence on a specified treatment-response model and undercoverage under modest misspecification remain, so I have decided to keep my original score.

Reviewer_4ZEbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles uncertainty quantification for time varying/sequential treatments, in particular in a performative feedback setting where one must account for the distribution shift which is induced by deploying the self-derived system. For this, the authors develop Self-consistent conformal prediction, a conformal prediction method which uses reweighting historical trajectories and self calibration for this purpose. The authors provide asymptotic coverage guarantees theoretically and show empirically on extensive (semi-)synthetic datasets reliable performance of their method and improvements over baselines tailored to slightly different settings.

Critical strengths
  • The problem of uncertainty quantification from historical data in a sequential treatment setting is highly important in medical applications and the motivation is clear and well described.
  • For a mostly OPE/causal inference paper (which is hard to evaluate on real data), the paper provides extensive and well motivated semi synthetic experiments on real world inspired settings and their method shows promising results.
  • The paper is mostly nicely structured and good to read and provides mostly the proper balance between detailed description and motivating explanations of the components (for concerns about contribution presentation see weaknesses).
  • The paper backs up most of the claims and provides proper details in the appendix for theory and benchmark design.
Critical weaknesses
  1. In some parts it is not clearly elaborated what the contribution of the paper is and where it actually mostly adapts existing ideas. E.g., Eq. 8, uses the idea of IPTW from sequential causal inference / offline RL. I think this should be properly referenced, otherwise it might look like the derivation would be novel (the proofs also do not reference). The same holds for further derivations. The same also holds for (a bit more obvious) claims in Section 2 where no other works are referenced to back the statements up. A clear description throughout the paper of (i) on what works they build upon and (ii) what is novel, is necessary to properly judge the contribution of the paper.
  2. The assumptions might need to be explained and justified better, i.e. how realistic they are. E.g., after Eq. 8, the authors state that it requires and overlap conditions which I think in this setting can be justified but this should be done in the paper. However, the weights seem to require access to the true historical treatment policy (conditional distribution) which I would think in healthcare settings is barely observed and needs to be estimated from the data, which is also proposed in Eq. 10. This estimation error in CP usually results in loss finite sample guarantes which is the main advantage compared to other uncertainty methods (i.e., for the proposed framework the guarantees are asymptotic). This should be discussed more, and maybe it could also be theoretically investigated what happens under misspecification of the historical policy estimate, which estimation rates are needed etc.

Minor:

  • Figure 2, especially the middle but also right part are not really clear to me. The plots without axes are more confusing. The figure should be improved and the caption be more clear and self-contained.
  • Overall in the (extended) related work there should be a more clear discussion of CP in causal inference, since in the static setting there are multiple works around this area and it should be double-checked whether extensions for the time varying treatment settings already exist and how they differ.
Questions for authors
  • Can you elaborate on how exactly applying CP at each timestep affects the distribution shift of the target policy to evaluate on, i.e. how would clinicians select the next treatments based on the prediction sets, or in theoretical terms where does the mapping form the threshold to the treatment policy come from? I would assume that in practice maybe you would optimize for the policy value or in this case some LCB using the sets?
  • Can you elaborate on the exact contributions over existing works for each important part of the theory, as well as on the plausibility of the assumptions and the implications of potential violations (see Weaknesses for all points in more detail above).
Reviewer_AyGmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies conformal prediction in sequential treatment settings where prediction sets influence treatment decisions and thereby change subsequent outcomes. It proposes Self-Consistent Conformal Prediction (SCCP), which identifies deployment coverage through cumulative treatment-policy importance weights and recursively selects stagewise thresholds from historical trajectories. Under identification, overlap, bounded-weight, and consistency assumptions, the authors establish asymptotic stagewise marginal coverage. Experiments include one synthetic benchmark and four semi-synthetic clinical benchmarks.

Critical strengths
  • The paper addresses an important failure mode of conformal prediction in sequential care: the prediction set can influence treatment and therefore the distribution under which coverage is evaluated.
  • The prefix importance-weighting formulation and recursive threshold selection provide a coherent way to estimate self-consistent stagewise coverage from historical trajectories.
  • The empirical study is relatively broad, covering a synthetic benchmark, four clinical data sources, multiple feedback strengths, calibration sizes, and component ablations.
Critical weaknesses
  1. The theoretical guarantee is primarily asymptotic and conditional.
    The main result establishes coverage up to an estimation error that vanishes asymptotically, conditional on a complete threshold schedule being returned. The explicit finite-sample bound is presented under known historical treatment probabilities and uniformly bounded importance weights, while the practical setting estimates the historical policy and may suffer from weight concentration. The paper does not provide a finite-sample guarantee that accounts for policy-estimation error, nor does it characterize the coverage or safety implications when no complete schedule is available.

  2. The causal and deployment assumptions limit the practical interpretation of the results.
    Identification requires overlap and invariance of the conditional transition and outcome law under the changed treatment policy. The clinical experiments use semi-synthetic environments and explicitly constructed alternative policies, so they do not test unobserved confounding, clinician response heterogeneity, delayed treatment effects, or violations of the assumed transition model. The reported coverage therefore demonstrates performance under the constructed feedback mechanism rather than under real deployment feedback.

  3. The empirical comparison and stress testing do not fully establish the method’s practical advantage.
    The paper compares several relevant baselines and includes useful ablations, but the positioning relative to recent policy-coupled conformal prediction and conformal policy-control methods could be made more explicit. The experiments also provide limited sensitivity analysis for overlap strength, importance-weight clipping or concentration, historical-policy estimation error, candidate-grid resolution, and policy misspecification. These factors directly determine whether SCCP can produce informative and reliable thresholds in practice.

Questions for authors

Please see Weaknesses.

Reviewer_CVUjOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies conformal prediction in sequential clinical settings where the prediction set itself influences treatment decisions, which changes the outcome distribution. Standard CP breaks because the deployment distribution P_q depends on the threshold q being calibrated. They formulate this as a self-consistent calibration problem: identify the off-diagonal coverage relation C(u,v) from historical trajectories via importance weighting, then find u=v where the diagonal hits nominal coverage. Stagewise recursion handles longitudinal coupling. Experiments on one synthetic and four semi-synthetic clinical benchmarks show SCCP maintains coverage at the nominal level with tighter prediction sets than conservative baselines.

Critical strengths

How sensitive are results to propensity estimation error? Can you add synthetic perturbation to the estimated mu and show coverage degradation? What happens when the coverage map has multiple self-consistent solutions? Is there a uniqueness result?

Critical weaknesses

The problem is important and understudied. Prediction systems that change the distribution they're calibrated on is a real issue in clinical decision support, and the framing q -> pi_q -> P_q -> coverage is precise. The methodological split — separate the treatment threshold from the coverage cutoff on the off-diagonal, then return to the diagonal — is elegant and correct. Experiments are thorough: five benchmarks, four baselines (Standard CP, ACI, MFCS, PRC), ablations on design choices, feedback strength sensitivity, and calibration-size scaling.

Questions for authors

Importance weighting is the crux of the method, but its robustness to propensity estimation error is not shown. In high-dimensional state spaces, estimating mu_t(a|s) well enough to avoid weight explosion is hard. The paper assumes consistent estimation and bounded likelihood ratios without empirical sensitivity analysis. All clinical benchmarks are semi-synthetic — the feedback model, how a prediction set actually changes clinician behavior, is baked into the simulation. Without a prospective study, the clinical utility claim remains aspirational. The overlap assumption is strong: historical data must support all treatments the induced policy might choose, which is not realistic in real EHR data.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.