- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper formalizes individual-level surrogacy for N-of-1 settings via the regret of a surrogate-map-induced policy and proposes an imputation-based evaluation that is identifiable under strong assumptions on latent invertibility and full mediation. While the problem is important and the formalization is clean, the core results rest on restrictive, untestable assumptions that are unlikely to hold in the motivating applications; the empirical support (synthetic DGPs matched to the assumptions + a heavily pre-processed observational wearable cohort) is insufficient to demonstrate reliable practical utility; and the overall advance over existing surrogate-index and causal-surrogacy literature is incremental. These foundational limitations make the work unsuitable for acceptance at a top venue.
Critical strengths
-
The formalization of individual-level regret (Definitions 2–3) cleanly distinguishes ranking of counterfactual outcomes for a single subject from population-average treatment-effect estimation, filling a genuine conceptual gap left by classical surrogacy criteria.
-
The synthetic experiments carefully isolate the effect of sample size, outcome-model misspecification, and assumption violations, and the Apple Heart & Movement Study application demonstrates that the proposed pipeline can be instantiated on complex longitudinal wearable data, producing concrete (if still observational) rankings of biomarkers.
Critical weaknesses
-
Critical dependence on strong, untestable assumptions.Identifiability of individual-level regret (Theorem 1) requires Assumption 2 (latent type is constant and fully recoverable by a measurable left-inverse of ) together with support inclusion. This is far stronger than standard unconfoundedness or Prentice-style conditions and is essentially untestable. The discussion acknowledges the issue but provides only limited ablations; in the Apple Heart & Movement Study application the assumption is simply postulated (“residual confounding captured in latent type ”). When the assumption fails, the pseudo-regret can be arbitrarily misleading.
-
Evaluation procedure is circular and inherits observational confounding.Pseudo-outcomes are obtained by regressing the long-term outcome on in an observational population study and then plugging the N-of-1 subject’s observed into that regression. Any residual confounding, model misspecification, or support mismatch between the observational cohort and the N-of-1 subjects directly contaminates the “ground-truth” ranking used for evaluation. The real-data pipeline (change-window construction, Cox pseudo-labels, cross-fitting with only 10 % held-out subjects) amplifies rather than mitigates these problems.
-
Empirical validation is weak and does not support the claimed practical utility. Synthetic experiments use a DGP deliberately constructed to satisfy the paper’s assumptions; finite-sample consistency is therefore unsurprising. The sole real-world illustration is a secondary analysis of the observational Apple Heart & Movement Study, mapped onto the N-of-1 framework via post-hoc change windows. Treatments are self-selected, residual time-varying confounding is uncontrolled, and no prospective or external validation is performed. Findings such as “VO₂max ranks well” or “PPG embeddings do not help” remain descriptive and clinically uninterpretable under the paper’s own assumptions.
-
Limited novelty relative to the surrogate-index and causal-surrogacy literature. The surrogate index of Athey et al. (and related causal maps) already constructs scalar summaries of intermediate measurements for treatment-effect estimation. The paper’s main conceptual move—reinterpreting the same conditional-expectation objects as ranking devices for individual-level regret—is modest. Learning results (Appendix A.7) recover essentially the same causal surrogate map under additive confounding, and generalization guarantees require an additional full-mediation assumption that is only marginally weaker than classical no-direct-effect conditions.
-
Scope of the guarantees is too narrow for the motivating use-cases. Even under the stated assumptions, evaluation is only valid for treatments already observed in the population study (or for new treatments that fully mediate through the same surrogate space and produce sufficiently diverse surrogate pairs). In the very settings the paper advertises—novel exercise regimens, new advertising campaigns, new drug doses—population support is absent by definition, rendering the evaluation procedure inapplicable precisely when it is most needed.
Questions for authors
Please refer to the weaknesses
We thank the reviewer for the detailed review and the appreciation of our formalization of individual-level surrogacy. Several listed weaknesses rest on what we believe to be a misunderstanding of our results, which we resolve below and in the general response (G1–G4). We have revised the presentation accordingly.
W1. Assumption 2. We agree that it is an untestable identification assumption, and our discussion says so. We do not agree that it is "far stronger" than unconfoundedness or Prentice-type conditions. Prentice-type criteria and the surrogate index rely on the absence of unobserved surrogate-outcome confounding, in which case our assumption (restated in G1, now allowing noisy surrogates) holds trivially if there is no latent U. We cover confounded settings, as long as the confounder is revealed by the treatment-response pair. Although standard conditions might identify population-average effects here, ranking of a single subject's treatments remains highly nontrivial (App. A.3). We explain in G2 why such an assumption is unavoidable. "Arbitrarily misleading": map comparisons remain valid under any within-subject constant bias, and otherwise admit a simple sensitivity analysis (G3).
W2. "Circular" evaluation. A surrogate map is a function of (S,X) only, whereas the pseudo-label uses the regression of Y on (A,X,S). Both are fitted on separate samples. If the evaluation were circular, we could predict ahead which map achieves the lowest pseudo-regret, and the surrogate index E[Y∣S,X], similar to the label up to inclusion of A, should win. This is not the case: it often achieves a higher pseudo-regret than other methods, such as control functions, so our evaluation carries real signal. In contrast to the reviewer's claim, our preprocessing (change windows, change-from-baseline, Cox model) makes the assumptions more plausible (G4). "Only 10% held-out" is per rotation: over the 20 rotations every participant serves as an evaluation subject twice. Map fitting, pseudo-outcome fitting and evaluation are fully disjoint.
W3. Empirical validation. The consistency of our estimator is not meant to be surprising: Fig. 2 illustrates that the pseudo-regret inherits the finite-sample rate of the pseudo-label estimator and is biased if that model is misspecified. In Fig. 8, we actually study violations of Assumption 2 and misspecification. In AHMS, one strength of our framework is that treatments are allowed to be self-selected within a subject (App. A.6), and we do control for time-varying confounding (G4). Our findings are meant to be initial and qualitative. In G4, we explain why no ground truth exists yet. If the reviewer knows of suitable N-of-1 trials with years of follow-up per treatment, we would be happy to validate our method there.
W4. Novelty. We extensively position our work against the surrogate index (SI) (l. 65-75, App. A.1, A.3, A.7). The SI, proposed for population-level surrogacy under Prentice-style assumptions, is one of the candidate maps we evaluate. By A.3, population-level surrogacy neither implies nor is implied by individual-level surrogacy, and in our setting the SI is provably suboptimal, as also observed in our experiments. We study a different notion of surrogacy, propose an evaluation framework instead of a single map, relax unconfoundedness, and establish when maps generalize to unseen treatments, whereas SI literature mostly considers a single binary treatment. The causal surrogate map's estimand and IV estimators are well-known and cited (App. A.1). Our claims are its optimality for individual surrogacy (Theorem 4) and its suboptimality under general confounding (Proposition 3). No direct effect is necessary for generalization of any map f(S,X) (App. A.5), we do not claim to generalize it. However, in our setting the Prentice criterion Y⊥A∣S fails even without direct effect, since conditioning on S opens the path A→S←U→Y. SI-style literature thus relies on a stricter statistical assumption, and Assumption 3 needs mediation only through the whole surrogate vector, not a scalar endpoint.
W5. New treatments. "Evaluation is only valid for treatments already observed in the population study" is incorrect. Generalization to new treatments is one of our main contributions: Theorem 2 gives explicit conditions under which our evaluation transfers to unseen treatments. Even for fully mediated treatments this is highly nontrivial. To the best of our knowledge, existing (population-level) surrogacy literature gives no such guarantees, assuming the same treatment for learning and deployment. Our result does not restrict the surrogate mechanism g0(a,⋅) of the new treatment and only requires sufficiently diverse surrogate response pairs at evaluation. The revision states Theorem 2 in the main text.
Ethics flag. The reviewer names no concern. If the reviewer states one, we will address it.