- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies prediction under observation-policy shift, where historical records are nonignorably missing and the known target observation policy differs from the historical one. The authors estimate target-policy risk via budgeted, passive randomized audits. They characterize the joint observation requirements for exact risk identification, show that these depend on the estimand, and propose a convex program that jointly chooses audit allocations and a loss reconstruction to trade off bias and variance. Small-scale synthetic and semi-synthetic experiments show that prediction gains come mainly from reconstruction, with limited gains from learning the allocation or fitting the reference from a paid pilot.
Critical strengths
This paper points out that the observation strategy employed during deployment introduces higher-order errors into the loss that the predictor itself does not observe. Therefore, which values to measure jointly depends on the estimation objective rather than on the predictor. The conditions for identifiability provide a precise necessary-and-sufficient characterization, accompanied by constructive non-identifiable counterexamples.
Critical weaknesses
- The author did not organize the paper with sound logic and clear, accessible language, and failed to include the necessary references to help readers understand the problem this paper aims to address.
- Positive results demonstrating the practical value of the method are scarce, and the few apparent gains almost all depend on artificially imposed known structures. The joint measurement design outperforms only under a single configuration with a pre-specified order, and neither the paper's own upper bounds nor its validation criteria are able to select it. The only stable gains come from the reconstruction procedure and are unrelated to the measurement design itself. Experiments with pronounced gains are built on favorable settings. For example, the parity-family experiment discloses the relevant interaction set in advance. Moreover, when the structural assumptions are correct, the joint model without supplementary measurements actually outperforms all auditing methods. Existing experiments do not show that the method can deliver actionable gains in practical scenarios where the structure is unknown.
Questions for authors
- Could the author explain why so many of the experimental results are not positive? Why are these numerically negative results reasonable in the context of this paper?
- Could the author organize the Methods section in a more logical and coherent manner? Instead of placing many key theories in the appendix.
- The defensive tone of this paper is all too evident. This style of writing severely undermines the paper's originality and makes it difficult for readers to engage with the text.
Thank you for recognizing the joint-measurement insight. We agree that the organization obscures the contribution and that the experiments do not establish generally superior allocation or unknown-structure acquisition.
Levis et al. (Statistics in Medicine, 2024) address informative missingness through double sampling. Juditsky & Nemirovski (Annals of Statistics, 2009) develop convex estimation for fixed observation maps. Here, the partial observation map is itself a budgeted decision. Inverse weighting and convex perspectives are established ingredients.
Recent comparisons are von Kleist et al. (JMLR, 2025), Evaluation of Active Feature Acquisition Methods for Time-varying Feature Settings, and Kobayashi et al. (ICML, 2026), Learning-To-Measure. These address retrospective evaluation and test-instance acquisition. Our target is a specified policy-induced functional under unrestricted initial coarsening, not empirical superiority to these systems.
The full results extend beyond one favorable configuration. Across the submitted 27 logistic configurations (Appendix S.1), the same prespecified degree-two design beats bias-aware full auditing in population MSE in 17/27, and fixed uniform two-subset acquisition with optimized reconstruction in 27/27. These are configurations of one query family, not independent datasets or successful data-driven selections. The main setting improves by 19.4% and 12.5%, respectively. The alternatives also optimize reconstruction and accept bias, isolating acquisition's contribution.
The parity example assumes the relevant set is known. It establishes a matching information/cost separation, not support discovery. We do not demonstrate gains when the target mechanism and relevant structure are unknown.
First, a uniform error bound need not rank designs by their MSE under one population. A favorable fixed-degree result therefore cannot establish successful degree selection. We additionally tested every validation size requested by M3Fu, charging all recovered values. Neither existing selector beat full-budget bound selection. This limits the operational selection claim.
Second, extracting more information from the same audits can improve prediction without making learned allocation worthwhile. On dense digits, scope weighting gives 0.23550 and learned-reference reconstruction gives 0.22479 using identical audits. Once reconstruction is fixed, pilot-reference versus uniform-reference allocation has small, inconsistent differences. Our new covariance-aware Brier allocation also provides no reliable improvement on the same splits. We will not attribute reconstruction's gain to allocation.
Third, valid structural assumptions can remove the need for follow-up. The correctly specified joint-model control is therefore expected to do well. It does not contradict a necessity theorem over unrestricted completion laws. It shows when a different assumption set supports a cheaper solution.
Li, Zrnic & Candès, Robust Sampling for Active Statistical Inference (NeurIPS, 2025), address deterioration from inaccurate uncertainty-guided sampling. This is relevant context, not a guarantee of our procedure.
The acceptance case is the task-specific information characterization and joint design formulation, supported by comparisons that isolate gains and failure regimes. We ask that these contributions be assessed on their stated finite-state scope, not as a claim of universal efficiency or unknown-mechanism learning.