acceptodds
Withdrawn from ICLR 2027

Joint Measurement Design for Prediction under Observation-Policy Shift

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A change in measurement policy can alter predictive risk even when the underlying data distribution remains fixed. We study budgeted passive follow-up for prediction under known observation policies, starting from nonignorably missing records. In a finite-state model, exact risk identification requires joint observation of every nonzero interaction in the target-policy loss. Even a simple logistic observation rule can therefore require full-record audits. We formulate a convex program that jointly chooses partial measurements and a loss reconstruction. It controls bias pointwise and mean-squared error under a declared reference envelope. Observable risk contrasts can reduce the information needed to compare predictors. For approximation selection, we distinguish the smallest error bound from the smallest actual error. We construct an unbiased candidate-MSE criterion from a separately charged validation sample and prove a finite-menu selection guarantee. Bias-matched logistic experiments separate the effects of approximation, reconstruction, and allocation. Paid-pilot experiments on repeated semi-synthetic image splits show larger prediction gains from reconstruction than from learning the allocation, whose gains are smaller and inconsistent. Validation and reference fitting can also cost more than they save. The results characterize the information identified by partial measurements, the error left by approximation, and the design decisions supported by measurements an investigator acquires.

Primary area:
learning theory
Keywords:
observation-policy shift, missing not at random, statistical identifiability, optimal experimental design, active feature acquisition, risk estimation, loss reconstruction, bias-variance tradeoff, convex optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_dnYDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies prediction under observation-policy shift, where historical records are nonignorably missing and the known target observation policy differs from the historical one. The authors estimate target-policy risk via budgeted, passive randomized audits. They characterize the joint observation requirements for exact risk identification, show that these depend on the estimand, and propose a convex program that jointly chooses audit allocations and a loss reconstruction to trade off bias and variance. Small-scale synthetic and semi-synthetic experiments show that prediction gains come mainly from reconstruction, with limited gains from learning the allocation or fitting the reference from a paid pilot.

Critical strengths

This paper points out that the observation strategy employed during deployment introduces higher-order errors into the loss that the predictor itself does not observe. Therefore, which values to measure jointly depends on the estimation objective rather than on the predictor. The conditions for identifiability provide a precise necessary-and-sufficient characterization, accompanied by constructive non-identifiable counterexamples.

Critical weaknesses
  1. The author did not organize the paper with sound logic and clear, accessible language, and failed to include the necessary references to help readers understand the problem this paper aims to address.
  2. Positive results demonstrating the practical value of the method are scarce, and the few apparent gains almost all depend on artificially imposed known structures. The joint measurement design outperforms only under a single configuration with a pre-specified order, and neither the paper's own upper bounds nor its validation criteria are able to select it. The only stable gains come from the reconstruction procedure and are unrelated to the measurement design itself. Experiments with pronounced gains are built on favorable settings. For example, the parity-family experiment discloses the relevant interaction set in advance. Moreover, when the structural assumptions are correct, the joint model without supplementary measurements actually outperforms all auditing methods. Existing experiments do not show that the method can deliver actionable gains in practical scenarios where the structure is unknown.
Questions for authors
  1. Could the author explain why so many of the experimental results are not positive? Why are these numerically negative results reasonable in the context of this paper?
  2. Could the author organize the Methods section in a more logical and coherent manner? Instead of placing many key theories in the appendix.
  3. The defensive tone of this paper is all too evident. This style of writing severely undermines the paper's originality and makes it difficult for readers to engage with the text.
1 comment

Thank you for recognizing the joint-measurement insight. We agree that the organization obscures the contribution and that the experiments do not establish generally superior allocation or unknown-structure acquisition.

  1. Contribution and relation to existing work. We ask which partial follow-up measurements identify a specified deployment loss and how to trade unresolved information against measurement cost. Exact reconstruction requires joint coverage of policy-induced interactions. The central convex program jointly chooses acquisition and reconstruction, with pointwise bias control and an MSE bound under an explicit envelope, or a more conservative distribution-free alternative.

Levis et al. (Statistics in Medicine, 2024) address informative missingness through double sampling. Juditsky & Nemirovski (Annals of Statistics, 2009) develop convex estimation for fixed observation maps. Here, the partial observation map is itself a budgeted decision. Inverse weighting and convex perspectives are established ingredients.

Recent comparisons are von Kleist et al. (JMLR, 2025), Evaluation of Active Feature Acquisition Methods for Time-varying Feature Settings, and Kobayashi et al. (ICML, 2026), Learning-To-Measure. These address retrospective evaluation and test-instance acquisition. Our target is a specified policy-induced functional under unrestricted initial coarsening, not empirical superiority to these systems.

  1. Favorable structure and the number of positive cases. The target kernel and loss are specified. Their interactions are computed from these functions, while the complete-data law is unknown. This is known-policy design, not discovery of an unknown deployment mechanism. That distinction belongs before the method.

The full results extend beyond one favorable configuration. Across the submitted 27 logistic configurations (Appendix S.1), the same prespecified degree-two design beats bias-aware full auditing in population MSE in 17/27, and fixed uniform two-subset acquisition with optimized reconstruction in 27/27. These are configurations of one query family, not independent datasets or successful data-driven selections. The main setting improves by 19.4% and 12.5%, respectively. The alternatives also optimize reconstruction and accept bias, isolating acquisition's contribution.

The parity example assumes the relevant set is known. It establishes a matching information/cost separation, not support discovery. We do not demonstrate gains when the target mechanism and relevant structure are unknown.

  1. Why report the negative findings? They distinguish three decisions that would otherwise be conflated.

First, a uniform error bound need not rank designs by their MSE under one population. A favorable fixed-degree result therefore cannot establish successful degree selection. We additionally tested every validation size requested by M3Fu, charging all recovered values. Neither existing selector beat full-budget bound selection. This limits the operational selection claim.

Second, extracting more information from the same audits can improve prediction without making learned allocation worthwhile. On dense digits, scope weighting gives 0.23550 and learned-reference reconstruction gives 0.22479 using identical audits. Once reconstruction is fixed, pilot-reference versus uniform-reference allocation has small, inconsistent differences. Our new covariance-aware Brier allocation also provides no reliable improvement on the same splits. We will not attribute reconstruction's gain to allocation.

Third, valid structural assumptions can remove the need for follow-up. The correctly specified joint-model control is therefore expected to do well. It does not contradict a necessity theorem over unrestricted completion laws. It shows when a different assumption set supports a cheaper solution.

Li, Zrnic & Candès, Robust Sampling for Active Statistical Inference (NeurIPS, 2025), address deterioration from inaccurate uncertainty-guided sampling. This is relevant context, not a guarantee of our procedure.

  1. Organization, references, and tone. We propose a single methodological sequence: define the record and target loss, characterize exact information requirements, state the exact reference-optimal design theorem, introduce controlled approximation, then connect risk contrasts and table reconstruction to prediction. The key theorem statements and the channel-to-record mapping will be in the main text. We will consolidate assumptions rather than repeat defensive qualifications throughout.

The acceptance case is the task-specific information characterization and joint design formulation, supported by comparisons that isolate gains and failure regimes. We ask that these contributions be assessed on their stated finite-state scope, not as a claim of universal efficiency or unknown-mechanism learning.

Reviewer_3xpqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies prediction under a known change in observation policy while the underlying feature–outcome distribution remains fixed. Historical records may have nonignorable missingness, and an investigator can purchase additional feature measurements through randomized passive audits subject to a budget. In a finite-state model with subsequently observed outcomes, deployment risks become expectations of known complete-data loss functions. The investigator jointly chooses the audit distribution and an observation-based estimator.

The paper characterizes exact identification and constructs unbiased estimators with reference-optimal reconstructions. Its central convex program jointly optimizes acquisition and approximate reconstruction, controlling bias pointwise and MSE under a specified reference envelope. Further results address observable risk contrasts, logistic-loss approximation, reference misspecification, and candidate selection using paid validation. Synthetic and semi-synthetic experiments are also presented as the complementary of the theoretical contributions.

Critical strengths

This paper studies a well-motivated measurement-design problem.

The authors provide a relatively comprehensive package of results.

The theoretical analysis is sound.

Critical weaknesses

My main concern is about the writing quality and layout of the manuscript.

While the authors provide some interpretations of each individual results, these local explanation does not produce a clear overall narrative. Here I give a few examples:

(i) Proposition 3 is conceptually important, but it is not a necessary step toward the central acquisition-design program. The paper would be easier to follow if it explicitly labeled it as a change of estimand.

(ii) After characterization identification in Section 3, the most natural next-step question should be "which identifying design gives the smallest estimation variance under the budget?". However, this is only answered in the Appendix (Theorem 8), while the paper use its machinery in the main text (Section 4). It is hard to understand the approximate-design program before receiving a clear account of the simpler exact-design problem that motivates it.

(iii) Both the title and experiments emphasize prediction, while the central theorem concerns fixed-query MSE. The mathematical bridge to learning is placed largely in the appendix.

In addition, the notations and terminology used in this paper also contain missing/delayed definitions and inconsistent conventions. Just to give a few examples: kj(z∣h,x)k_j(z| h, x) is used without definition (page 2), notation hh is used under different meaning (hh vs. hjh_j vs. hah_a, vs hg,ah_{g,a}), the connection between gj(h,x,y)g_j(h, x, y) and gj,o,y(u)g_{j, o,y}(u) is not defined, inconsistent notations between q(a∣o)q(a | o) (Section 3) to qaq_a (Section 4).

Overall, while I see potential in this study, I feel the current manuscript does not meet the quality for publication.

Questions for authors

NA

1 comment

Thank you for recognizing the problem's motivation and theoretical soundness. We agree that the reader should not have to reconstruct the logical order. Your suggested sequence is better.

  1. State the central question before the supporting results. We ask which budgeted partial measurements permit accurate estimation of a specified deployment loss under nonignorable historical missingness. The central result jointly chooses the measurement probabilities and an approximate reconstruction of that loss. Exact identification determines when zero reconstruction error is possible. Predictor contrasts change the target being reconstructed. They are useful consequences, not prerequisites for the joint program.

The proposed main-text sequence is therefore: operational example and observation model → exact information requirements → optimal exact reconstruction and acquisition → controlled approximation → prediction. The guarantees belong in this order in the main text, with proofs in the appendix.

  1. Bring exact optimal design before approximation (II). We agree that Theorem 8 is needed in the main argument. In one completion cell with a uniform reference and query coefficients αS\alpha_S, its exact-design problem is min⁡q∈Δ(A),,∑aqaca≤b ∑S≠∅αS2pS(q).\min_{q\in\Delta(\mathcal A),,\sum_a q_ac_a\le b}\ \sum_{S\ne\varnothing}\frac{\alpha_S^2}{p_S(q)}. This minimizes reference variance among single-record, pointwise-unbiased reconstructions, not variance under every law. A nonzero coefficient at zero coverage has infinite objective. This makes the transition to Eq. (10) immediate: allow reconstruction coefficients to change, bound the resulting pointwise residual, and optimize bias and variance together. The general channel formulation will follow this special case.

  2. Mark Proposition 3 as a different estimand (I). It is not a premise of Theorem 4. Replace absolute losses gfg_f by gf−gf0g_f-g_{f_0}. A shared unidentified component can cancel, reducing the measurements needed for the decision. We will put this in a separate task-choice paragraph after the main design argument. The qualification about policy-specific offsets under worst-policy selection will remain.

  3. Make the prediction bridge explicit (III). There are two distinct routes. For a finite predictor class, reconstruct its loss contrasts and minimize their estimates. If every contrast error is at most ee, the selected predictor has excess risk at most 2e2e. For the experiments, reconstruct the target observation/outcome table and form η(z)=tz1/(tz0+tz1)\eta(z)=t_{z1}/(t_{z0}+t_{z1}). The identity R(η^)−R(η)=∑zP(Z=z)(η^z−ηz)2R(\widehat\eta)-R(\eta)=\sum_z P(Z=z)(\widehat\eta_z-\eta_z)^2 connects table estimation to prediction. Positive cell masses and consistent table estimates give consistent predictions. This does not make Theorem 4 a finite-sample Brier guarantee for the nonlinear ratio, projection, and smoothing pipeline.

We have also derived the requested covariance-aware local Brier criterion in our response to M3Fu. It weights each pair of table errors through ∇ηz=(−ηz,1−ηz)/(tz0+tz1)\nabla\eta_z=(-\eta_z,1-\eta_z)/(t_{z0}+t_{z1}). Its pilot-fitted implementation did not improve mean dense-digit Brier risk on the same 40 splits. We will retain that result and identify the prediction study as a test of allocation and reconstruction, rather than evidence that the scalar MSE bound directly controls the fitted predictor.

  1. Repair each notation gap. The target observation is Ztar=(M′,XM′)Z_{\rm tar}=(M',X_{M'}), with kj(z∣h,x)=∑m′πj(m′∣h,x)1(m′,xm′)=z.k_j(z\mid h,x)=\sum_{m'}\pi_j(m'\mid h,x)\mathbf1{(m',x_{m'})=z}. For o=(h,m,xm)o=(h,m,x_m), gj,o,y(u)g_{j,o,y}(u) means gj(h,x,y)g_j(h,x,y) with the recorded coordinates fixed at xmx_m and the missing coordinates set to uu. The audit output ZaudZ_{\rm aud} contains the revealed values and eventual outcome, with the native record implicit when conditioning on oo. Its channel CaC_a is different from the deployment kernel kjk_j. We will write qo(a)q_o(a) throughout and reserve hh for context, replacing conditional score means and reconstruction functions by separate symbols. We will also define aˉj=v−1∑iaj(Ui)\bar a_j=v^{-1}\sum_i a_j(U_i) beside Eq. (11).

  2. Position the result where it is introduced. Levis et al. (Statistics in Medicine, 2024) provide double sampling under informative missingness. Juditsky & Nemirovski (Annals of Statistics, 2009) study convex estimation through a fixed affine observation map. Our additional choice is a cost-constrained mixture of partial observation channels jointly with a controlled reconstruction residual. Recent active-acquisition evaluation (von Kleist et al., JMLR, 2025) supplies complementary retrospective identification frameworks. These comparisons should accompany the main formulation, rather than require consulting the appendix.

We will use the available main-text space for these definitions, the exact-design theorem, and the learning bridge

Reviewer_M3FuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies prediction when the rule for observing features changes but the underlying data distribution stays the same. The paper first identifies which features must be measured together to estimate prediction risk. It then proposes a convex program that jointly chooses additional measurements and an estimator, allowing some bias to reduce cost and variance. The authors also study how to choose an approximation using validation data whose measurement cost is included. The experiments show benefits from partial measurements and better reconstruction. However, the gains from learning the measurement allocation are smaller and less consistent.

Critical strengths

The identification result clearly explains why measuring individual features may not provide enough information. Moreover, the paper shows that comparing predictors can require fewer measurements than estimating each predictor’s risk

Critical weaknesses

(i) The "Introduction" is rather poorly organized, and I find it hard to follow. It does not clearly separate the main contribution from the supporting results. It also presents the three-sensor formula and several technical terms before explaining the basic ideas. As a result, the reader has to work out how these parts fit together.

(ii) The organization and changes in notation also make the model difficult to understand. Section 2 introduces several observation mechanisms quickly, and later sections move to channel notation without explaining the connection clearly. In particular, it is not clear to me how the channel output z/Zz/Z relates to the earlier observation model. The empirical average aˉj\bar a_j in Equation (11) is not explicitly defined there; the reader has to find its definition in Appendix F.2. These explanations would be more useful in the main text, especially since it ends partway through page 8.

(iii) Theorem 4 provides a bias bound under every completion law. However, the MSE bound requires the assumption p≤κp0p \leq \kappa p_0. In the logistic experiments, this condition holds by construction, whereas fitting a reference distribution from a pilot sample does not establish it. It is therefore not clear to me how a user can obtain a useful MSE bound from the acquired data.

(iv) The optimization enumerates possible complete records, so its computational cost is a concern as the problem grows. The main prediction experiments use four binary features, while the dense logistic experiments use at most eight. Table 17 also illustrates the cost of a larger action menu: in the overlapping-menu experiment, solve time increases from 0.0020 to 0.3915 seconds as the menu grows from 4 to 256 actions. Although Proposition 5 provides an approximation class, it does not remove state enumeration. The experiments therefore leave the method’s performance on larger problems unclear.

(v) The practical gains from selecting an approximation and learning an allocation are still limited. In Table 2, bound selection chooses bias-aware full auditing, although the degree-two design has lower actual MSE. Furthermore, paid validation does not improve on selection without validation in any of the six tested settings. When reconstruction is held fixed, Table 4 reports pilot-minus-uniform Brier differences of −0.68, −6.75, and +4.59, with paired standard errors of 9.95, 10.34, and 15.99, respectively, all in units of 10−510^{-5}. Each difference is smaller than its standard error, and the sign changes across families. Thus, the 40 repetitions do not show a clear allocation benefit, but they also do not establish that no benefit exists.

Questions for authors
  • In Table 2, the degree-two design has lower actual MSE, but bound selection chooses bias-aware full auditing. For d=8d=8, N=250N=250, and b=0.5b=0.5, can either selector based on Equation (11) beat the no-validation MSE of 0.00376430.0037643 with a validation size 2≤v≤152\leq v\leq15, chosen before observing the validation data? Please use n=250−vn=250-v and bmain=(125−8v)/(250−v)b_{\mathrm{main}}=(125-8v)/(250-v), so that the comparison includes both the measurement cost and the loss of analysis records.

  • Section 6 minimizes the unweighted sum of target-table variances. However, for a fixed policy and context, the predictor uses the ratio η(z)=tz,1/(tz,0+tz,1)\eta(z)=t_{z,1}/(t_{z,0}+t_{z,1}), where tz,yt_{z,y} is a target-table entry. For positive table entries, could you derive the variance and covariance terms that determine the leading excess Brier risk of the stated learner? Would using the resulting weights, estimated from the same paid pilot, improve the dense-digits allocation results in Table 3?

2 comments

Thank you for the careful review and two concrete questions. We agree that the exposition should separate identification, acquisition design, and prediction. Your questions also distinguish mathematical validity from whether a procedure improves performance at the available budget.

  1. Organization and notation (I–II). We propose to organize the main argument as: observation model → exact identification → minimum-variance exact design → controlled approximation → prediction. Proposition 3 will be identified as a change of estimand, not a prerequisite. For a fixed pre-audit record oo, qa=q(a∣o)q_a=q(a\mid o), and the audit output is the acquired values together with the eventual outcome. Thus Ca(z∣u)C_a(z\mid u) is its conditional distribution given completion uu. It is distinct from the deployment observation. In Eq. (11), aˉj=v−1∑iaj(Ui)\bar a_j=v^{-1}\sum_i a_j(U_i). These definitions belong at first use. We will replace the introductory parity formula with a plain-language example before introducing its mathematics.

  2. What the MSE guarantee assumes (III). You are correct: fitting p0p_0 does not establish p≤κp0p\le\kappa p_0. Theorem 4 makes two different claims. Its pointwise residual bounds bias under every law. Its reference-energy MSE bound requires the stated envelope. Exact reconstruction remains unbiased with an incorrect reference, but its efficiency need not improve. We do not infer a valid envelope from the small pilot.

Without an envelope, the robust formulation already given in Appendix C.2 imposes, for every complete state uu, ∑a,zCa(z∣u)va(z)2/qa≤tall.\sum_{a,z}C_a(z\mid u)v_a(z)^2/q_a\le t_{\rm all}. Then ϵ2+tall/n\epsilon^2+t_{\rm all}/n bounds MSE without a distributional assumption. This is convex but potentially conservative. It supplies an assumption-free alternative, not a claim that pilot data yield a sharp bound. This distinction should appear beside Theorem 4.

  1. Validation at every requested size (question 1). We ran both Eq. (11) selectors for every prespecified v=2,…,15v=2,\ldots,15, using exactly n=250−vn=250-v and bmain=(125−8v)/(250−v)b_{\rm main}=(125-8v)/(250-v). All eight candidates were reconstructed at each remaining budget, without using the evaluation law. Neither selector improved on the no-validation MSE 0.003764260.00376426.

The best raw and confidence-penalized results both occur at v=2v=2: 0.004065480.00406548 and 0.004373030.00437303. These averages sum all 65,536 ordered validation pairs exactly, up to numerical arithmetic. For each larger vv, we averaged exact conditional analysis MSE over 200,000 independent validation blocks. For v≥4v\ge4, even the best constructed candidate under the true law has higher MSE than the no-validation comparator. At v=2,3v=2,3, selection error also matters. Thus the additional measurements cost more than selection saves here. This answers the requested finite comparison, not an impossibility claim for every selector.

  1. Prediction-directed weighting (question 2). Your proposed refinement is well motivated. Fix a policy and context. Let sz=tz0+tz1s_z=t_{z0}+t_{z1}, ηz=tz1/sz\eta_z=t_{z1}/s_z, and let Σ\Sigma be the covariance of the single-record table score. For positive cells and fixed positive coverage, the leading expected excess Brier risk is 1n∑zηz2Σz0,z0+(1−ηz)2Σz1,z1−2ηz(1−ηz)Σz0,z1sz+o(n−1).\frac1n\sum_z\frac{\eta_z^2\Sigma_{z0,z0}+(1-\eta_z)^2\Sigma_{z1,z1}-2\eta_z(1-\eta_z)\Sigma_{z0,z1}}{s_z}+o(n^{-1}). This follows from ∇ηz=(−ηz,1−ηz)/sz\nabla\eta_z=(-\eta_z,1-\eta_z)/s_z and the Brier decomposition. Contexts and policies are then averaged. The covariance term cannot be replaced by diagonal query weights alone. Equivalently, optimize variances of [(1−ηz)gz1−ηzgz0]/sz[(1-\eta_z)g_{z1}-\eta_z g_{z0}]/\sqrt{s_z}.

We fitted these weights from the same paid pilot and reran dense digits at budget 0.8 on the same 40 splits, holding reconstruction, costs, and evaluation fixed. Brier-directed allocation gives 0.2248810.224881, versus 0.2247910.224791 for pilot-reference allocation. The paired difference is +9.00×10−5+9.00\times10^{-5}, with SE 11.17×10−511.17\times10^{-5}. It does not establish an improvement. The expansion is asymptotic, and the pilot weights are estimated. Objective-specific sampling is consistent with Gilbert, Yu & Rotnitzky (Statistics in Medicine, 2014), but does not guarantee a finite-sample win.

  1. Scale and practical gains (IV–V). We agree that Proposition 5 does not remove the 2d2^d state enumeration and that Table 4 establishes no reliable allocation gain. The contribution is a finite-state characterization and joint bias–variance design, not a scalable unknown-structure learner. We will credit reconstruction for its measured gains, not allocation or selection. Li, Zrnic & Candès, Robust Sampling for Active Statistical Inference (NeurIPS, 2025), likewise distinguish valid inference from efficiency under inaccurate uncertainty estimates. Their safeguards are not guarantees of our method.

Thank you for answering my questions and running more experiments. Your replies clarify several points, but I still have concerns. The paper is difficult to follow, and the new results do not show clear benefits from validation or learning which measurements to take. I also agree with the other reviewers that the experiments are limited and that the link between the theory and prediction results needs to be clearer. The planned changes are helpful, but they do not fully address these concerns. Therefore, I will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.