Joint Measurement Design for Prediction under Observation-Policy Shift

A change in measurement policy can alter predictive risk even when the underlying data distribution remains fixed. We study budgeted passive follow-up for prediction under known observation policies, starting from nonignorably missing records. In a finite-state model, exact risk identification requires joint observation of every nonzero interaction in the target-policy loss. Even a simple logistic observation rule can therefore require full-record audits. We formulate a convex program that jointly chooses partial measurements and a loss reconstruction. It controls bias pointwise and mean-squared error under a declared reference envelope. Observable risk contrasts can reduce the information needed to compare predictors. For approximation selection, we distinguish the smallest error bound from the smallest actual error. We construct an unbiased candidate-MSE criterion from a separately charged validation sample and prove a finite-menu selection guarantee. Bias-matched logistic experiments separate the effects of approximation, reconstruction, and allocation. Paid-pilot experiments on repeated semi-synthetic image splits show larger prediction gains from reconstruction than from learning the allocation, whose gains are smaller and inconsistent. Validation and reference fitting can also cost more than they save. The results characterize the information identified by partial measurements, the error left by approximation, and the design decisions supported by measurements an investigator acquires.