Active data acquisition with side information via Discrete Diffusion priors

Modern learning systems depend on high-fidelity data, yet acquisition is costly: higher measurement fidelity increases power, storage, and the risk of collecting irrelevant content, while aggressive cost reduction can discard information critical to downstream analysis. We address this cost-fidelity trade-off with an information-theoretic framework that balances specificity and generality: acquiring data relevant to a broad set of tasks without tying acquisition to any particular classifier. Mutual information serves as a model-agnostic relevance metric. We formulate budget-constrained pixel selection as: choose a mask distribution $p(x\mid z)$, conditioned on side information $z$, to maximize $I(Y;C)$ between a target discrete image $C$ and a partial observation $Y$, subject to sparsity constraints on the sensing action $X$. Because $H(C)$ is independent of the mask, this is equivalent to minimizing $H(C\mid Y)$. The acquisition problem is inherently hard in high dimensions; when $p(y\mid x,c)$ is unknown, we approximate it with a frozen Discrete Denoising Diffusion Probabilistic Model (D3PM) that provides a differentiable conditional-entropy surrogate. A mask generation network implements $p_{\varphi }(x\mid z)$ with masked-only entropy, empirical sparsity-timestep calibration, and Gumbel-Softmax relaxation. We evaluate the framework on MNIST, CIFAR, and FastMRI datasets, comparing our method against popular techniques such as LOUPE, greedy and variable density selection, etc.