acceptodds
Under review as a conference paper at ICLR 2027

Active data acquisition with side information via Discrete Diffusion priors

Abstract

Modern learning systems depend on high-fidelity data, yet acquisition is costly: higher measurement fidelity increases power, storage, and the risk of collecting irrelevant content, while aggressive cost reduction can discard information critical to downstream analysis. We address this cost-fidelity trade-off with an information-theoretic framework that balances specificity and generality: acquiring data relevant to a broad set of tasks without tying acquisition to any particular classifier. Mutual information serves as a model-agnostic relevance metric. We formulate budget-constrained pixel selection as: choose a mask distribution , conditioned on side information , to maximize between a target discrete image and a partial observation , subject to sparsity constraints on the sensing action . Because is independent of the mask, this is equivalent to minimizing . The acquisition problem is inherently hard in high dimensions; when is unknown, we approximate it with a frozen Discrete Denoising Diffusion Probabilistic Model (D3PM) that provides a differentiable conditional-entropy surrogate. A mask generation network implements with masked-only entropy, empirical sparsity-timestep calibration, and Gumbel-Softmax relaxation. We evaluate the framework on MNIST, CIFAR, and FastMRI datasets, comparing our method against popular techniques such as LOUPE, greedy and variable density selection, etc.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
discrete diffusion, side information, information theory, fastmri

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2QxHOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies budget-constrained active data acquisition, using side information such as class labels and low-resolution image summaries to select measurements that preserve information about the complete image. A frozen discrete denoising diffusion model, D3PM, estimates uncertainty about unobserved content and supports two acquisition strategies: a one-shot mask generator trained against an entropy surrogate, and sequential acquisition that updates observations and selects uncertain locations over multiple rounds. The experiments examine gradient estimators, budget enforcement, dependence on side information, and the importance of a separate mask state. Sequential acquisition substantially improves reconstruction on MNIST, while a variance-greedy variant also improves reconstruction on CIFAR-10. On fastMRI, however, entropy-based acquisition does not consistently outperform heuristic sampling, and the best results come from static mask search optimized directly against reconstruction error. The paper thus demonstrates the potential of discrete diffusion priors for active acquisition while exposing discrepancies between entropy surrogates and reconstruction quality.

Critical strengths
  1. The paper provides useful empirical analysis of optimization and evaluation pitfalls in active acquisition. Rather than reporting final performance alone, the authors examine whether reducing the surrogate objective actually improves reconstruction, and investigate straight-through gradient estimation, budget overshoot, dependence on side information, and ambiguity between masked and observed states. In the tested implementation, straight-through Gumbel-Softmax gradients can point against the objective's estimated gradient, whereas DisARM improves optimization. Enforcing exact sampling budgets also substantially reduces some apparent gains. These analyses help distinguish methodological improvements from implementation and evaluation artifacts and provide useful guidance for subsequent active-acquisition research.

  2. Sequential acquisition achieves substantial gains in controlled experiments, supported by targeted ablations. On MNIST at a 10% sampling budget, 16-step label-conditioned sequential acquisition reduces the average number of erroneous pixels from 46.7 under random sampling to 5.63. The authors compare random initial probes, different numbers of acquisition rounds, and priors with and without a separate mask state. Appendix E further evaluates the separate mask state's benefit across five prior fine-tuning seeds. These results support the practical value of updating observations sequentially and distinguishing an unobserved pixel from an observed black pixel.

Critical weaknesses
  1. The information-gain formulation is not fully aligned with the acquisition criterion used for the main CIFAR-10 results. Section 4.3 derives a posterior-entropy criterion from the information-gain identity for noiseless single-pixel measurements. However, Section 6, Figure 3, and Appendix C identify the main positive CIFAR-10 results as variance-greedy acquisition. For a discrete variable with eight intensity values, variance and categorical entropy do not generally induce the same ranking, so these results cannot directly establish the effectiveness of entropy-based information-gain acquisition. Moreover, the implementation uses a learned approximate posterior and selects multiple locations per round. Although the paper acknowledges redundancy within batches, its characterization of the sequential method as directly maximizing information gain needs to specify these qualifications. Without a matched comparison of entropy and variance using the same initial probe, acquisition rounds, and budget, it remains unclear which acquisition principle the CIFAR-10 gains support. This weakens the alignment between the paper's central theoretical interpretation and its empirical conclusions.

  2. The experiments do not sufficiently establish the framework's claimed utility across downstream tasks or its advantage in a practical acquisition setting. The paper emphasizes an objective independent of a particular downstream model and suggests that one mask can serve multiple tasks. However, the principal evaluations on MNIST and CIFAR-10 concern image reconstruction, generally performed by the same D3PM used to guide acquisition. These results primarily establish reconstruction benefits when the acquisition strategy is paired with that prior, rather than transfer across models or tasks. The independent U-Net B evaluation on fastMRI is a meaningful control, but the best results come from static search optimized against U-Net A's reconstruction error, rather than the proposed D3PM entropy-based strategy. Entropy-based sequential acquisition remains worse than variable-density sampling at several budgets. The MRI experiments therefore provide informative negative findings, but do not establish an advantage of the discrete-diffusion information-theoretic framework in that application. The paper should more clearly distinguish the contribution of its empirical analysis from claims about the general effectiveness of its acquisition method.

Questions for authors
  1. Entropy versus variance as the acquisition criterion. The main CIFAR-10 results are labeled variance-greedy, whereas Section 4.3 derives a posterior-entropy criterion. Could the authors define the variance score precisely and compare the two criteria using the same prior, initial probe, budget, and acquisition rounds? If variance-based acquisition accounts for the principal gains, how should the claim that the sequential method directly maximizes information gain be qualified?

  2. Transfer of acquired masks across models and tasks. Are there experiments that evaluate the same masks obtained through D3PM-driven acquisition with an independent reconstructor or on a downstream task other than reconstruction? Since the best static-search result on fastMRI is optimized directly against reconstruction error, which current results support the claim that one mask can serve multiple downstream tasks?

1 comment

We thank the reviewer for the review and run new experiments for these points. "Rev." refers to the revised PDF, and other numbers refer to the original submitted version.

W1/Q1. Entropy vs variance. Same CIFAR-10 prior, label, initial probe, budget and 16 rounds, and only the per-pixel score differs (PSNR in dB, 2 seeds x 1280 images, rev. Table 11):

srandomrandom probe + entropyrandom probe + variancegrid probe + entropygrid probe + variance
0.120.1820.9621.0421.6321.64
0.324.3226.3126.3726.7726.79
0.527.4930.8530.8730.8630.87

The entropy criterion of Sec. 4.3 reaches the CIFAR-10 gains, and variance adds at most 0.08 dB. The submitted entropy result (label-greedy, Sec. 6) fell behind because it started from no observations, while the variance rows started after a probe. Fig. 3 now reports entropy.

Variance score: V_i = sum_ch

sumkpi(k)lk2−(sumkpi(k)lk)2sum_k p_i(k) l_k^2 - (sum_k p_i(k) l_k)^2

, with l_k = k/7, the posterior variance of the intensity summed over channels. Unlike entropy it accounts for the order of intensity bins, so the two need not rank pixels alike, but here they reach the same PSNR.

Qualification. Sec. 1 and Sec. 8 now state that each step greedily maximizes the model's information gain for single noiseless pixels under its learned posterior, and ignores redundancy between the pixels picked within a step. At 32x32 this costs little (rev. Appendix F).

W2/Q2. Transfer across models and tasks. Yes. All masks are now scored by models that take no part in acquisition: (a) a U-Net inpainter, (b1) a classifier on its reconstruction, (b2) a classifier on the raw measurements. Label-free D3PM sequential masks give the lowest inpainting error at every budget on MNIST and CIFAR-10 (on MNIST, 6.1 vs 19.3 errors for the best baseline at s = 0.1) and the highest (b1) accuracy, but not the highest (b2) accuracy (rev. Table 3). The masks therefore transfer across reconstructors and to classification of the reconstruction, though not to every task, and Sec. 1 no longer states that one mask serves many downstream tasks.

On fastMRI we agree, and the revision states it as a negative result.
Analysis vs method claims. We agree. The revised contributions list the method results separately from the findings on one-shot training, and the abstract no longer credits the fastMRI static search to our criterion.

Reviewer_wew5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an information-based framework for active data acquisition using a pre-trained discrete diffusion prior. It explores a one-shot mask policy trained with DisARM and sequential measurement selection guided by the prior’s predictive uncertainty. The framework is evaluated on reconstruction tasks using MNIST, CIFAR-10, and fastMRI. The paper also examines gradient estimation, mask-state encoding, and the relationship between the entropy surrogate and reconstruction quality.

Critical strengths
  • The paper is well-written and easy to follow.

  • The discrete prior enables direct per-pixel entropy computation with a single forward pass per acquisition round, avoiding posterior sampling for entropy-based acquisition scoring

  • The mask-state ablation provides useful empirical evidence that distinguishing unobserved pixels from observed background values improves reconstruction with the absorbing diffusion prior.

  • Extensive experiments are conducted across three tasks.

Critical weaknesses

Limited methodological novelty. The core idea of using a pre-trained diffusion model to guide sequential measurement selection through an entropy-based criterion has already been explored in Active Diffusion Subsampling (Nolan et al., 2025). Although the proposed method introduces a discrete absorbing prior and direct per-pixel entropy computation, the fundamental acquisition strategy remains unchanged. The sequential method therefore appears to be an incremental extension of ADS.

Missing comparisons with DPS variants. The paper lacks comparisons with closely related task-adaptive learned subsampling methods, including DPS, A-DPS, and PGA-DPS [1–3], which use straight-through Gumbel-Softmax for sampling optimization. Although the authors report unfavorable STGS gradient alignment under their entropy-based objective, this analysis does not substitute for comparisons with these methods, which optimize task-specific losses.

The MRI results further motivate these comparisons: at a 25% sampling budget, the learned policy achieves the lowest entropy surrogate but underperforms heuristic masks in reconstruction quality, while a static mask optimized against the reconstructor performs best. PGA-DPS is particularly relevant because it combines prior-based initial sampling with context-guided active acquisition. Comparisons under matched ACS constraints and total measurement budgets are needed to establish the practical benefit of the proposed approach.

Availability of label information at test time. The MNIST experiments use ground-truth class labels as side information for active sampling. Although this is a valid conditional reconstruction setting, the availability of such labels in practical acquisition scenarios is not sufficiently justified. A label-free evaluation is needed to establish whether the reported gains persist without oracle class information.

Insufficient validation of task-agnostic acquisition. A central motivation is to acquire information useful across downstream tasks rather than optimize a particular task model. However, the evaluation focuses predominantly on reconstruction, and the MNIST and CIFAR-10 experiments use the same prior for acquisition and reconstruction. It therefore remains unclear whether the selected measurements provide benefits across tasks or mainly favor reconstruction under that prior. Evaluation across multiple downstream tasks or independent models would better substantiate the claimed broader utility.

References:

  1. Huijben et al., “Deep Probabilistic Subsampling for Task-Adaptive Compressed Sensing,” ICLR 2020.

  2. Van Gorp et al., “Active Deep Probabilistic Subsampling,” ICML 2021.

  3. Kang and Seo, “Prior-aware and Context-guided Group Sampling for Active Probabilistic Subsampling,” ICLR 2026.

Questions for authors

Please address the concerns raised in the Weaknesses section, along with the following additional questions:

  1. In the MRI experiments, how much does the entropy term contribute beyond the zero-filled reconstruction loss? An ablation removing only the entropy term would clarify the contribution.

  2. Section 7.1 states that a 32-line ACS block is always sampled. Does this apply to all compared methods, including LOUPE, with ACS lines counted toward the total measurement budget? Please confirm that LOUPE and ACS + variable density were evaluated under matched acquisition constraints.

  3. Figure 3 reports variance-greedy acquisition on CIFAR-10, whereas the proposed information-gain criterion is entropy-based. Can the authors compare entropy-greedy and variance-greedy acquisition using the same initial probe, measurement budget, and number of acquisition rounds to clarify whether the reported gains support the proposed criterion?

  4. The one-shot policy is trained with independent Bernoulli masks but evaluated using budget-exact noisy top-k masks. Since DisARM provides an unbiased gradient estimator for the Bernoulli training objective, how does this distribution mismatch affect the entropy objective and reconstruction performance at evaluation?

3 comments

We thank the reviewer for the review and run new experiments for these points. "Rev." refers to the revised PDF, and other numbers refer to the original submitted version.

W1. Novelty vs ADS. We agree that the sequential maximum-entropy principle is not new, and the revision says so. What the discrete prior adds is the information gain of every pixel in closed form from one forward pass per round: 17 passes per MNIST image, against about 16,000 for ADS. Run from its public code (rev. Appendix F), ADS makes more errors at s <= 0.1 (11.4 vs 6.1 at s = 0.1) and slightly fewer at s >= 0.2, and is about 2000x slower. The other contributions are the separate mask state and the analysis of one-shot training.

W2. DPS, A-DPS, PGA-DPS. We ran DPS with the same U-Net and loss as our LOUPE, so the two differ only in the sampler. On MNIST it is the best static baseline but makes more errors than label-free sequential acquisition (19.3 vs 6.1 at s = 0.1, rev. Table 3). On fastMRI, ACS-locked at the same total budget, it is worse than ACS + VD and our static search at every budget (rev. Table 5). A-DPS and PGA-DPS are now discussed in related work. We have since implemented both on fastMRI with the same U-Net, loss and ACS-locked budget as our DPS. Under U-Net B, neither reaches ACS + VD at any budget (e.g. A-DPS 0.02499 and PGA-DPS 0.02723 vs 0.02471 at s = 0.25), except PGA-DPS at s = 0.75 (0.00757 vs 0.00764, static search 0.00751). In our implementation both policies choose nearly the same rows for every slice, with the temperature annealed or fixed at 2 as in the PGA-DPS paper. We have spent only a few days on these runs, so we treat them as preliminary and do not include them in the revised paper, we'll continue working on this until the final revision. Like our static search, they optimize a fixed task loss, and we agree with the reviewer's MRI reading: with a fixed, known reconstructor, optimizing the mask against it is the better choice, and the revision says so.

W3. Labels at test time. Label-free versions of every sequential experiment are now reported beside the labelled ones (rev. Sec. 6.1). On MNIST, label-free acquisition makes 7.0 errors at s = 0.1 against 52.5 for random (4.6 with the label), and on CIFAR-10 it loses at most 0.08 dB.

W4. Task-agnostic acquisition. Masks are now scored by models that take no part in acquisition: (a) a U-Net inpainter, (b1) a classifier on its output, (b2) a classifier on the raw measurements. Label-free D3PM sequential masks give the lowest inpainting error at every budget on both datasets and the highest (b1) accuracy, but not the highest (b2) accuracy (rev. Table 3). We now state the claim at that scope.

Q1. Entropy term in the MRI policy. Removing it improves the policy at s <= 0.4 (0.0267 vs 0.0284 at s = 0.25) and is level at s >= 0.5, so the term does not help on MRI. Both variants still trail ACS + VD.

Q2. ACS matching. All methods except LOUPE kept the 32 ACS rows, counted in the same total k. LOUPE used the same total k without the lock. An ACS-locked LOUPE improves at every budget but stays behind our static search (0.02449 vs 0.02389 at s = 0.25, rev. Table 5). Sec. 7.1 now states this.

Q3. Entropy vs variance. With the same prior, label, probe, budget and 16 rounds, the two scores differ by at most 0.08 dB (rev. Table 11). The submitted entropy result fell behind because it started from no observations. Fig. 3 now reports entropy.

Q4. Bernoulli training vs budget-exact evaluation. Bernoulli draws overshoot the budget by 1-11%. At s = 0.1 this accounts for about 2 errors, while mask placement accounts for about 24 (rev. Table 13). The entropy objectives agree to 0.0004 nats.

Thanks for the substantial additional experiments. The rebuttal addresses most of my concerns, and I am raising my score to 3 (weak acceptance).

I encourage the authors to include the finalized A-DPS and PGA-DPS comparisons in the camera-ready version.

We thank the reviewer for the quick feedback. We'll continue to work on the A-DPS and PGA-DPS until the final revision.

Reviewer_ZLeMOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies budgeted active sensing, where a policy selects a measurement mask to maximize mutual information between observations and a target. A frozen discrete diffusion (D3PM) supplies a per-pixel conditional-entropy surrogate used to train a one-shot mask generator or a sequential greedy policy. Experiments span MNIST, CIFAR-10, and single-coil fastMRI, comparing against random, LOUPE, and learned baselines. The method is intended to provide a task-agnostic acquisition criterion.

Critical strengths
  1. Framing acquisition as maximizing I(Y; C) under a budget, with a diffusion prior supplying a tractable conditional-entropy surrogate, avoids tying the policy to one downstream classifier, and the paper is rigorous about what is a surrogate versus the true criterion. The observed-pixel/whole-image comparison (Sec. 4.1) and the use of the unbiased DisARM estimator rather than Gumbel-Softmax (Sec. 5.1, Appendix A) show the right level of care.
  2. On MNIST at budget s = 0.1, label-greedy sequential acquisition with 16 steps makes 5.63 errors per image against 46.7 for random masks (roughly 8× fewer), and reconstructs 26% of test images exactly (Table 2), with the prior concentrating steps on the digits rather than the background. The paper then reports plainly that the same criterion gives only 0.3–0.7 dB on CIFAR-10 and disagrees with reconstruction quality on fastMRI, rather than hiding these results.
Critical weaknesses
  1. On fastMRI, the learned policy using the entropy surrogate gains only 14.9% NMSE reduction, behind a static greedy row-swap search (29.4%) and LOUPE (27.2%) (Table 3); the best result comes from a baseline adaptation rather than the proposed surrogate. On MNIST, the one-shot generator's numbers depend substantially on being given the target's own 8×8 summary (23.0 errors) versus no summary (26.4) or random (46.7) (Table 1) — a low-resolution version of the target being reconstructed, which would not be available in a genuine one-shot setting. On CIFAR the gains are 0.3–0.7 dB and the target summary barely helps, so the one-shot method is not demonstrated to be useful outside the easy, side-information-rich case.
  2. The image experiments are at 32×32 with noiseless pixel measurements and 3–9 state quantization; the motivating applications (adaptive LiDAR, event cameras, real-time sensing) are not tested. The mapping from budget to diffusion timestep via a Monte Carlo survival curve is a calibration heuristic, and the subadditivity bound ignores inter-pixel dependencies, which the paper notes can lead greedy steps to pick redundant pixels.
Questions for authors

1.In the one-shot setting, what is the gain when the side information is genuinely available at acquisition time rather than being the target's own summary (e.g., a neighboring frame or a previous low-resolution scan)? The "target's own summary" condition appears to leak the target; please report the realistic Z = prior-frame or class-only setting as the primary number. 2.On fastMRI, why does the entropy surrogate rank masks so differently from reconstruction NMSE, and is the mismatch due to the prior ignoring intensity order? Would an energy-based or NMSE-aware criterion (the static search) always be preferable when a reconstructor is available, and what is the argument for the entropy surrogate in that case? 3.Can the sequential method be tested at a realistic scale (e.g., 256×256 or actual k-space with noisy measurements), and does the dependency-aware correction (avoiding redundant pixels) improve over the per-pixel greedy policy? 4.How sensitive are the results to the budget-to-timestep calibration, and what happens when the budget corresponds to a timestep outside the prior's training distribution (off the 100–1000 step range on MNIST)?

1 comment

We thank the reviewer for the review and run new experiments for these points. "Rev." refers to the revised PDF, and other numbers refer to the original submitted version.

Q1/W1. Realistic side information. Agreed: the target's own summary is an oracle, and rev. Table 1 now lists the realistic rows first. With class-only Z, the MNIST generator makes 26.4 errors at s = 0.1 against 46.7 for random, and the target's summary adds little beyond that (23.0). Without any side information, label-free sequential acquisition makes 7.0 errors. On fastMRI, Z is a scout from the measured ACS rows, the "previous low-resolution scan" case. On CIFAR-10 we agree that the one-shot gain requires the target's summary, and the revision states this.

W1, fastMRI numbers. A correction: in Table 3, 29.4% is ACS + random and 27.2% our static search, and LOUPE is not in that table. The conclusion stands: under U-Net B the learned policy trails every heuristic.

Q2. Entropy vs NMSE on fastMRI. (a) Low entropy comes from rows near the k-space center, which a reconstructor already predicts, so the lowest-entropy mask gains least from U-Net B (Table 3). A further 256x256 experiment shows that for global measurements such as k-space rows, posterior uncertainty locates the error poorly (rev. Sec. 8). (b) Not mainly. Ranking rows by expected energy, an order-aware score, is worse than entropy in all 12 settings of Table 8. Order does matter on CIFAR-10 (rev. Appendix F). (c) Energy-based: no (Table 8). NMSE-aware: yes, when the reconstructor and task are fixed and known, and the revision says so. (d) Entropy is meant for the case where they are not: label-free D3PM masks give the lowest error under an independent inpainter at every budget on MNIST and CIFAR-10 (rev. Table 3).

Q3. Scale, noise, dependency-aware correction. Yes. In a further experiment on 256x256 faces with noisy pixels (sigma = 0.05), sequential acquisition lowers MSE by 26.0% against the best static mask, on 64 of 64 held-out images. The dependency-aware correction is what makes this work at that scale: per-pixel top-k gains only 0.2%. At 32x32 it matters little (rev. Appendix F), and the limitations now say so. On noisy MNIST pixels the D3PM criterion matches or outperforms a noise-aware score up to sigma = 0.35 (rev. Table 15).

Q4. Timestep calibration. We shifted t(s) by an offset at every prior query. Sequential acquisition, random at offset 0 in brackets (rev. Table 12):

offset-200-100-500+50+100
MNIST s=0.1, errors [46.1]5.04.54.85.37.47.8
MNIST s=0.3, errors [23.6]0.000.010.000.010.010.01
CIFAR s=0.1, dB [20.1]11.417.620.320.920.920.0
CIFAR s=0.3, dB [24.3]24.725.926.126.326.426.4

Results are stable at s = 0.3 for offsets up to 100 steps and sensitive at s = 0.1, where sequential acquisition still outperforms random at the same offset from -50 to +100 (20.0 vs 19.5 dB at +100). No reported budget falls outside the prior's range: on MNIST t(s) runs from 967 (s = 0.05) to 195 (s = 0.95).

W2. Real sensors. Agreed. LiDAR and event cameras are now motivation only, and the 32x32 noiseless setting is stated explicitly.

Comments on the paper

1 comment

We thank the reviewers and the AC. We ran new experiments for these concerns, and the revised PDF (changes in blue) makes these changes.

  • Claims recalibrated. The information-gain claim is qualified, the task-agnostic claim is stated at the scope the new transfer results support, and fastMRI is stated as a negative result with its explanation.
  • Shared concerns answered with new experiments: entropy vs variance with matched settings (2QxH, wew5), label-free and realistic side information (ZLeM, wew5), and transfer to independent models and classification (2QxH, wew5).
  • New baselines and checks: DPS, ADS, ACS-locked LOUPE and DPS on fastMRI, timestep calibration, pixel noise, and a 256x256 experiment (ZLeM, wew5).
  • Structure. The contributions present the empirical analysis separately from the method results. New results are in Sec. 6.1 and Appendix F.

The individual replies give the specific numbers. We will update this reply if there are other changes in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.