- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies budget-constrained active data acquisition, using side information such as class labels and low-resolution image summaries to select measurements that preserve information about the complete image. A frozen discrete denoising diffusion model, D3PM, estimates uncertainty about unobserved content and supports two acquisition strategies: a one-shot mask generator trained against an entropy surrogate, and sequential acquisition that updates observations and selects uncertain locations over multiple rounds. The experiments examine gradient estimators, budget enforcement, dependence on side information, and the importance of a separate mask state. Sequential acquisition substantially improves reconstruction on MNIST, while a variance-greedy variant also improves reconstruction on CIFAR-10. On fastMRI, however, entropy-based acquisition does not consistently outperform heuristic sampling, and the best results come from static mask search optimized directly against reconstruction error. The paper thus demonstrates the potential of discrete diffusion priors for active acquisition while exposing discrepancies between entropy surrogates and reconstruction quality.
Critical strengths
-
The paper provides useful empirical analysis of optimization and evaluation pitfalls in active acquisition. Rather than reporting final performance alone, the authors examine whether reducing the surrogate objective actually improves reconstruction, and investigate straight-through gradient estimation, budget overshoot, dependence on side information, and ambiguity between masked and observed states. In the tested implementation, straight-through Gumbel-Softmax gradients can point against the objective's estimated gradient, whereas DisARM improves optimization. Enforcing exact sampling budgets also substantially reduces some apparent gains. These analyses help distinguish methodological improvements from implementation and evaluation artifacts and provide useful guidance for subsequent active-acquisition research.
-
Sequential acquisition achieves substantial gains in controlled experiments, supported by targeted ablations. On MNIST at a 10% sampling budget, 16-step label-conditioned sequential acquisition reduces the average number of erroneous pixels from 46.7 under random sampling to 5.63. The authors compare random initial probes, different numbers of acquisition rounds, and priors with and without a separate mask state. Appendix E further evaluates the separate mask state's benefit across five prior fine-tuning seeds. These results support the practical value of updating observations sequentially and distinguishing an unobserved pixel from an observed black pixel.
Critical weaknesses
-
The information-gain formulation is not fully aligned with the acquisition criterion used for the main CIFAR-10 results. Section 4.3 derives a posterior-entropy criterion from the information-gain identity for noiseless single-pixel measurements. However, Section 6, Figure 3, and Appendix C identify the main positive CIFAR-10 results as variance-greedy acquisition. For a discrete variable with eight intensity values, variance and categorical entropy do not generally induce the same ranking, so these results cannot directly establish the effectiveness of entropy-based information-gain acquisition. Moreover, the implementation uses a learned approximate posterior and selects multiple locations per round. Although the paper acknowledges redundancy within batches, its characterization of the sequential method as directly maximizing information gain needs to specify these qualifications. Without a matched comparison of entropy and variance using the same initial probe, acquisition rounds, and budget, it remains unclear which acquisition principle the CIFAR-10 gains support. This weakens the alignment between the paper's central theoretical interpretation and its empirical conclusions.
-
The experiments do not sufficiently establish the framework's claimed utility across downstream tasks or its advantage in a practical acquisition setting. The paper emphasizes an objective independent of a particular downstream model and suggests that one mask can serve multiple tasks. However, the principal evaluations on MNIST and CIFAR-10 concern image reconstruction, generally performed by the same D3PM used to guide acquisition. These results primarily establish reconstruction benefits when the acquisition strategy is paired with that prior, rather than transfer across models or tasks. The independent U-Net B evaluation on fastMRI is a meaningful control, but the best results come from static search optimized against U-Net A's reconstruction error, rather than the proposed D3PM entropy-based strategy. Entropy-based sequential acquisition remains worse than variable-density sampling at several budgets. The MRI experiments therefore provide informative negative findings, but do not establish an advantage of the discrete-diffusion information-theoretic framework in that application. The paper should more clearly distinguish the contribution of its empirical analysis from claims about the general effectiveness of its acquisition method.
Questions for authors
-
Entropy versus variance as the acquisition criterion. The main CIFAR-10 results are labeled variance-greedy, whereas Section 4.3 derives a posterior-entropy criterion. Could the authors define the variance score precisely and compare the two criteria using the same prior, initial probe, budget, and acquisition rounds? If variance-based acquisition accounts for the principal gains, how should the claim that the sequential method directly maximizes information gain be qualified?
-
Transfer of acquired masks across models and tasks. Are there experiments that evaluate the same masks obtained through D3PM-driven acquisition with an independent reconstructor or on a downstream task other than reconstruction? Since the best static-search result on fastMRI is optimized directly against reconstruction error, which current results support the claim that one mask can serve multiple downstream tasks?
We thank the reviewer for the review and run new experiments for these points. "Rev." refers to the revised PDF, and other numbers refer to the original submitted version.
W1/Q1. Entropy vs variance. Same CIFAR-10 prior, label, initial probe, budget and 16 rounds, and only the per-pixel score differs (PSNR in dB, 2 seeds x 1280 images, rev. Table 11):
The entropy criterion of Sec. 4.3 reaches the CIFAR-10 gains, and variance adds at most 0.08 dB. The submitted entropy result (label-greedy, Sec. 6) fell behind because it started from no observations, while the variance rows started after a probe. Fig. 3 now reports entropy.
Variance score: V_i = sum_ch
sumkpi(k)lk2−(sumkpi(k)lk)2, with l_k = k/7, the posterior variance of the intensity summed over channels. Unlike entropy it accounts for the order of intensity bins, so the two need not rank pixels alike, but here they reach the same PSNR.
Qualification. Sec. 1 and Sec. 8 now state that each step greedily maximizes the model's information gain for single noiseless pixels under its learned posterior, and ignores redundancy between the pixels picked within a step. At 32x32 this costs little (rev. Appendix F).
W2/Q2. Transfer across models and tasks. Yes. All masks are now scored by models that take no part in acquisition: (a) a U-Net inpainter, (b1) a classifier on its reconstruction, (b2) a classifier on the raw measurements. Label-free D3PM sequential masks give the lowest inpainting error at every budget on MNIST and CIFAR-10 (on MNIST, 6.1 vs 19.3 errors for the best baseline at s = 0.1) and the highest (b1) accuracy, but not the highest (b2) accuracy (rev. Table 3). The masks therefore transfer across reconstructors and to classification of the reconstruction, though not to every task, and Sec. 1 no longer states that one mask serves many downstream tasks.
On fastMRI we agree, and the revision states it as a negative result.
Analysis vs method claims. We agree. The revised contributions list the method results separately from the findings on one-shot training, and the abstract no longer credits the fastMRI static search to our criterion.