acceptodds
Under review as a conference paper at ICLR 2027

Sparse Data Diffusion via Sparsity Bits

Abstract

Sparse data is fundamental to scientific simulations in physics and biology, from particle physics experiments to single-cell RNA sequencing, where exact zeros encode absence rather than a weak signal. However, existing generative models struggle to faithfully capture exact zeros, often blurring absence with weak signals, calling for an explicit representation of exact zeros. We introduce Sparse Data Diffusion (SDD), a novel method for generating sparse data. SDD extends continuous state-space diffusion models with an explicit representation of exact zeros by modeling sparsity through the introduction of Sparsity Bits. Empirical validation in particle physics and single-cell RNA sequencing demonstrates that SDD achieves higher quality than baselines in capturing the sparse patterns inherent to these domains, while preserving the quality of the generated data.

Primary area:
generative models
Keywords:
diffusion models, data sparsity, sparse data generation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_uMQmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors provide a diffusion model that explicitly models sparsity by diffusing discrete SBs along continuous variables in a shared continuous state-space. The Sparse Data Diffusion (SDD) model aims to generate sparse data. The SDD extends continuous state-space diffusion models with an explicit representation of exact zeros by modeling sparsity through the introduction of Sparsity Bits.

Critical strengths
  1. The problem of generating sparse data is interesting.
  2. The paper is clearly presented.
Critical weaknesses
  1. The main concern is the novelty of the work. In terms of the technical contribution, the proposed SDD method is a limited extension of the previous diffusion model.
  2. The effect of the SDD model largely depends on the denoising neural network fθf_\theta. However, the design of fθf_\theta is not discussed.
Questions for authors

Please justify the proof of proposition1. The current version is not very clear.

1 comment

We thank Reviewer uMQm for the feedback and address each concern below.

Novelty

SDD is intentionally simple, but to our knowledge it is the first diffusion model to jointly diffuse discrete and continuous variables in a shared continuous state space (Section 4). Without domain-specific design, it outperforms domain-specific and sparsity-aware baselines.

Denoising network design

Section 3.1 (paragraph "DM architecture & training") describes the denoising networks. SDD uses the same architectures as the DDPM/DDIM baselines, changing only the input and output dimensions, which isolates the effect of the Sparsity Bits. Full details are in the supplementary code.

Reviewer_ZhPNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a modelling choice to improve diffusion in sparse data. They do so through a) concatenating a binary sparsity channel to the images to explicitly model sparsity b) Add a cross-entropy term in loss for this sparsity channel. c) Binarize the sparsity channel of the generated sample and elementwise multiply to the actual content. They show the performance of this model in two scientific use cases: a) muon signal and background and b) scRNA, and show improvement in performance either quantitatively or qualitatively for these data, mainly over a previous benchmark SED.

Critical strengths
  1. The paper is easy to follow. The motivations of the paper are clear, and the preliminary reasoning behind using the methodology described in the paper is also clear.
Critical weaknesses
  1. In the case of the MNIST (App.F), the sparsity map seems to be essentially equal to the content (there is only a minority of pixels that have intermediate values); if so, isn't the heavyside step function the only difference between SDD and vanilla diffusion in such cases, meaning explicitly modelling the sparsity map doesn't help where data magnitude is close to binary, sort of limiting the setting defined in Eq. 1?
  2. The claim for no post-hoc processing in SDD in various places in the paper is wrong, as the heavy-side step function on sparsity is effectively thresholding similar to DDPM-T.
  3. The theoretical analysis and proof seem to be trivially true by construction, as the marginal prob of a content bit being zero is equal to the prob of a sparsity bit being zero. I am finding it difficult to understand what value it adds to the paper.
Questions for authors
  1. The one mentioned in weakness 1.
  2. Can you justify the theoretical value of preposition 1?
1 comment

We thank Reviewer ZhPN for the feedback and address each concern below.

Sparsity maps and content (Appendix F)

We believe there is a misunderstanding about Figures 11 and 12. For each method, the left column shows the generated images and the right column shows their sparsity pattern (zero vs. nonzero pixels). The right column does not show the Sparsity Bits predicted by the model.

Heaviside step vs. post-hoc thresholding

We agree that the Heaviside step is a thresholding operation. It is a necessary consequence of modeling the binary Sparsity Bits in a continuous state space, and it differs from post-hoc thresholding in what is thresholded. DDPM-T and DDIM-T apply a global threshold to the generated values, tuned to match the dataset's average sparsity, which removes all low-intensity values. SDD instead applies a fixed threshold of 0 to the learned Sparsity Bits, so the model decides for each dimension whether it is zero, and small nonzero values are preserved. We have clarified this distinction in Section 2.3 of the revised version.

Reviewer_ijB2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Sparse Data Diffusion (SDD), a diffusion framework for generating sparse data. The method introduces sparsity bits to explicitly represent zero and nonzero entries. It jointly models continuous values and sparsity bits through Gaussian diffusion and applies a binary mask to produce exact zeros during generation. Experiments on particle physics, single-cell RNA sequencing, and image datasets show improvements in sparsity modeling and several data quality metrics.

Critical strengths
  1. The paper addresses a meaningful problem in sparse data generation. Explicitly modeling zero entries is well motivated, especially for scientific data where sparsity patterns carry important information.

  2. The proposed method is simple and easy to integrate with existing diffusion frameworks. It does not require major changes to the forward or reverse diffusion process.

  3. The experiments cover different domains and include comparisons with standard diffusion, threshold-based methods, and specialized baselines. The results show clear improvements in sparsity distribution matching and several scientific evaluation metrics.

Critical weaknesses

Weaknesses

  1. The benefit of joint diffusion is unclear. The sparsity bits are still modeled through a continuous Gaussian process, without explicit discrete states at intermediate steps. Why is this better than first generating a binary mask through diffusion and thresholding, then generating continuous values conditioned on the mask? Both approaches can model dependencies between sparsity and values.

  2. The theoretical result is weak. Proposition 1 only bounds the marginal zero probability using bit-wise prediction error. This is straightforward and does not address the joint distribution of sparsity patterns or their relationships with continuous values, which are central to the paper's motivation.

  3. The role of the loss function is underexplored. The improvement may come mainly from the additional cross-entropy supervision rather than joint diffusion itself. Why use cross entropy instead of zero-sensitive reconstruction losses? A comparison of different objectives would help clarify the source of the improvements.

Questions for authors
  1. Can the authors explain the advantage of joint diffusion over a two-stage approach that first generates a sparsity mask and then generates continuous values conditioned on it? Is there evidence that joint diffusion better captures their dependencies?

  2. How much of the improvement comes from the additional cross-entropy loss rather than the joint diffusion design? Would a standard diffusion model with a zero-sensitive loss achieve similar results?

1 comment

We thank Reviewer ijB2 for the constructive feedback. We address each concern below.

Benefit of joint diffusion over a two-stage process

In SDD, the sparsity mask and the continuous values are denoised together, so at every step each can adapt to the current estimate of the other. A two-stage approach fixes the mask first and cannot revise it once values are generated, so errors in the mask carry over directly to the final sample. It also requires two models and two sampling passes, whereas SDD uses a single network.

Loss function

As described in our general response, no experiment uses a cross-entropy term. SDD and the DDPM/DDIM baselines share the same architecture and the same ℓ2\ell_2 objective, so the improvements can be attributed to the Sparsity Bits rather than to an additional loss term.

Reviewer_VKExOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper tackles the problem of generating continuous data with sparse bits. In this setting, standard diffusion models struggle to capture exact zero bits, blurring absence with weak signal. Sparse Data Diffusion is proposed as a solution, which jointly diffuses the data and a sparse signal, which represents whether each dimention is zero or not. Validation is the method includes: calorimeter images, scRNA data, and sparse image generation.

Critical strengths

The paper is clearly written and considers an important problem of sparse data generation. The method is easy to follow and simple to use. Compare to previous baseline SED, this method preserves the spatial patterns in the dataset that are meaningful for real-world applications ranging from biology to physical.

Critical weaknesses
  1. The algorithm is computationally heavier than the baseline algorithm (SED). For a d-dimensional distribution, the output dimension of the proposed algorithm is 2d regardless of how sparse the data is. As a comparison, SED exploits the fact the the underlying sparse structure of the problem and learns a score function that can be have significantly lower dimensions than the target distribution, depending on how sparse the distribution is. The above fact also raises concern on the algorithm performance in higher-dimensional problems.

  2. Data for sparse bits doesn't seem balanced in the datsets. The sparsity levels of the benchmarks are reported as: ``95.2% for Muon Signal, 97.9% for Tabula Muris, 50.2% for Fashion-MNIST, 95.2% for Muon Background, 95.9% for Human Lung Pulmonary Fibrosis, and 80.9% for MNIST.'' Such high sparsity levels imply imbalanced label distribution for sparse bits, and could possibly lead to issues during training without further implementation adaptation.

  3. The algorithm design follows the simple idea of jointly diffusing the continuous data and sparse bits. The paper could benefit from discussing and comparing with other design ideas in modeling mixed-type data distributions. As an example, prior work [1] uses separate diffusion models to diffuse continuous and discrete data, and carefully designs the coupling of the two diffusion models. It seems such algorithms can be directly adapted to the setting considered in this work. Additionally, I believe an ablation study that compares the joint diffusion and separate diffusion would be helpful and bring insights into the algorithm design. [1]. CoDi: Co-evolving Contrastive Diffusion Models for Mixed-type Tabular Synthesis.

Questions for authors

Please see the "critical weaknesses" section for critical questions that materially affect your assessment.

1 comment

We thank Reviewer VKEx for the positive assessment and address each concern below.

Resource usage

SDD represents each sample with 2d2d dimensions, but the overhead is small in practice. Per-step training and sampling times are virtually identical to DDPM on all datasets (Table 3, Appendix B). SED is more efficient by design, but its compression and autoregressive decoding reduce diversity and introduce artifacts (Figures 1 and 4), which SDD avoids. We have expanded the discussion of this trade-off in the revision.

Imbalanced SB label distribution

Although the SBs are highly imbalanced (up to 97.9% zeros), we observed no negative effect on training and needed no rebalancing. The predicted SB logits concentrate sharply at ±1\pm 1, and the fraction at −1-1 closely matches the true sparsity even on the most imbalanced datasets (Figure 7, Appendix C). SDD also reproduces the full per-sample sparsity distribution more faithfully than all baselines (Figures 3 and 6).

Comparison to CoDi

CoDi assumes each column's type (continuous or categorical) is known in advance and fixed across samples. In our setting (Equation 1), whether a coordinate is exactly zero varies per sample, so CoDi cannot be applied to the raw data directly. The approaches differ in modeling: SDD diffuses both parts jointly with one network, whereas CoDi couples two separate diffusion models. We have added this discussion to the related work.

Reviewer_y6gQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Many tasks contain sparse data where most of the dimensions are zero. Standard diffusion models do not take sparsity into consideration during training and sampling. The authors tackle this problem by modeling indicator dimensions, called Sparsity Bits (SBs), to indicate which dimensions are zero and which are non-zero. They show performance on par with or better than previous works on biology, physics, and image datasets.

Critical strengths
  1. Modeling sparsity using additional sparsity bits is a natural approach and seems effective.
  2. The authors tackle a unique problem which is not widely addressed in the field.
Critical weaknesses
  1. A simple training-free approach is to use DDPM or DDIM but add a thresholding step after obtaining each x^0\hat{x}_0. To my understanding, this baseline is missing in the current version. Please see question 1 for more details.
  2. While it is natural to additional model the sparsity bits, SDD doubles the dimensionality of the data, resulting in slower inference. The authors should report the runtime overhead and/or compare with the baselines under matched compute, especially since latent methods such as SED claim faster inference by ignoring the zero values. It would also be good to add more discussion on this point.
  3. For some datasets such as MNIST, pixel values are naturally quantized. This would enable the use of discrete diffusion which would mitigate the need to additionally model the sparsity bits, and may also result in better or more efficient performance. Could the authors add a discussion with or compare SDD with discrete diffusion?
Questions for authors
  1. To my understanding (please correct me if I am wrong), DDPM-T and DDIM-T apply post-hoc thresholding to the final generated samples. However, it is more natural to apply thresholding to each intermediate x^0\hat{x}_0 instead since this would mimic a training-free version of SDD. Could the authors clarify where thresholding is performed in DDPM-T and DDIM-T? And if thresholding is only performed on the generated sample, could they also add an experiment comparing x^0\hat{x}_0-thresholding with SDD? I believe this a critical setting that will demonstrate if SDD is more beneficial than a naive training-free modification.
  2. Equation 7 and Algorithm 1 suggest using cross-entropy loss to model the SBs but line 192 propose using the ℓ2\ell_2 loss to model the SBs. Which loss is used for the experiments? And could you also briefly discuss why that loss is chosen (the main paper current suggests using ℓ2\ell_2 loss to match the diffusion/flow-matching objective).
  3. Could the authors add a discussion with or compare SDD with discrete diffusion? (Please refer to weakness 3 for details)
4 comments

We thank Reviewer y6gQ for the constructive feedback. We address each point below.

Thresholding at each step

DDPM-T and DDIM-T threshold only the final generated samples (Section 3.1). Following the reviewer's suggestion, we have added per-step variants, DDPM-ST and DDIM-ST, which threshold each intermediate x^0\hat{x}_0 estimate. SDD outperforms both variants on all datasets. Zeroing values during sampling pushes the trajectory away from the noisy data the denoiser was trained on, and these errors compound over the sampling steps.

Runtime overhead

Table 3 (Appendix B) reports per-step training and sampling times. Since DDPM and SDD share the same architecture apart from the Sparsity Bits, this comparison isolates their cost. The per-step times are virtually identical across all datasets.

Discrete diffusion

Our method targets continuous data with exact zeros (Equation 1), which is the case for our main applications, calorimeter images and scRNA data. Discrete diffusion does not apply to this setting. We discuss this distinction in the related work (Section 4, paragraph "Diffusion models") and have made it more explicit in the revision.

We thank the reviewer for their new experiments and explanations. I have several clarifying questions regarding the new results.

Stepwise Thresholding

I have a concern and clarifying question regarding the explanation for why DDPM/DDIM-ST performs poorly. They say that zeroing values during sampling pushes the trajectory away from the noisy training distribution resulting in compounding errors. However, stepwise thresholding only zeros the intermediate x^0\hat{x}_0 estimate and not the intermediate samples xtx_t. Based on the DDPM/DDIM update, the thresholded intermediate x^0\hat{x}_0 are noised back to time t, so there should be no issues with out-of-distribution inputs. Please refer to Equation 12 of https://openreview.net/forum?id=St1giarCHLP. Could the authors please clarify if they re-added noise to the thresholded intermediate x^0\hat{x}_0 at each step, and explain the degraded performance again?

Runtime overhead

We thank the reviewer for this additional comparison. The runtime comparison with DDPM looks promising.

Thanks for the reply. We address the raised point below:

Stepwise thresholding

Yes, the thresholded x^0\hat{x}_0 is re-noised at each step: DDPM-ST and DDIM-ST use the standard updates (Eq. 12 of Song et al., 2021), so xt−1x_{t-1} itself is not thresholded. The degradation instead comes from a bias toward zero. At high noise levels, x^0\hat{x}_0 is a smooth estimate, and the threshold matched to the dataset sparsity zeros most of it, including weak but real signal. The update moves xt−1x_{t-1} toward this sparsified estimate, so the next prediction contains less signal and is thresholded again. Re-noising does not correct this bias, so it accumulates over the steps. We changed the description in Section 3.5 and Appendix F accordingly.

We thank the reviewer for their response. Most of my concerns have been addressed and I raise my score to 3.

Comments on the paper

1 comment

We thank all reviewers for their time and constructive feedback. We have revised the paper accordingly.

Loss formulation. All reported experiments use a plain ℓ2\ell_2 loss applied across all dimensions, as already stated in the original submission. This ensures comparability, as SDD and the DDPM/DDIM baselines then share the same objective and differ only in the additional SB dimensions. Section 2.2 and Algorithm 1 now state this loss directly.

Proof. We expanded the scope of the theoretical result from the marginal zero probability of single coordinates to the joint sparsity patterns of generated samples (Section 2.4). A short version is given in the main paper, and the full proof in Appendix G.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.