acceptodds
Under review as a conference paper at ICLR 2027

XGB-DDPM: Noise-Refreshed Gradient Boosting for Scalable Diffusion on the CPU

Abstract

We present a method to optimize stochastic denoising objectives with Gradient-Boosted Decision Trees (GBDTs) by refreshing noise realizations across boosting rounds. Previous efforts relied on expanding the training data over many noise realizations, incurring extreme memory and runtime costs, especially for already-large datasets. We implement this method in an XGBoost fork and use it to construct XGB-DDPM: a DDPM-style tabular diffusion model combining Gaussian and multinomial diffusion for numerical and categorical features, which are denoised by XGBoost regressors and classifiers. Here, XGBoost leverages the round-wise noise refresh to estimate the diffusion loss. We show that XGB-DDPM can be trained more efficiently and scales to much larger datasets than previous GBDT-based diffusion models which expand the training data to estimate the diffusion loss. Additionally, we show that XGB-DDPM achieves similar fidelity to neural tabular diffusion models (e.g., TabDDPM, TabSyn, TabDiff, and CDTD) without increasing memorization or membership disclosure risk. XGB-DDPM is a good alternative for practitioners who want to train diffusion models on potentially large data but do not have access to GPUs.

Primary area:
generative models
Keywords:
Generative model, Diffusion model, Gradient Boosting, XGBoost, Tabular synthesis, Tabular data generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uP9aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors proposed a GBDT-based diffusion model for tabular data generation. Their main contribution is a round-wise noise refresh mechanism that allows XGBoost to optimize stochastic denoising objectives without explicitly expanding the training dataset. Experiments demonstrate competitive generative fidelity and improved computational scalability compared with existing GBDT-based diffusion methods

Critical strengths
  1. The proposed noise-refresh mechanism addresses a computational bottleneck in GBDT-based diffusion training. The experiments demonstrate substantial reductions in training time compared with dataset duplication while maintaining competitive generation quality.

  2. The paper compares against several established tabular generative models across multiple datasets, including a large-scale benchmark. The ablation studies also provide useful evidence for the effectiveness of the proposed training strategy.

Critical weaknesses

One of the main concerns is that the theoretic analysis establishes that the stochastic empirical loss is an unbiased estimator of the expected diffusion loss, conditioned on the current boosted model. However, it alone does not guarantee that the decision tree fitted using the refreshed noise provides a descent direction for the expected objective. In particular, the newly fitted tree depends nonlinearly on the sampled noise realizations, and unbiased loss or gradient estimation does not imply unbiasedness of the resulting boosting update. Then, neither expected loss reduction nor convergence of the proposed stochastic boosting procedure is established. Since round-wise noise refresh is the central methodological contribution, the absence of a theoretical characterization of its optimization behavior limits the mathematical foundation of the proposed approach.

Questions for authors

1.Can the authors establish sufficient conditions under which the proposed stochastic boosting update yields an expected decrease in the diffusion objective? Is it possible to provide a convergence or stability analysis accounting for the stochastic noise refresh and the approximation error introduced by fitting decision trees?

  1. Table 4 shows increased MIA Lift following the introduction of noise refresh. Can the authors clarify this potential fidelity–privacy trade-off and provide a direct statistical comparison?

  2. Clarify whether computational comparisons use equivalent hardware and training budgets, particularly when comparing CPU-based XGB-DDPM with neural diffusion models.

4 comments

Thanks for your insightful review of our work. Below we will address the mentioned Critical Weakness, and the questions.

Critical Weaknesses and Question 1

We agree with these comments. Therefore, we have included a new paragraph in the Methods section entitled “Optimization across noise-refreshed boosting rounds”. There, we show the theoretical justification for how and when the noise-refreshed boosting trajectory decreases the expected diffusion objective (and when trees in individual boosting rounds provide a descent direction).

In short, the justification follows from 3 assumptions:

  • Smoothness of the diffusion objective. This can be simply shown through smoothness of the numerical and categorical loss (i.e., bounded second-order derivatives).
  • Expected gradient alignment in trees. This follows from typical weak-learning conditions in boosting.
  • Bounded second moment of XGBoost trees. This follows directly from XGBoost's second-order approximation and L2 leaf regularization.

We show that these 3 conditions imply that the noise-refreshed boosting procedure decreases the expected diffusion objective at each round; in particular for small enough learning rate. This directly ties in to the observation we already made regarding the learning rate, i.e., that learning rate should be set small enough so that no individual noise realization disproportionately affects the full model.

Note that, to account for the increased space this theory takes up, we moved the diffusion specifications (i.e., the math for Gaussian and multinomial diffusion) to the Appendix; this was mainly reiterating past work anyway.

Question 2

Since the noise-refresh procedure improves the estimation of the diffusion objective, it improves the accuracy of the learned generative distribution (i.e., increases fidelity). Since we model the training distribution more accurately, the generated points are prone to be more similar to the training data than before. If we accurately reproduce identifiable portions of the training data, this can yield increased privacy risk. Note that this is purely a result of more accurately modelling the distribution, but in particular, of portions of the distribution which are typical of the training data (when compared to the held-out test data).

This is the typical fidelity-privacy tradeoff which is well-known for tabular generative models. You can also observe it in the other baseline models in Table 2: high-fidelity baselines such as TabDDPM and CDTD score relatively poorly in terms of MIA Lift. Finally, we’d like to point out that our method provides a better fidelity-privacy tradeoff than other competitive baselines such as TabDDPM and CDTD in our experimental datasets (see Table 2 and Figure 4).

Question 3

Hardware was specified in Appendix B.4. Naturally, we cannot use equivalent hardware for CPU-based models, such as XGB-DDPM, and the neural diffusion models; we use CPUs for the CPU-based models and GPUs for neural diffusion models (actually all methods based on deep neural networks). See Appendix B.4 for the specific hardware used. The hardware was selected to provide a realistic expectation of how these methods may perform in somewhat restricted hardware settings; in particular, XGB-DDPM could be majorly scaled up by using larger CPU clusters.

Thanks again for your thorough review.

Dear reviewer uP9a,

we’d be glad to hear any remaining questions you may have regarding our work. We have put substantial effort into addressing your concerns by updating the Methods section of the paper. If you feel we’ve substantially addressed your concerns, we kindly request to consider reflecting this in your score.

Thanks again for your review of our work.

Thanks for your reply, but I incline to keep my grade.

Dear reviewer uP9a,

Could you please elaborate?

We have put substantial effort into addressing your only mentioned weakness of the paper by incorporating new proofs. It seems odd that this would not influence your rating of this work.

We are glad to hear any remaining concerns or questions regarding this matter.

Thanks again for your review and consideration of our work.

Reviewer_UGJoOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper suggests the usage of XG Boost as the architecture of choice for diffusion models extending to tabular data.

Critical strengths
  1. The paper suggests a clean extension of existing tree based methods for tabular diffusion models to XG Boost.
  2. The results on the benchmarks appear promising.
Critical weaknesses
  1. The method mainly reuses existing formulation and does not significantly utilize any advantages provided by the boosting technique itself or the learned nature of the decision tree.

  2. Related to 1 while it is reasonable to assume it performs better in generation a significant benefit of a tree based model over a NN in this setup is the power it offers for auxiliary tasks and downstream usage such evaluations are missing or not explored here.

Questions for authors

I have no other significant questions for the authors.

6 comments

Thanks for your thorough review of our work. We will address the two Critical Weaknesses you mentioned point-by-point.

Point 1

We agree that we were mainly reusing existing formulation; mainly considering the diffusion specifications (i.e., the math for Gaussian and multinomial diffusion). Considering comments from other reviewers, we’ve now included more theoretical justification regarding how the noise-refreshed boosting trajectory decreases the expected diffusion objectives (see the paragraph "Optimization across noise-refreshed boosting rounds" in the revised PDF). We moved the diffusion specifications to the Appendix.

However, we’d like to note that the method directly and mainly utilizes advantages provided by the boosting technique. Since, marginalizing over noise realizations by refreshing the seed across boosting rounds, is only possible when doing boosting. Therefore, we’d say we’re directly taking advantage of the nature of the boosting technique.

Regarding the “learned nature of the decision tree”, we’re not sure what is meant. The only relevant considerations for us are that these tree-based methods:

  • can be efficiently trained on CPU
  • effectively and natively handle mixed-type data (mix of numerical and categorical features)
  • provide low-bias estimators for the denoising process (this is particularly true for boosted trees)

Since we utilize the above, we’d say that we do leverage the advantages of decision trees directly in our method. However, we feel it is possible we may have misinterpreted your comment. If so, we’re glad to consider a clarification.

Point 2

We understand the interest in using our method for auxiliary tasks. For example, you might consider imputation or probabilistic prediction. However, we do not consider our main contribution (the noise-refreshed boosting procedure) to be particularly suited for that. Noise-refreshing provides a much more runtime and memory efficient method than dataset duplication for marginalizing over diffusion noise realizations. However, dataset duplication is more accurate. For probabilistic prediction, in which there is only a single target, these memory and runtime issues are not relevant in practice. These issues arise when training many feature-specific models in parallel. Regarding imputation, the ForestDiffusion paper already shows that simpler classical methods such as MissForest are more accurate.

Thanks again for your review of our work.

Reviewer_UGJoOpenReview ↗

Response to the author(s)' comment.

Thank you for clarifying that the proposed novelty lies in terms of the estimate of the expectation (the resampling strategy which is more efficient memorywise than pure copying ! However, I noticed that currently there are seperate models for each timestep, label and feature, a setup identical to ForestDiffusion. My question was that given that tabular datasets through their features form a joint distribution it is reasonable to expect that the learned rules form a joint representation as well hence could we leverage this for a more streamlined architecture ? To give an example consider the following columns,

Categorical feature (n)Continuous feature (x)Model prediction
A-2.00
A-1.00
A0.51
A1.51
B-2.00
B-1.00
B0.51
B1.51
C-2.02
C-1.02
C0.52
C1.52
RulePredicted classPrecision
n = A AND x < 001.00
n = A AND x > 011.00
n = B AND x < 001.00
n = B AND x > 011.00
n = C21.00

Models like XGboost are quite amenable to such probes as the line of works from [1] show further recent works such as [2] show that even trained discriminative tree based models can define better denoising pathways.

[1] M. T. Ribeiro, S. Singh, and C. Guestrin. “Anchors: High-Precision Model-Agnostic Explanations.” In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2018.

[2] S. N. Ramachandran and S. Sra. “Trees to Flows and Back: Unifying Decision Trees and Diffusion Models.” In Proceedings of the International Conference on Machine Learning (ICML), 2026.

Thank you for the clarification.

To actually allow a more streamlined architecture we see 2 main options:

  • learn a single joint model
  • within each feature-specific model, reuse information from other learned trees

Learning a single joint model

The simplest option to reuse information across features, timesteps, and labels, would be to simply condition on that information as input features. For example, add the timestep, label, and feature index as input, and potentially mask (with a NaN token) the output feature in the input. This could train a single model to learn general reusable splits across different features and/or labels and timesteps. However, as we already show in Section 4.2.2 for timesteps and labels, this is not a good idea for XGBoost. XGBoost's efficiency is roughly linear in the #rows (when using "hist"). Therefore, extending the data and training less models does not yield a speedup. Also, when training separate models, the modelling task is easier, such that we can use less boosting rounds (through early stopping) and yield lower-bias predictions (and thus higher generative quality).

More directly, we could use XGBoost multi-output trees. This learns a single tree-structure by aggregating the loss over all output features. This way it does reuse the information across the features. Cresswell et al. (see paper References) already tried this. However, the results are not convincing: training time is higher since XGBoost has to search over a higher-dimensional leaf space, and fidelity is worse on most metrics. We also think this is a not a good fit considering our noise-refreshing strategy, as this is another source of slowing down the model training, since we need to reassign histogram bins at each round. However, multi-output trees can yield faster inference, which might be interesting.

Feature-specific models: reuse information from other trees

It is not directly obvious how learned rules/splits could be reused across features. In particular, it is difficult to assess exactly which rules those would be, without already training the feature-specific denoiser. Also note here that our goal is to learn a diffusion objective. Models like ARF are already able to learn a general rule-set which partitions the data space. But, this is a completely different architecture and goal.

On this same note, we think there's some interesting directions considering your suggestion to improve the timestep-specific denoisers. For example, reusing a warm-started booster from earlier timesteps, and then only applying a few corrective boosting rounds to denoise the new timestep. There's also some risks here, as boosters from earlier timesteps may point in the wrong direction and new rounds have limited corrective capacity due to a low learning rate; this would require significantly more investigation.

Finally, please note that the main contribution of our work is the resampling method to optimize stochastic denoising objectives through boosted models. XGB-DDPM is just a relatively simple instantiation of a diffusion model using this; it uses quite basic diffusion kernels, and as you mention, separate denoisers for each feature, timestep, and label. See also the final sentence of the Discussion section. We leave further improvements of the diffusion model itself, such as more efficiently reusing joint representations, for future work.

Ps.: there might be other interesting directions. If you have any interesting ideas considering your suggestion, we'd be very glad to hear!

Reviewer_f6GLOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper alternates ForestDiffusion’s dataset-duplication trick (K=50–100 noisy copies of every row) with a per-boosting-round noise refresh. In details, at each round, fresh diffusion noise is drawn, the noised inputs and their histogram bin assignments are recomputed, and the boosting update is taken on that single realization. This is implemented in an XGBoost fork. On top of it they build XGB-DDPM, a DDPM with Gaussian diffusion for numerical features (XGBoost regressors) and multinomial diffusion for categorical (XGBoost classifiers), feature-wise and timestep-wise models, v-prediction, cosine schedule, DDIM sampler. Evaluated on 10 TabArena datasets plus poker-hand (1.025M rows) against 9 baselines on 10 metrics.

Critical strengths
  1. The idea is simple, correct in spirit, and the systems payoff is real.
  2. careful empirical protocol
  3. poker-hand is the right stress test
  4. The incremental ablation (Table 4) is rich
Critical weaknesses
  1. The theory is thinner than the presentation implies. Eq. (5) establishes unbiasedness conditional on F_{b−1}, which is a one-step property; F_{b−1} is itself a function of all past noise draws, this lacks the information about the boosting trajectory optimizes the true expected diffusion objective.

2.The central comparison may be missing. The paper’s claim is about refresh versus duplication, but Table 4’s “Base” uses a single noise realization per observation/timestep (K=1), so the ablation measures refresh vs. no marginalization — not refresh vs. the K=50 duplication it is proposed to replace.

Questions for authors

See weaknesses.

2 comments

Thanks for your insightful review of our work. We will address the two Critical Weaknesses you mentioned point-by-point.

Point 1

We agree with your comment. Therefore, we have included a new paragraph in the Methods section entitled “Optimization across noise-refreshed boosting rounds”. There, we show the theoretical justification for how and when the noise-refreshed boosting trajectory decreases the expected diffusion objective. This addresses this information.

In short, the justification follows from 3 assumptions:

  • Smoothness of the diffusion objective. This can be simply shown through smoothness of the numerical and categorical loss (i.e., bounded second-order derivatives).
  • Expected gradient alignment in trees. This follows from typical weak-learning conditions in boosting.
  • Bounded second moment of XGBoost trees. This follows directly from XGBoost's second-order approximation and L2 leaf regularization.

We show that these 3 conditions imply that the noise-refreshed boosting procedure decreases the expected diffusion objective at each round; in particular for small enough learning rate. This directly ties in to the observation we already made regarding the learning rate, i.e., that learning rate should be set small enough so that no individual noise realization disproportionately affects the full model.

Note that, to account for the increased space this theory takes up, we moved the diffusion specifications (i.e., the math for Gaussian and multinomial diffusion) to the Appendix; this was mainly reiterating past work anyway.

Point 2

We agree with your comment. We have now included the noise-refresh vs. K=50 comparison in Table 4.

Thanks again for your review. We feel that the additions we've made to the Methods section and Table 4, based on your review, have strenghtened our work.

Dear reviewer f6GL,

we’d be glad to hear any remaining questions you may have regarding our work. We have put substantial effort into addressing your concerns by updating the Methods section of the paper. If you feel we’ve substantially addressed your concerns, we kindly request to consider reflecting this in your score.

Thanks again for your review of our work.

Comments on the paper

1 comment
Area_Chair_CPn4OpenReview ↗

Please acknowledge the rebuttal

I would like to thank the authors for their rebuttal. Dear Reviewers, as we are approaching the end of the discussion period, I would appreciate it if you could take a moment to acknowledge the rebuttal (if you haven't already done so) and see if it addresses your concerns or no. Thank you for your time and service.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.