acceptodds
Desk rejected at ICLR 2027

A Systematic Study of SG-MCMC Samplers for In-Context Energy-Based Tabular Generation with TabPFN

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Energy-based models built on pre-trained tabular in-context learners sample synthetic data from a classifier-derived energy landscape, relying on Stochastic Gradient Langevin Dynamics (SGLD) without comparing it against alternative Stochastic Gradient-Markov Chain Monte Carlo schemes. Using TabPFNv2 as an energy oracle, we investigated various sampler choices, revealing that substituting SGLD with a momentum-based sampler yields substantial and consistent improvements in model fidelity. Even without injected noise, in-context gradients exhibit fluctuations driven by the underlying geometry of the energy surface. The momentum can filter it by making short-run generation a structural necessity. For coverage, we show a clean double dissociation: replica exchange alone reaches under-represented modes under biased initialization. These findings suggest a primary design heuristic for in-context energy-based tabular generation: leverage momentum to enhance fidelity and employ replica exchange to maintain coverage.

Primary area:
generative models
Keywords:
Tabular Data Generation, Tabular Foundation Models, In-Context Learning, Energy-Based Models, Sampling Methods, Prior-Data Networks

Related papers

Discussion (0)

Sign in to comment.