A Systematic Study of SG-MCMC Samplers for In-Context Energy-Based Tabular Generation with TabPFN

Energy-based models built on pre-trained tabular in-context learners sample synthetic data from a classifier-derived energy landscape, relying on Stochastic Gradient Langevin Dynamics (SGLD) without comparing it against alternative Stochastic Gradient-Markov Chain Monte Carlo schemes. Using TabPFNv2 as an energy oracle, we investigated various sampler choices, revealing that substituting SGLD with a momentum-based sampler yields substantial and consistent improvements in model fidelity. Even without injected noise, in-context gradients exhibit fluctuations driven by the underlying geometry of the energy surface. The momentum can filter it by making short-run generation a structural necessity. For coverage, we show a clean double dissociation: replica exchange alone reaches under-represented modes under biased initialization. These findings suggest a primary design heuristic for in-context energy-based tabular generation: leverage momentum to enhance fidelity and employ replica exchange to maintain coverage.