acceptodds
Under review as a conference paper at ICLR 2027

Complexity-Adaptive Stein Transport for Parallel Tempering

Abstract

Sampling from complex distributions with parallel tempering can be limited by poor exchange between neighboring temperatures. Transport can improve these exchanges, but a fixed expressive map may be unnecessary when cross-temperature changes have simple structure. We introduce CAT-PT, which organizes transports into a hierarchy of structured velocity classes. CAT-PT fits a candidate in each class by minimizing a Stein residual that measures the annealing variation left unexplained by the corresponding motion. CAT-PT then tests increasingly expressive transports and stops as soon as one achieves a prescribed exchange-acceptance target. We show that the accumulated Stein residual directly controls finite-gap exchange rejection. In structured regimes, CAT-PT achieves round-trip performance governed by the residual variation after transport rather than by the ambient dimension. Under additional conditions, this residual scaling is order-optimal within the same transport class. This adaptivity can avoid the cost of a fixed high-capacity accelerator when a simple transport is sufficient. Numerical experiments validate the predicted communication and computational performance.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Sampling, Probability transport, Stein operators

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_txJXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CAT-PT to improve exchanges in parallel tempering. The idea is to use the simplest transport that gives a high enough acceptance rate. For each pair of neighboring distributions, the method fits increasingly flexible velocity fields by minimizing a Stein residual, then checks their acceptance rates on held-out samples. The chosen maps are fixed during sampling, with a Metropolis correction.

The theory bounds exchange rejection using the residual and gives conditions under which round-trip performance depends on the remaining variation rather than the full dimension. Synthetic experiments show that simple structured maps can give better sampling efficiency per second than the neural transport baseline, even when they achieve fewer round trips per sweep.

My reproducibility check covered a small-scale reconstruction of the Gaussian experiment. It recovered the qualitative trend, but not the reported CAT-PT acceptance rates. The main timing comparisons were not tested.

Critical strengths
  1. Theorem 1 links the Stein residual used in fitting to rejection across a finite temperature gap. This gives the fitting objective a clear justification.

  2. I also like the idea of trying cheap transports first and adding complexity only when held-out acceptance is too low. This could avoid unnecessary computation on edges where a simple map is enough. The idea is useful, though the claimed speed gains need stronger evidence.

Critical weaknesses
  1. The speed comparison is hard to assess. Appendix J leaves out key NF-APT training settings and the hardware and timing setup. Reported runtimes also exclude calibration and fitting. Since efficiency is a main claim, these gaps limit how much weight I can put on the reported gains.

  2. The experiments give limited evidence for adaptive selection itself. Figure 2 compares raw exchange with just one structured candidate, and the Max-Structured comparison covers only the quartic translation case. A fixed cheap Stein transport on the main benchmarks would be a useful comparison. Without it, it is hard to tell how much adaptation adds beyond simply using a cheaper map.

Questions for authors
  1. Could you provide the full NF-APT configuration, hardware, and timing procedure? What are the calibration and fitting times for both methods, and do the speed gains remain when these costs are included?

  2. How does CAT-PT compare with fixed-level Stein transports using the same fitting procedure on the main benchmarks? Please also report which levels CAT-PT selects across edges. This would help show how much adaptive selection adds beyond using a cheap map.

1 comment

Thank you for recognizing our algorithm design and theoretical results. We have conducted additional experiments to address your concerns.

W1 & Q1: Runtime comparison and experimental protocol.

Our original large-scale experiments were conducted on rented servers. For these supplementary experiments, all methods were run serially on the same local machine (AMD Ryzen 5 7640H, 6 cores/12 threads, 32 GB RAM), using the CPU backend and PyTorch with 6 threads. Each run used 16 PT ensembles, 1,000 warm-up sweeps, and 5,000 production sweeps. For NF-APT, we used a 20-layer RealNVP with hidden width 128 and two hidden layers per subnet. Training used batch size 256, Adam (learning rate 0.001), with 500–5,000 steps per round and validation-based early stopping. Complete runtime is measured from process startup to termination. The results are:

TaskMethodTime (s)RR/sGM ESS/s
GaussianRaw DEO-PT17.26±0.130.00.03522.4±409.3
CAT-PT21.45±0.182495.01861.4±15.63330.9±143.2
NF-APT505.36±4.511621.4±36.351.3±1.5139.3±6.4
GMMRaw DEO-PT72.98±0.4531.0±2.86.8±0.614.7±0.3
CAT-PT93.02±0.48507.8±2.687.4±0.855.3±1.6
NF-APT864.50±2.40588.7±2.910.9±0.16.4±0.2
Many-WellRaw DEO-PT64.84±1.27126.4±2.031.2±0.3125.5±3.1
CAT-PT80.07±1.28360.2±2.072.0±1.1191.3±9.1
NF-APT808.93±5.51265.3±1.45.2±0.115.7±0.7

We also evaluate sampling accuracy against analytical reference values, using two common metrics: mean RMSE and second-moment RMSE.

TaskAccuracy metricRaw DEO-PTCAT-PTNF-APT
GaussianMean RMSE (×10⁻³).447±.065.380±.044.390±.029
Second-moment RMSE (×10⁻³).884±.137.772±.069.793±.063
GMMMean RMSE.0410±.0073.0160±.0011.0153±.0035
Second-moment RMSE (×10⁻³)6.68±.802.39±.312.31±.45
Many-WellMean RMSE (×10⁻³)51.79±8.5032.89±2.4140.65±6.55
Second-moment RMSE (×10⁻³)4.26±.374.52±1.105.21±.64

The reported mean and second-moment observables follow each benchmark's construction: coordinate averages for Gaussian, the full mean vector and mean-square statistic for GMM, and averages over the four active coordinates for Many-Well. CAT-PT achieves moment estimation accuracy comparable to NF-APT, while substantially reducing runtime.

W2 & Q2: The benefits of adaptive selection

We also added fixed-level ablations: L1 is translation plus global scaling; L2 adds coordinate-wise scaling.

TaskMethodTime (s)RR/sGM ESS/s
GaussianCAT-PT21.45±0.182495.0±0.01861.4±15.63330.9±143.2
Fixed L121.32±0.232495.0±0.01872.8±20.03352.4±177.1
GMMCAT-PT93.02±0.48507.8±2.687.4±0.855.3±1.6
Fixed L188.20±3.32113.1±1.220.5±0.913.8±0.7
Fixed L286.91±0.18508.1±3.993.5±0.960.0±0.7
Many-WellCAT-PT80.07±1.28360.2±2.072.0±1.1191.3±9.1
Fixed L177.62±0.92357.9±3.273.8±1.3203.6±10.7

CAT-PT selected L1 on all 15 Gaussian edges; L2 on 23/27 and L3 on 4/27 GMM edges; and L1/L2/L3 on 15/27, 8/27, and 4/27 Many-Well edges, respectively. These results illustrate different regimes. For Gaussian, L1 is sufficient, and CAT-PT behaves almost identically to Fixed L1. For GMM, Fixed L1 performs substantially worse, whereas CAT-PT selects richer candidates and approaches Fixed L2. For Many-Well, CAT-PT uses different complexity levels across edges. Adaptive selection doesn't always outperform the best fixed level chosen with hindsight. Its benefit is that the appropriate complexity need not be known in advance and can vary across neighboring temperature pairs.

We will incorporate these results into the paper. We hope these additional results address your concerns and appreciate your consideration of this new evidence.

Reviewer_bSLTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a procedure to sequentially learn velocity fields for transport between different temperatures in parallel tempering. The loss minimized is the integrated squared continuity equation loss. The authors try to adapt the complexity of the model to the complexity of the move by considering a nested sequence of models. The method is validated on Shifted Gaussian, Structured Many-Well, and 40-node Gaussian mixture.

Critical strengths
  1. The loss is nice: it allows optimizing for the continuity equation without knowing the normalizing constants, in contrast to, e.g. Mate and Fleuret (2023). But it requires having the exact samples. Here the paper uses current PT iteration as a proxy, which is a natural idea. (It does raise the question of whether the algorithm can escape the bad initial iterations though.)

  2. The idea that simple transformations should be sought first if possible is valid. (I wonder though, given the current structure of the paper, whether Lasso regression or similar ideas could be used to select mm automatically.)

Critical weaknesses
  1. The very first question anyone would ask is what are the classes V0,…,VMV_0, \ldots, V_M and their dimensions pmp_m. Very terse description is given in the main text, and details are buried somewhere in the 45-page appendix. This is not the right way to present such a capital aspect of the algorithm.

  2. Without clear understanding for my first point, I am unable to assert how expressive the classes VmV_m are. Ideally, one wants that as mm grows, the complexity of VmV_m increases in a significant enough away to be able to capture complex transport geometries. That's what should be explained, in particular when the title contains "Complexity-aware".

Questions for authors

The questions correspond to the identified weaknesses. More specifically:

  1. Give me a clear description of the classes VmV_m. Which classes did you use in the experiments? If the paper is accepted, please put these in the main text.

  2. Given two consecutive distributions πk\pi_k and πk+1\pi_{k+1}, do there exist sufficiently large MM such that the loss tends to zero?

I have other questions/suggestions that are mentioned in the "Critical Strengths" section, but they are less consequent for my decision.

4 comments
AuthorsOpenReview ↗

Author Rebuttal (1/2)

We thank you for recognizing the merits of our normalizer-free Stein objective and the idea of selecting the simplest sufficient transport. We appreciate your constructive feedback and address your concerns below.

W1 & Q1: Specification of the velocity hierarchy.

We agree that the hierarchy deserves a more concrete description in the main text.

The hierarchy ${V_m}$ is a user-specified input to CAT-PT, representing different levels of complexity in modeling distributional changes along the annealing path. It can be constructed by progressively introducing elementary motions, such as translations, scaling, blockwise interactions, or nonlinear deformations. Different problems may call for different hierarchies, depending on the available structural knowledge and the desired balance between transport accuracy and computational cost. Given a hierarchy, CAT-PT estimates the velocity coefficients from calibration data and adaptively selects the first candidate meeting the prescribed acceptance criterion.

In our experiments, we use the following nested affine hierarchy:

  • $V_0={0}$, with $p_0=0$ (raw exchange).
  • $V_1={v(x)=b+\alpha x}$, with $p_1=d+1$.
  • $V_2={v(x)=b+\operatorname{diag}(a)x}$, with $p_2=2d$.
  • $V_3={v(x)=b+Ax:\ A\text{ is block-diagonal}}$, with $p_3=2d+\sum_B |B|(|B|-1)$.

Here, $b,a\in\mathbb R^d$, $\alpha\in\mathbb R$, and the block partition is specified before fitting. For two-dimensional blocks, the nontrivial levels have $d+1$, $2d$, and $3d$ parameters, respectively.

These levels progressively capture translations and global scaling, coordinate-wise scaling, and within-block interactions such as shearing and rotation. All admit closed-form flows, inverses, and log-Jacobians, enabling efficient transported exchanges. Figure 2 uses a separate specialized candidate with global scaling and coordinate-wise corrections on eight supplied coordinates.

Following your suggestion, we will clarify the flexibility of the hierarchy and move its experimental specifications, parameter counts, and geometric interpretations into the main text.

W2 & Q2: Expressive capacity of the velocity hierarchy.

A sufficiently expressive hierarchy can indeed make the optimal population Stein residual arbitrarily small under suitable regularity conditions. The finite affine classes used in our experiments prioritize computational efficiency and are not universally expressive. However, this is not a restriction of the general CAT-PT framework. We can establish an approximation result by extending the hierarchy with Gaussian-kernel velocity fields.

Consider the Gaussian RBF kernel

$$ k_\sigma(x,z)=\exp\left(-\frac{|x-z|^2}{2\sigma^2}\right). $$

We can progressively enrich the structured hierarchy with basis fields $k_\sigma(\cdot,z_\ell)e_j$, where the centers ${z_\ell}$ are dense in $\mathbb R^d$ and $e_j$ denotes a coordinate direction. This yields nested finite-dimensional classes with increasing expressive capacity.

Under suitable regularity conditions, including a strictly positive $C^1$ density $\pi_t$, a square-integrable spatial score, and a centered path score $s_t\in L_0^2(\pi_t)$, the Stein images of these classes are dense in $L_0^2(\pi_t)$. Consequently,

$$ E_m(t)=\inf_{v\in V_m}|s_t+\mathcal S_t v|_{L^2(\pi_t)}^2\longrightarrow 0 \quad\text{as }m\to\infty. $$

The intuition is that Gaussian kernels provide sufficiently rich localized velocity directions to approximate the required density changes. More precisely, any centered function orthogonal to all the corresponding Stein features must have a vanishing spatial gradient, by the injectivity of Gaussian convolution, and hence must be zero. Thus, the hierarchy can approximate any centered square-integrable path score arbitrarily well, without requiring the target distribution to be Gaussian or unimodal.

Moreover, when the path score has an integrable $L^2$ norm along an edge, the accumulated optimal residual also converges to zero:

$$ L_{k,m}=\int_{t_k}^{t_{k+1}}\sqrt{E_m(t)},dt\longrightarrow 0. $$

We focus on the main result here; we will include the formal statement, detailed proof, and required conditions in the revised appendix.

We would like to clarify that CAT-PT does not aim to drive the residual to zero. Although richer classes can improve population approximation, they also introduce additional fitting and computational costs, while finite-sample and temporal approximation errors remain relevant. Our objective is instead to achieve sufficient exchange performance at minimal necessary complexity.

AuthorsOpenReview ↗

Author Rebuttal (2/2)

We also thank you for the two additional suggestions. We would like to clarify both points.

Initial PT calibration. CAT-PT does not require independent or exact samples from intermediate distributions. It first runs ordinary PT with a burn-in period, then collects separate fitting and held-out validation blocks with a discarded gap between them. Our analysis explicitly accounts for three sources of uncertainty: incomplete equilibration, dependence between the two blocks, and finite effective sample size. Under the corresponding mixing and concentration conditions, the held-out acceptance estimates are uniformly accurate across all candidate transports with high probability, ensuring reliable complexity selection. We acknowledge that severely inadequate initial mixing can compromise these guarantees; CAT-PT does not assume this difficulty disappears automatically. Importantly, once the transports are fitted and frozen, the Metropolis–Hastings correction preserves the exact target distribution regardless of calibration accuracy. Thus, calibration quality affects efficiency and selection guarantees, but not the invariance of the resulting sampler.

Lasso-based selection. We appreciate this suggestion and agree that sparse regression could be useful for selecting velocity directions. However, coefficient sparsity and transport complexity are not necessarily equivalent: the latter also depends on the computational cost of evaluating the flow, its inverse, and its Jacobian. Moreover, Lasso selects coefficients through a penalized fitting objective, whereas CAT-PT directly evaluates the exchange acceptance of the resulting finite transport and stops once the desired performance is achieved. The two approaches are therefore complementary: sparse regression could be used within candidate classes, while CAT-PT determines whether the resulting transport is sufficient. We will clarify this distinction in the revision.

We hope these additional results address your concerns. We sincerely appreciate your constructive feedback and would be grateful if you would consider revising your assessment in light of these new results.

Reviewer_bSLTOpenReview ↗

Rebuttal acknowledgement

I'd like to thank the authors for the detailed rebuttals. The authors have now confirmed that the class V0,…,V3V_0, \ldots, V_3 here is particularly simple and comprises only linear transformations. Therefore, like other reviewers, I understandably have concerns about the scalability of this method for more complex problems where non-linear transports are needed.

I need a more careful analysis of when adjacent distributions on parallel tempering can be transported to/from each other via only linear maps. One relevant work that I've come across recently: https://arxiv.org/abs/2505.10466 Additionally, letting temperatures be close to each other may make linear maps sufficient. Again this needs to be evidenced both theoretically and practically.

I do think the paper has merits, but at the current stage seems a bit premature for publication. A resubmission taking into accounts suggestions from the reviewers would stand a better chance at acceptance.

Thank you for raising this important point.

CAT-PT is a general framework that supports both affine and nonlinear velocity fields; the affine hierarchy in our experiments is a computational choice, not a restriction of the algorithm. In the general formulation, basis fields can be nonlinear functions of the state, while the velocity remains linear in its coefficients. Thus, the same quadratic Stein regression and held-out selection procedure applies, provided the induced flows, inverses, and Jacobians satisfy the regularity and computational conditions in Assumption 2.

For example, consider a one-dimensional nonlinear velocity field vθ(x)=θtanh⁡(x)v_\theta(x)=\theta\tanh(x). Its induced transport over an interval of length hh has the closed-form expression

Th(x)=asinh⁡(eθhsinh⁡(x)).T_h(x)=\operatorname{asinh}(e^{\theta h}\sinh(x)).

It also has an explicit inverse and Jacobian, and can therefore be included as a nonlinear candidate in CAT-PT without changing the fitting or selection procedure.

When are affine transports sufficient? Appendix E.1 (Proposition 6) provides a general sufficient condition: if a time-dependent coordinate transformation makes a high-dimensional conditional distribution invariant along the annealing path, its induced velocity removes the corresponding distributional change. When this transformation has global, diagonal, or block-affine structure, it belongs to our experimental hierarchy. More generally, equations (87)–(88) show that approximate representation is also sufficient, with the approximation error explicitly controlled.

Smaller temperature gaps can help, but involve a trade-off. By Theorem 1, the accumulated population residual on an edge of width hkh_k satisfies

Lk,m≤hksup⁡t∈[tk,tk+1]Em(t).L_{k,m}\le h_k\sup_{t\in[t_k,t_{k+1}]}\sqrt{E_m(t)}.

Thus, smaller gaps reduce the finite-edge mismatch, even when the affine transport is not exact. However, they require more replicas. Theorem 2 characterizes this trade-off: under its conditions, the required ladder size scales as O(1+rd)O(1+\sqrt{r_d}), where rdr_d measures the residual variation left unexplained by the selected class.

Our Gaussian, multimodal GMM, Many-Well, and real-data Bayesian experiments demonstrate that the affine hierarchy is sufficient for the problems considered, with CAT-PT adapting complexity across neighboring temperatures.

Reviewer_nMZmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CAT-PT, a transport-accelerated parallel tempering method that selects transport complexity separately for each neighboring temperature pair. Candidate velocity fields are fitted by Stein regression within a nested hierarchy of structured classes. Held-out exchange acceptance then determines the simplest sufficient candidate, which is frozen for production sampling with a Metropolis correction.

The theoretical analysis bounds finite-gap exchange rejection by the accumulated Stein regression residuals and derives conditional round-trip guarantees. Synthetic experiments compare CAT-PT with ordinary PT, neural transport methods, and several structured ablations.

Critical strengths

Novelty and relevance of the algorithm: Selecting the simplest structured transport that achieves sufficient exchange acceptance is a thoughtful alternative to deploying a fixed expressive neural map everywhere. Combining Stein regression, held-out selection, and Metropolis-corrected exchanges gives the method a clear practical motivation. I find this combination promising and consider it the paper’s principal contribution.

Coherent theoretical framework: Theorem 1 connects the fitting objective to the exchange rejection. Proposition 1 transfers reliable acceptance certification into a communication guarantee, and Theorem 2 identifies conditions under which residual variation governs the required ladder size. These results are conditional and some steps are straightforward, but together they explain why structured transport can help. The distinction between approximation error within the chosen class and additional fitting error is particularly useful.

Critical weaknesses

The main theoretical guarantees leave the central approximation and learning difficulties unresolved: Theorem 1 establishes that a small accumulated Stein residual implies high exchange acceptance, but does not establish that the proposed hierarchy or fitted candidates achieve such a residual. Its proof is relatively direct once the residual is identified as the score in moving coordinates.

Proposition 1 is primarily a supporting guarantee: on the uniform validation event, the acceptance threshold follows directly from the selection rule, and the round-trip bound follows by substituting this threshold into the existing communication formula. It is useful for checking the procedure, but has limited standalone theoretical significance.

Theorem 2 combines these ingredients under assumptions that a suitable class has controlled residual variation and that fitting error satisfies a dimension-independent bound. These premises encode much of the desired behavior. The theorem does not establish that an adequate class can be identified or fitted at an affordable cost on general problems.

The evaluation focuses too narrowly on communication and speed. Round trips and ESS per second are relevant, but they do not fully establish sampling accuracy. A sampler can move between temperatures frequently while still estimating mode probabilities poorly. The paper reports that all 40 mixture components are visited, but visiting every mode does not show that each is sampled with the correct frequency. Since these synthetic targets allow reference calculations, the authors should also report errors in estimated moments and mode probabilities.

There is also no Bayesian inference experiment. The paper presents Bayesian likelihood tempering as an application, yet all experiments use synthetic distributions. A Bayesian inference task would test whether the method remains useful when the required transport is less obvious and calibration samples must be obtained from a nontrivial posterior. Posterior estimation errors against a reliable reference, together with total runtime including calibration and fitting, would provide stronger evidence of practical value.

Questions for authors

Theory and calibration: Can you identify a nontrivial problem family for which the residual bound, fitting requirements, and cost of obtaining calibration samples can all be controlled explicitly as dimension grows?

3 comments
AuthorsOpenReview ↗

Author Rebuttal (1/3)

Thank you for your thoughtful feedback and for recognizing the main idea of our work. We address your concerns below.

Q1: Can these conditions be achieved simultaneously in a nontrivial problem family?

Yes. We construct a problem family in which the approximation residual, fitting sample requirement, and calibration cost can all be explicitly controlled.

Let $q=d-r$, where $r$ is fixed, and split the state into $w\in\mathbb R^q$ and $z\in\mathbb R^r$. Consider the annealing path

$$ \pi_t^{(d)}(w,z) =\mathcal N(w;\gamma t\mathbf 1_q,I_q)\rho_t(z), $$

where $\rho_t(z)\propto\phi_r(z)e^{-tU(z)}$, $\phi_r$ is standard Gaussian, $\gamma\ne0$, and $U$ is smooth, bounded, and nonconstant. The high-dimensional component $w$ undergoes a shared translation, while the fixed-dimensional component $z$ changes nonlinearly and can even be multimodal (e.g., a bimodal density for $r=1$ with $U(z)=Be^{-z^2/2}$ and $B>1$).

Consider a velocity that translates all coordinates in $w$ using one shared coefficient, $v_\theta(w,z)=(\theta\mathbf 1_q,0)$. Its optimal coefficient is $\theta^\star=\gamma$. The transport removes the high-dimensional motion exactly, but leaves a nonzero residual from the nonlinear component:

$$ E_1(t)=\operatorname{Var}_{\rho_t}(U(Z)) \le \frac{B^2}{4}, $$

where $B=\operatorname{osc}(U):=\sup_z U(z)-\inf_z U(z)$. Thus, the approximation error remains bounded independently of $d$, without requiring an exact transport of the full distribution.

This structure can also be learned from finite data. Although the nonlinear component introduces genuine regression noise, the estimation error of the shared coefficient decreases as $q^{-1/2}$, compensating for the $\sqrt q$ factor in the regression residual. Consequently, $n=O(\log(J/\delta_{\rm fit}))$ stationary fitting observations per temperature node suffice for a dimension-independent fitting guarantee, where $J=K+1$ is the number of temperature nodes. The population-optimal coefficient is constant along the path, so endpoint averaging introduces no additional interpolation error.

We further control the cost of obtaining calibration observations. We use a preliminary PT kernel that refreshes the Gaussian component exactly and updates the nonlinear component using independence MH with proposal $\phi_r$. The boundedness of $U$ gives this local kernel an explicit minorization condition. With $O(e^B\log J)$ local updates per node between raw-PT exchange sweeps, the resulting ensemble pilot has a geometric mixing bound independent of $d$. This allows us to obtain representative fitting and validation observations at a controlled cost, rather than assuming independent stationary samples are available.

Combining these results with Theorem 1 and held-out selection, the total pre-production cost, including calibration, fitting, and validation, is $C_{\rm pre}=O(d)$ for fixed $r$, $B$, acceptance target, and confidence levels. A ladder with a dimension-independent number of edges suffices. Under DEO and ELE, the resulting round-trip rate is $\Omega(1)$, compared with $\Theta(d^{-1/2})$ for raw PT on this family.

W1 & W3: Sufficient conditions for approximation and learning.

We agree that the conditions underlying Theorems 1 and 2 deserve a clearer explanation. Our current manuscript already provides analyses of these sufficient conditions in the appendices, but their roles and connections to the main theoretical guarantees are not sufficiently highlighted. We clarify them below.

When can the approximation residual be small?

The key is whether the dominant distributional change can be captured by a relatively simple velocity field. Appendix E.1 (Proposition 6) provides a general sufficient condition: if a time-dependent coordinate transformation makes the conditional distribution of a high-dimensional block invariant along the annealing path, the corresponding velocity removes this part of the distributional change. The remaining Stein residual is then determined only by the path score of the residual marginal. Thus, if the second moment of this marginal score is controlled, the approximation residual remains small even as the ambient dimension grows. Importantly, exact representation is not necessary: equations (87)–(88) further show that the error from approximating the ideal velocity contributes additively to the residual bound. This explains when the small-residual condition required by Theorem 1 can be achieved.

AuthorsOpenReview ↗

Author Rebuttal (2/3)

When can the assumptions of Theorem 2 be satisfied?

Theorem 2 requires both a candidate class with controlled residual variation, $E_{m_d}(t)\le C_0r_d$, and a fitting error of order $\varepsilon_{k,m_d}^{\rm fit}\le C_1h_k(1+\sqrt{r_d})$, with constants independent of the ambient dimension. The first condition follows under the structural conditions described above when the residual marginal score has second moment of order $r_d$.

For the fitting condition, Assumption 4 in Appendix D.1 provides sufficient statistical conditions: bounded Stein features and centered responses, a well-conditioned feature covariance, and bounded population-optimal coefficients. Under these conditions, Proposition 4 establishes finite-sample bounds on coefficient estimation and excess Stein residual. Proposition 5 further controls the error from averaging endpoint coefficients, provided the population-optimal coefficients vary smoothly along the annealing path. These results clarify that dimension-independent fitting requires the relevant statistical constants, not just the parameter count, to remain controlled.

Finally, reliable adaptive selection requires representative calibration data and accurate held-out acceptance estimates. Assumption 3 and Proposition 3 control initialization error, dependence between fitting and validation blocks, and validation concentration, respectively. Together, they ensure that the selected candidate meets the prescribed population acceptance target with high probability.

The example in Q1 demonstrates that these approximation, fitting, and calibration conditions can hold simultaneously in a non-Gaussian setting. We will reorganize the theoretical appendix around these connections and bring the key sufficient conditions and their implications into the main text.

W2: Role of Proposition 1.

We agree that Proposition 1 serves primarily as a supporting guarantee. Its role is to connect held-out acceptance certification with global communication under DEO and ELE, while ensuring that the frozen Metropolis-corrected sampler preserves the target distribution. Together with Theorem 1, it completes the link from Stein fitting to finite-gap exchange performance and then to round-trip communication. We will clarify this role in the revised presentation.

W4: Sampling accuracy beyond round trips and mode visits.

We thank you for pointing out that communication metrics alone are insufficient to establish sampling accuracy, and that a real-data Bayesian experiment would provide stronger evidence of practical value. We have conducted four additional experiments to address both concerns. Our original large-scale experiments were conducted on rented servers. For these supplementary experiments, we used our local machine (AMD Ryzen 5 7640H, 32 GB RAM), with all methods running serially on the same CPU configuration. Each result is averaged over three independent runs, with 16 PT ensembles per run, 1,000 warm-up sweeps, and 5,000 production sweeps.

We also evaluate sampling accuracy against analytical reference values. We report three types of errors: (i) mode-weight TV, which measures whether the relative probabilities of different modes are correctly recovered; (ii) RMSE of posterior moments and expectations; and (iii) relative covariance error, which measures the accuracy of the estimated covariance structure.

TaskAccuracy metricRaw DEO-PTCAT-PTNF-APT
GaussianCoordinate-averaged mean RMSE (×10⁻³)0.447 ± 0.0650.380 ± 0.0440.390 ± 0.029
First-coordinate mean RMSE (×10⁻³)1.288 ± 0.4711.529 ± 0.1241.573 ± 0.117
Coordinate-averaged second-moment RMSE (×10⁻³)0.884 ± 0.1370.772 ± 0.0690.793 ± 0.063
APT GMMMode-weight TV ↓0.2547 ± 0.00730.1359 ± 0.00330.1360 ± 0.0042
Full-vector mean RMSE ↓0.0410 ± 0.00730.0160 ± 0.00110.0153 ± 0.0035
Relative covariance error ↓0.1871 ± 0.02920.0977 ± 0.01690.0852 ± 0.0111
Many-WellMode-weight TV ↓0.0910 ± 0.00690.0485 ± 0.00400.0600 ± 0.0063
Active-coordinate mean RMSE (×10⁻³)51.79 ± 8.5032.89 ± 2.4140.65 ± 6.55
Active-coordinate second-moment RMSE (×10⁻³)4.26 ± 0.374.52 ± 1.105.21 ± 0.64
Positive-probability RMSE (×10⁻³)15.37 ± 2.639.70 ± 0.6512.02 ± 1.97

CAT-PT substantially improves mode-weight estimation over Raw DEO-PT and achieves moment and covariance estimation accuracy comparable to NF-APT.

AuthorsOpenReview ↗

Author Rebuttal (3/3)

W5: Bayesian inference on real data.

We additionally conducted Bayesian logistic regression on the real Pima Indians diabetes dataset, comprising 532 observations and seven standardized predictors, resulting in an eight-dimensional posterior. We use Gaussian priors and a likelihood-tempering path from the prior to the posterior.

For reference, we ran four independent Polya–Gamma Gibbs chains, each retaining 50,000 samples after 2,000 burn-in iterations (200,000 reference samples in total). The minimum reference ESS was 60,599, and the maximum rank/folded split R-hat was 1.000052.

Posterior accuracy metric ↓Raw PTCAT-PTNF-APT
Parameter mean RMSE (×10⁻³)2.350 ± 0.2081.721 ± 0.0401.749 ± 0.046
Mean RMSE / posterior SD0.01649 ± 0.001390.01245 ± 0.000250.01257 ± 0.00047
Posterior SD relative RMSE0.01731 ± 0.000990.01267 ± 0.000160.01285 ± 0.00113
Parameter second-moment RMSE (×10⁻³)2.707 ± 0.3012.243 ± 0.1032.228 ± 0.156
Predictive probability RMSE (×10⁻³)1.778 ± 0.1790.945 ± 0.0751.017 ± 0.014

CAT-PT achieves posterior estimation accuracy comparable to NF-APT across parameter means, standard deviations, and predictive probabilities. Its predictive probability RMSE is slightly lower, demonstrating that improved communication translates into accurate posterior estimates on real data.

Complete runtime.

We measure complete runtime from process startup to termination, including initialization, calibration, fitting, validation, warm-up, production sampling, diagnostics, and output. Reference generation is performed separately and excluded.

TaskRaw PT (s)CAT-PT (s)NF-APT (s)NF / CAT
Shifted Gaussian17.26 ± 0.1321.45 ± 0.18505.36 ± 4.5123.56×
APT GMM72.98 ± 0.4593.02 ± 0.48864.50 ± 2.409.29×
Structured Many-Well64.84 ± 1.2780.07 ± 1.28808.93 ± 5.5110.10×
Pima Logistic43.74 ± 0.8759.20 ± 0.36528.22 ± 4.138.92×

For Pima, calibration takes approximately 10 seconds for both methods, while sample collection, fitting, and validation take 10.31 seconds for CAT-PT versus 402.37 seconds for NF-APT. Across all four tasks, CAT-PT requires only 4.2%–11.2% of NF-APT's complete runtime. Although NF-APT completes more total round trips under the fixed production budget on Pima, CAT-PT achieves approximately 5× higher round trips per second and 8.9× higher parameter GM ESS/s, with comparable posterior accuracy.

We hope these additional results address your concerns. We sincerely appreciate your constructive feedback and would be grateful if you would consider revising your assessment in light of these new results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.