acceptodds
Under review as a conference paper at ICLR 2027

Sharp Concentration for Gibbs Distributions under Gaussian Score Noise

Abstract

Gibbs distributions, or softmax distributions under a uniform reference measure, convert scores into probabilistic decisions. We study their stability when the scores are perturbed by correlated Gaussian estimation errors. The forward Kullback-Leibler (KL) divergence between the population and noisy Gibbs distributions is exactly a nonlinear log-partition remainder after its first-order response to the score error is removed. Our main result gives a sharp, dimension-independent concentration bound for this remainder: after normalization by the largest variance of a pairwise score difference, its centered positive moment-generating function is bounded by that of a Gamma random variable with shape . The result holds for arbitrary finite action sets, reference probabilities, temperatures, Gaussian correlations, and noise magnitudes, and a balanced two-action problem approaches equality in the small-noise limit. The proof combines decreasing covariance envelopes under Gaussian smoothing with an explicit bound on the nonzero average gradient caused by asymmetric priors. We derive high-probability KL certificates, sample-complexity bounds, a data-dependent guarantee for unknown covariance, stronger results for structured action spaces, and trajectory-level guarantees for deterministic entropy-regularized planning. Numerical experiments examine tightness and the practical size of these certificates.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Gibbs distributions, softmax, Gaussian concentration, uncertainty quantification, KL divergence, moment-generating functions, policy stability, entropy-regularized reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cc9jOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes studying the stability of a distribution when its scores suffer a Gaussian perturbation. The main goal is to find a deviation inequality for the "non linear" residue of the log evidence. This is done in theorem 3.1.

The rest of the paper concerns discussing theorem 3.1 and its application to decision theory as well as providing some numerical illustrations on the tightness of the bound.

Critical strengths

The paper is concern with an important problem. Also, the main theorem is a nice result of interest for the community.

Critical weaknesses

I found the paper extremely hard to read. A lot of the definitions are re-explained with words and conversely some of the maths are done only with phrases and could benefit of formal definitions. Also, the paper discusses at length things that are relatively easy in my opinion while some other slightly more complicated things are done really fast. For example, in section 3.2 the authors discuss the fact that the Taylor remainder depends on the actual value that is being approximated, and this takes the same length as the section 3.1 where the authors use several (granted standard) concentration inequalities.

Another intriguing phrase is what appears in the beginning of section 3.3, namely phrase 145. The authors have already explained that it is not needed to introduce a covariance, namely in line 092, but here they introduce it. This covariance will not be used in the next lines at all, thus it begs the question of why stating it just before the main theorem ?

Corollary 3.2 is also extremely puzzling. The authors define a quantity for all α\alpha. Then the implication seems to be true only for α=1/2\alpha=1/2, namely equation 10. But it is not stated that they chose the q for α=1/2\alpha=1/2 before the conclusions, only after ! So the reader is uncertain of what does the result actually means and has to delve in the really entangled proofs to understand.

Some other terms are really hard to understand, such as : (155-156) "The theorem is therefore stronger than the local quadratic approximation: it retains the same Gamma benchmark ...". I have no idea what is a Gamma benchmark, even if I might suspect what it is...

Section 4 starts with the phrase "The proof tracks". It has not yet been explicited the proof of what. While we guess it is the proof of the main theorem, it shows well that we are always wondering if we are on the right track when reading this paper. I could not understand, even after several tries, the structure of the proof of theorem 3.1. There is a section in the appendice that has this name, but also section 4 seems to be trying to provide a roadmap of the proof.

Questions for authors

See above

1 comment
AuthorsOpenReview ↗

Awaiting Your Feedback on Our Rebuttal

Dear Reviewer cc9j,

Thank you again for your review. As the discussion period is drawing to a close, we would appreciate it if you could take a look at our rebuttal. We would be happy to clarify any points or address any remaining concerns while the discussion is still open.

Best regards,

The Authors

Reviewer_1aPaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the sensitivity of the Gibbs/softmax policy when the underlying scores are perturbed by correlated Gaussian noise. The primary contribution is a tight concentration bound on the forward KL divergence between the ideal policy and the perturbed policy; this bound holds for any noise level and, once normalized, does not depend explicitly on the number of actions.

These results provide a high-probability stability guarantee, establish a sample complexity of O(1/ϵ)O(1/\epsilon) to achieve a KL error of ϵ\epsilon, and extend to entropy-regularized planning under deterministic state transitions. Empirical evaluations demonstrate that this bound offers improvements over baseline methods, though it may still remain somewhat conservative.

Critical strengths
  • Establishes sharp Gamma-concentration bounds that are dimension-independent and hold for arbitrary levels of Gaussian noise, as well as various correlation structures.
  • Employs sophisticated proof techniques to address issues related to noise-dependent curvature and asymmetric probabilities; the sharpness of these results is demonstrated via an example in the small-noise regime.
  • Provides useful extensions covering scenarios with unknown covariance, structured action spaces, and entropy-regularized deterministic planning.
Critical weaknesses

No at the moment. I also dont have time to check the proof.

Questions for authors
  1. Can the forward-KL guarantee yield a useful high-probability bound on true regularized policy loss, which involves reverse KL? What additional assumptions would be needed?

  2. Given the substantial gap between certified and observed KL errors, can the sharper Gamma bound improve an actual sampling or temperature-selection procedure compared with the log-Sobolev baseline?

Reviewer_Ams2OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This is a well-written theory paper. It addresses how softmax policies can drift. It proves that the drift can be described in terms of a scaled variable for any number of actions, noise level, or correlation between the errors. The authors also give practical guarantees, and the bounds are quite good. One way to describe the question that they address is: How much can Gaussian score uncertainty change the resulting (Gibbs) distribution? Suppose we build a softmax policy from scores we have estimated. These estimates would carry Gaussian errors that may be correlated across actions. The question of interest is: how far can that policy be from the one built on true scores, as measured by KL divergence?

Critical strengths

The paper addresses a well-posed problem (see paper summary above). The main contribution is relevant since softmax policies built from estimated scores are present in reinforcement learning in a large number of use cases and are even relevant to large language models.

Critical weaknesses

This is not a critical weakness. It's more of a limitation: Because the scale is set by the worst pair of actions, the certificates appear to be too loose. The gain over a simpler standard bound is only a fixed 32% at a given rate. The experiments use synthetic noise, with no guidance on choosing the temperature or data budget.

Questions for authors

How about the following? This is a strong theory paper. It would be greatly helpful from a practical perspective if there was some guidance on the following: How would one choose the temperature to keep the policy stable under a fixed data budget? If the certificate is 10–100× loose, is it still useful for comparing designs, such as two temperatures or two data budgets?

Minor:

  • Potential typo: Stop this positive stopped supermartingale (redundant use of stop)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.