acceptodds
Under review as a conference paper at ICLR 2027

Draw the Graph, Draw the Negatives: A Local Sampling Framework for Contrastive Estimation

Abstract

Noise Contrastive Estimation (NCE) provides a powerful framework for learning unnormalized models by contrasting data with negative samples. While classical NCE draws negatives globally from a reference distribution, Conditional NCE (CNCE) instead generates local negatives around each data point. Existing local constructions, however, are largely restricted to binary discrimination and simple negative-sampling schemes. In this work, we propose a general framework for local negative sampling in NCE that extends the binary view to multi-way classification through graphical negative-sampling schemes. We characterize when the geometry of a generation graph hides the origin of the sampling, in which case the proposal terms cancel and the exact Bayes posterior reduces to an energy-only softmax. Our framework recovers an existing method as a hidden-anchor construction and connects graphical negative sampling to contrastive divergence. Building on this framework, we further introduce a new method, Generative Origin Recognition on cyclic Indices Contrastive Estimation (GORI-CE), which generates negatives through a cyclic Gaussian bridge. Experiments on both synthetic and real data demonstrate the effectiveness of our method and highlight negative-sampling geometry as a useful design principle for energy-based learning.

Primary area:
generative models
Keywords:
noise contrastive estimation, energy-based models, negative sampling, density estimation, conditional noise contrastive estimation, contrastive divergence, generative modeling

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_38yTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a graph-based local sampling framework for noise-contrastive estimation. The work aims to address geometric shortcuts that can reveal the positive sample in a locally generated candidate pool without learning the target energy function. The authors first formulate a multi-way classification task over jointly sampled candidates. Then they use symmetric generation graphs to construct sampling kernels whose proposal terms cancel, yielding an energy-only objective. They further introduce GORI-CE, a cyclic Gaussian-bridge sampler, and establish Fisher consistency under stated assumptions and a connection to contrastive divergence. Experiments cover synthetic density estimation, five real-data density-estimation benchmarks, and latent-space image generation on SVHN, CIFAR-10, and CelebA.

Critical strengths
  1. The geometric shortcut identified in local multi-negative NCE is an interesting problem. Using graphs to design the candidate pool is an intuitive way to address it.
  2. The paper has a strong theoretical focus, with detailed derivations supporting the proposed sampling framework.
Critical weaknesses
  1. It is unclear whether the gains come from the cyclic structure or simply from sampling negatives at different distances. A multi-scale Gaussian baseline with a similar distance distribution and the appropriate NCE correction would help answer this question.
  2. The paper lacks ablation studies on the number of negative samples, the sampling distance, and the number of anchors in cyclic anchoring. The authors should vary these settings to show how each affects performance.
  3. The image experiments are limited to SVHN, CIFAR-10, and CelebA, all at low resolutions. Results on a larger and more diverse dataset, such as ImageNet at a suitable resolution, would make the case for scalability and generalization more convincing.
  4. The experimental evaluation is still limited in terms of datasets and metrics. The authors should provide a broader assessment of generation quality, sample diversity, and computational efficiency.
Questions for authors

See weaknesses. If the authors can adequately address these concerns, I would consider raising my score.

6 comments

We thank the reviewer for their thoughtful comments and constructive feedback. We address each point below, with the corresponding changes incorporated into the revised manuscript and highlighted in blue.


W1: Multiscale comparison

It is unclear whether the gains come from the cyclic structure or simply from sampling negatives at different distances.

  • On Multiscale comparison: Thank you for raising this important point. To investigate whether the gains of GORI-CE arise solely from multi-scale negative sampling, we leverage our graph-based Local NCE framework to construct a symmetric, non-cyclic multi-scale sampler based on a hierarchical Gaussian anchor (Appendix E.1, Figures 4–6 of the revised paper). The sampler generates negatives at three distance scales while preserving the energy-only NCE objective. We match the candidate budget J=19J=19 and overall RMS displacement, with a negative-distance distribution comparable to that of GORI-CE.
  • On HEPMASS and MINIBOONE, the multi-scale anchor sampler outperforms Anchor at larger RMS distances, confirming that multi-scale sampling contributes to the improvement. However, GORI-CE consistently achieves higher mean test log-likelihood at these scales, suggesting that multi-scale distances alone do not fully explain its advantage and that the cyclic dependence structure may provide additional benefits.

W2: Ablation on RMS vv, negative counts JJ, and anchor count KK

The paper lacks ablation studies on the number of negative samples, the sampling distance, and the number of anchors in cyclic anchoring.

  • On RMS vv: We perform an ablation study on the RMS v∈0.1,0.25,0.5,0.75,1.0v\in\\{0.1,0.25,0.5,0.75,1.0\\} across five density-estimation benchmarks and three image datasets. Please see Section 6.2, Figure 3; Appendix E.2, Figure 7 of the revised paper. While the methods perform comparably at smaller distances, GORI-CE exhibits substantially less performance degradation as v increases, demonstrating greater robustness to the sampling scale.

  • FID vs. RMS distance vv (CelebA 64; mean ± SD over 3 prior seeds; FID against the test set. Lower is better.)

    Methodvv = 0.10.250.50.751
    Anchor (ED)61.0 ± 6.946.7 ± 2.346.8 ± 0.954.3 ± 2.058.7 ± 2.3
    Cycle (GORI)60.4 ± 7.745.8 ± 0.646.6 ± 0.348.8 ± 0.451.2 ± 0.7
    CA 2×1061.5 ± 8.346.3 ± 0.746.8 ± 0.651.8 ± 0.957.3 ± 0.8
    CNCE63.2 ± 8.846.8 ± 0.848.1 ± 1.055.7 ± 3.660.1 ± 1.9
    Baseline (no scale)CD-1CD-5CD-10DSM
    FID65.1 ± 7.148.8 ± 1.448.3 ± 2.751.4 ± 0.1
  • On Negative count JJ: We chose the default J=19J=19 to use a negative-sample budget comparable to the M=16M=16 samples employed in the image generation experiments of ED [1]. We additionally conducted an ablation with J∈3,19,49,79,99J\in\\{3,19,49,79,99\\} on HEPMASS and MINIBOONE in Appendix E.2, Figure 8 in the revised version. Both Anchor and GORI-CE improve as JJ increases, but GORI-CE exhibits substantially larger gains, with a widening performance gap over Anchor.

  • Test log-likelihood vs. number of negatives JJ (HEPMASS and MINIBOONE; mean ± SD over seeds 0–2; J+1J+1 candidates per pool. Higher is better.)

    DatasetMethodJJ = 319497999
    HEPMASSAnchor (ED)-20.58 ± 0.04-19.43 ± 0.06-18.89 ± 0.07-18.61 ± 0.04-18.55 ± 0.01
    HEPMASSCycle (GORI)-20.47 ± 0.05-18.47 ± 0.06-17.68 ± 0.08-17.46 ± 0.11-17.33 ± 0.10
    MINIBOONEAnchor (ED)-43.59 ± 0.64-41.66 ± 0.65-39.89 ± 0.92-38.95 ± 0.34-38.74 ± 0.55
    MINIBOONECycle (GORI)-43.22 ± 1.30-32.48 ± 0.57-28.60 ± 0.85-27.30 ± 0.32-26.53 ± 0.63
  • On anchor count KK: We vary K∈1,2,5,10,20K\in\\{1,2,5,10,20\\} in cyclic anchoring on HEPMASS and MINIBOONE, fixing J=19J=19 and v=1v=1 (Appendix E.2, Figure 9 of the revised paper). Increasing KK consistently improves test log-likelihood, but even with K=20K=20, cyclic anchoring remains substantially worse than GORI-CE. This suggests that increasing the number of anchors alone cannot fully reproduce the benefits of cyclic sampling.

  • Test log-likelihood vs. Number of anchors KK (HEPMASS and MINIBOONE, mean ± SD over seeds 0–2; cyclic anchoring with KK anchors × 20/K20/K nodes each; K=1K=1 is Anchor (ED), K=2K=2 is CA 2×10)

    DatasetKK = 1 (Anchor)2 (CA 2×10)5 (CA 5×4)10 (CA 10×2)20 (CA 20×1)Cycle (GORI)
    HEPMASS-19.43 ± 0.06-19.37 ± 0.08-19.28 ± 0.03-19.16 ± 0.05-19.08 ± 0.02-18.47 ± 0.06
    MINIBOONE-41.66 ± 0.65-40.46 ± 0.38-39.23 ± 1.31-38.43 ± 0.25-37.33 ± 0.76-32.48 ± 0.57

W3: ImageNet

The image experiments are limited to SVHN, CIFAR-10, and CelebA, all at low resolutions. Results on a larger and more diverse dataset, such as ImageNet at a suitable resolution, would make the case for scalability and generalization more convincing.

  • On additional dataset(ImageNet): Our original experiments on SVHN, CIFAR-10, and CelebA64 follow the established evaluation settings of latent-space EBMs [1,2,3,4], where quantitative results are typically reported at 32×3232\times32 or 64×6464\times64 resolution. Higher-resolution studies in this line often involve additional components, such as diffusion-amortized MCMC [4]. Moreover, existing EBM studies commonly evaluate on ImageNet32 [5–8], while ImageNet64 is a widely used benchmark for unconditional generative modeling [9–12]. We therefore extend our evaluation to ImageNet64, retaining the resolution of CelebA64 while substantially increasing dataset scale and diversity (1.28M training images, 1,000 classes).
  • Since our primary focus is on EBM training objectives rather than generator architectures, we use a publicly available ImageNet-pretrained autoencoder DC-AE [13] as the frozen first stage and train only the latent EBM prior. All methods share the same autoencoder, sampling procedure (GMM-based SIR followed by MALA), and evaluation protocol (FID, precision, and recall), allowing us to isolate the effect of the EBM training objective.
  • All Local NCE methods achieve competitive FID scores of approximately 85–86, comparable to CD-10, while DSM achieves the lowest FID (84.0). In terms of precision and recall, GORI-CE achieves higher recall than DSM (0.216 vs. 0.188) while maintaining comparable precision. These results demonstrate that our Local NCE framework remains competitive on a substantially larger and more diverse dataset, supporting its applicability beyond the original image benchmarks. We have added these results to Appendix E.4 of the revised paper.

W4: Sample quality and cost

The authors should provide a broader assessment of generation quality, sample diversity, and computational efficiency.

  • On sample quality and cost: We additionally evaluate sample quality and diversity across the Local NCE methods and CD/DSM baselines using the improved precision and recall metrics [14] (Appendix E.3, Figure 10 of the revised paper). We also compare wall-clock time per training update (Appendix E.3, Figure 11 of the revised paper).
  • The results reveal dataset-dependent quality–diversity trade-offs. In particular, GORI-CE achieves one of the highest precision values on CelebA64 and remains competitive on SVHN and CIFAR-10, although no method consistently dominates both metrics. Local NCE methods require only 2.0–2.3 ms per update, substantially less than DSM (3.2 ms) and CD-1/5/10 (4.1/7.8/12.3 ms).
  • Together, these results highlight the practical value of the Local NCE framework, which supports diverse sampling constructions while maintaining competitive generation quality and low computational cost.

Reference

[1] Tobias Schröder, Zijing Ou, Jen Lim, Yingzhen Li, Sebastian Vollmer, and Andrew Duncan. Energy discrepancies: a score-independent loss for energy-based models. Advances in Neural Information Processing Systems, 36:45300–45338, 2023.

[2] Pang et al. Learning Latent Space Energy-Based Prior Model. NeurIPS 2020.

[3] Xiao and Han. Adaptive Multi-stage Density Ratio Estimation for Learning Latent Space Energy-based Model. NeurIPS 2022.

[4] Yu et al. Learning Energy-Based Prior Model with Diffusion-Amortized MCMC. NeurIPS 2023.

[5] Du and Mordatch. Implicit Generation and Modeling with Energy-Based Models. NeurIPS 2019.

[6] Du et al. Improved Contrastive Divergence Training of Energy-Based Models. ICML 2021.

[7] Lee et al. Guiding Energy-based Models via Contrastive Latent Variables. ICLR 2023.

[8] Zhu et al. Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood. ICLR 2024.

[9] Chrabaszcz et al. A Downsampled Variant of ImageNet as an Alternative to the CIFAR Datasets. arXiv:1707.08819, 2017.

[10] Child. Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images. ICLR 2021.

[11] Nichol and Dhariwal. Improved Denoising Diffusion Probabilistic Models. ICML 2021.

[12] Kingma et al. Variational Diffusion Models. NeurIPS 2021.

[13] Chen et al. Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models. ICLR 2025.

[14] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.

Thanks for the response. I appreciate the authors' efforts to address my concerns. However, I still have three concerns:

  1. ImageNet results: Figure 12 provides visual results, but I could not find a table with the exact FID, precision, and recall values.
  2. Image resolution: The evaluation is still limited to 64×64 or lower, which is insufficient for modern image generation tasks.
  3. Generation quality: The reported FID scores are much worse than those of existing strong generative models, raising concerns about practical effectiveness. Overall, I appreciate the additional efforts, and I will maintain the score.
Reviewer_X9eZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a general framework for NCE which supports multiple negative sampling, based on symmetry as design principle. The framework shows that graphs generate symmetric kernels, which in turn enable a proper support for multiple negatives. Lastly, they propose a NCE method based on cyclic graph called GORI-CE and test it on synthetic data, tabular data, and image data.

Critical strengths

The paper proposes some interesting theoretical contributions, especially showing that symmetric graphs implies symmetric kernels. The proposed framework reconnects existing methods such as ED with a graph perspective. They show that ED corresponds to a direct star graph, while the proposed method uses a cycle graph, which works better because it exposes the classifier to both local and distant negatives (opposed to the star graph). I am not expert in the NCE field so I cannot judge with certainty the novelty of the work.

Critical weaknesses

I am not an expert in NCE so my concerns are mostly experimental

1.The experimental comparison is a bit narrow and focuses solely on the same family. For example some methods based on contrastive divergence should be compared. The image results (tab. 2) are not conclusive, most results are within noise - besides direct star. 2. The main claim for the cycle kernel is that is spreads negative across different distances; but it seems that the experiments are run with a fixed RMS value and results are not reported at different values. Hence I think the experiments do not fully support the claim

Minor:

The presentation of the paper could be made a bit easier to follow and is a bit heavy in the maths. In the current form, it is not trivial to understand what is prior method and what is novel.

Questions for authors
  • How do authors explain the higher margin of the proposed method on tabular data compared to images ?

  • Using J=19 negative samples with the cycle kernel limits the actual distance of the negatives, given that the RMS is fixed. How was J=19 (or 16) picked? has authors performed experiment with higher J? Pushing J or RMS to higher values could be interesting when compared against mode jumping (which is mentioned in the related works)

8 comments

We thank the reviewer for their thoughtful comments and constructive feedback. We address each point below, with the corresponding changes incorporated into the revised manuscript and highlighted in blue.


W1-1: Recent baseline

The experimental comparison is a bit narrow and focuses solely on the same family.

  • On recent baseline: We thank the reviewer for suggesting comparisons with additional EBM training objectives. We have added denoising score matching (DSM) and contrastive divergence (CD-1, CD-5, and CD-10) baselines for image generation. Please see Section 6.2, Appendix E.3, and Appendix F.3 in the revised paper.
  • To isolate the effect of the prior-learning objective, all methods share the same frozen generator, latent training data, energy architecture, and number of training updates. Across SVHN, CIFAR-10, and CelebA64, GORI-CE achieves competitive FID relative to DSM and CD, while exhibiting substantially greater robustness to the sampling RMS distance. Furthermore, GORI-CE requires only 2.1 ms per training update, compared with 3.2 ms for DSM and 4.1–12.3 ms for CD.
  • These comparisons focus on the effectiveness and computational efficiency of learning a latent EBM prior, rather than end-to-end image generation performance against diffusion models.

W1-2: Direct Star in image generation

The image results (tab. 2) are not conclusive, most results are within noise - besides direct star.

  • On the Exclusion of Direct Star from Image Comparisons: Regarding Direct star, its geometric correction dominates the classification logits in high-dimensional image latent spaces, causing the loss and gradients to vanish. On SVHN, the energy network remains at its zero initialization, yielding the same performance as the untrained prior. We therefore exclude Direct star from the image comparisons and clarify this behavior in Appendix F.3 of the revised version.

Q1: Small performance gap in image generation

How do authors explain the higher margin of the proposed method on tabular data compared to images ?

  • On the smaller performance gap in image generation: The larger performance margins on tabular data can be partly explained by the different RMS distances used in the original comparisons: v=1 for tabular density estimation and v=0.25 for image generation. As shown in the additional RMS sweeps (Figure 3 and Appendix E.2, Figure 7 of the revised paper), the methods perform comparably at smaller distances, while GORI-CE exhibits substantially greater robustness as v increases.

W2, Q2: Ablation on RMS vv, negative counts JJ

But it seems that the experiments are run with a fixed RMS value… How was J=19 (or 16) picked? Pushing J or RMS to higher values could be interesting…

  • On RMS distance v: We perform an ablation study on the RMS distance v∈0.1,0.25,0.5,0.75,1.0v\in\\{0.1,0.25,0.5,0.75,1.0\\} across five density-estimation benchmarks and three image datasets. Please see Section 6.2, Figure 3; Appendix E.2, Figure 7 of the revised paper. While the methods perform comparably at smaller distances, GORI-CE exhibits substantially less performance degradation as v increases, demonstrating greater robustness to the sampling scale.

  • FID vs. RMS negative distance vv (CelebA 64; mean ± SD over 3 prior seeds; FID against the test set. Lower is better.)

    Methodvv = 0.10.250.50.751
    Anchor (ED)61.0 ± 6.946.7 ± 2.346.8 ± 0.954.3 ± 2.058.7 ± 2.3
    Cycle (GORI)60.4 ± 7.745.8 ± 0.646.6 ± 0.348.8 ± 0.451.2 ± 0.7
    CA 2×1061.5 ± 8.346.3 ± 0.746.8 ± 0.651.8 ± 0.957.3 ± 0.8
    CNCE63.2 ± 8.846.8 ± 0.848.1 ± 1.055.7 ± 3.660.1 ± 1.9
    Baseline (no scale)CD-1CD-5CD-10DSM
    FID65.1 ± 7.148.8 ± 1.448.3 ± 2.751.4 ± 0.1
  • On Negative count J: We chose the default J=19J=19 to use a negative-sample budget comparable to the M=16M=16 samples employed in the image generation experiments of ED [1]. We additionally conducted an ablation with J∈3,19,49,79,99J\in\\{3,19,49,79,99\\} on HEPMASS and MINIBOONE in Appendix E.2, Figure 8 in the revised version. Both Anchor and GORI-CE improve as JJ increases, but GORI-CE exhibits substantially larger gains, with a widening performance gap over Anchor.

  • Test log-likelihood vs. number of negatives JJ (HEPMASS and MINIBOONE; mean ± SD over seeds 0–2; J+1J+1 candidates per pool. Higher is better.)

    DatasetMethodJJ = 319497999
    HEPMASSAnchor (ED)-20.58 ± 0.04-19.43 ± 0.06-18.89 ± 0.07-18.61 ± 0.04-18.55 ± 0.01
    HEPMASSCycle (GORI)-20.47 ± 0.05-18.47 ± 0.06-17.68 ± 0.08-17.46 ± 0.11-17.33 ± 0.10
    MINIBOONEAnchor (ED)-43.59 ± 0.64-41.66 ± 0.65-39.89 ± 0.92-38.95 ± 0.34-38.74 ± 0.55
    MINIBOONECycle (GORI)-43.22 ± 1.30-32.48 ± 0.57-28.60 ± 0.85-27.30 ± 0.32-26.53 ± 0.63

W3: Presentation

The presentation of the paper could be made a bit easier to follow and is a bit heavy in the maths. In the current form, it is not trivial to understand what is prior method and what is novel.

  • Presentation: To improve readability, we have made several revisions. First, we added a summary of our main contributions in the revised Introduction to provide a clear overview of the paper. In Section 3.2, we clarified the motivation for kernel symmetry. We also added an intuitive explanation after Definition 1 of how symmetry eliminates the kernel terms from the posterior, giving an energy-only objective. At the end of Section 4.1, we added an explanation of how the mathematical operations, including graph Laplacians and Schur complements, lead to the Gaussian sampling kernel. Finally, we revised Section 6 and Appendix E to better explain how GORI-CE's multi-scale sampling contributes to its robustness across RMS distances.
  • If there are other places here you need further clarification, please let us know and we are more than happy to clarify further.

Reference

[1] Tobias Schröder, Zijing Ou, Jen Lim, Yingzhen Li, Sebastian Vollmer, and Andrew Duncan. Energy discrepancies: a score-independent loss for energy-based models. Advances in Neural Information Processing Systems, 36:45300–45338, 2023.

I thank the authors for the response and appreciate the new baselines, and the revisions to the text presentation.

However I have still concerns about the results:

  • Table 2 is now missing and it is way easier to read than Figure 3.
  • It seems to me that Figure 3 does not report GORI-CE ? I am unable to compare it on image generation with the new results. (Also DSM is never defined in the paper, until page 32 of the appendix. Please proof read your manuscript)
  • In the tables reported on the authors comments I have still the same issue: FID results are mostly within a standard deviation. The motivation that for tabular data the v=1 does not stand for images as the best FID is achieved with v=0.25.
  • Related to the point above, I wonder how the results on the tabular datasets change with v?
Reviewer_Q8a9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes a general framework for local Noise Contrastive Estimation (NCE) based on graphical negative-sampling schemes. The authors identify symmetry conditions under which the sampling-kernel terms cancel from the classification posterior. They further establish Fisher consistency under suitable assumptions, reinterpret the finite-particle Energy Discrepancy (ED) estimator within this framework, and connect symmetric local NCE to contrastive divergence. Building on these results, the paper introduces GORI-CE, which uses a cyclic Gaussian bridge to generate negative samples at different distance scales. Experiments demonstrate the effectiveness of the proposed approach.

Critical strengths
  1. The proposed framework provides a systematic interpretation of multi-way local NCE and explains how the geometry of jointly generated negatives affects the classification posterior.
  2. The graph-based formulation provides a principled way to construct symmetric sampling kernels. GORI-CE is computationally simple, and its Gaussian-bridge construction produces negative samples with different marginal displacement scales while preserving an energy-only objective.
Critical weaknesses
  1. The main idea of using symmetric proposals to cancel sampling terms has been studied in multiproposal MCMC (Tjelmeland, 2004; Calderhead, 2014; Holbrook, 2023). The connection between NCE and contrastive divergence was also established for RNCE and CNCE by Olmin et al. (2024). The paper could better explain how its theoretical results differ from these existing works.
  2. Section 5.2 motivates GORI-CE through its ability to generate negatives at different distances. However, compared with ED, GORI-CE also changes the correlations among negative samples. The current experiments do not fully establish whether the gains can be explained by multi-scale sampling alone or whether the cyclic dependence structure provides additional benefits.
Questions for authors
  1. Could the authors clarify how Definition 1 and Theorem 4 differ from existing results on symmetric multiproposal MCMC (e.g., Holbrook, 2023) and the NCE–contrastive divergence connection (Olmin et al., 2024)?
  2. Could the authors compare GORI-CE with a symmetric non-cyclic multi-scale sampler that has a similar negative-sample distance distribution, under the same candidate budget and average RMS displacement? This would help determine whether the performance gains mainly come from multi-scale sampling or from the specific correlations induced by the cyclic structure.

Holbrook, A. J. (2023). Generating MCMC proposals by randomly rotating the regular simplex.

3 comments

We thank the reviewer for their thoughtful comments and constructive feedback. We address each point below, with the corresponding changes incorporated into the revised manuscript and highlighted in blue.


W1, Q1: Novelty on Definition 1 and Theorem 4

Could the authors clarify how Definition 1 and Theorem 4 differ from existing results on symmetric multiproposal MCMC and the NCE–contrastive divergence connection?

  • On Definition 1. We sincerely thank the reviewer for the feedback regarding the clarity of the novelty. While we agree that the cancellation of proposal terms under symmetric multiproposal kernels has been studied in prior work [1,2], our contribution is to formulate this principle for multi-way Local NCE through natural permutations of a joint conditional sampling kernel, without requiring full exchangeability. Furthermore, Definition 2 and Theorem 3 establish a constructive graph-theoretic sufficient condition based on definition of generation graph, enabling systematic design of symmetric samplers with cyclic and latent-variable structures. To the best of our knowledge, this general graph-theoretic criterion has not been established in prior work. We have clarified these distinctions in Appendix C.2 in the revised version.
  • On Theorem 4. The NCE–CD correspondence was previously studied for RNCE and CNCE [3]. We extend this result to multi-way Local NCE with jointly dependent conditional negatives by constructing a reversible multiproposal CD kernel whose Rao–Blackwellized update matches the Local NCE gradient. Unlike the independent proposals of RNCE or pairwise comparisons of CNCE, our construction applies to structured joint candidate pools. The proof uses balanced natural permutations (Lemma 10) to establish detailed balance, as detailed in Appendix C.1.
  • We have also added an explicit summary of our contributions to the Introduction of the revised paper to better distinguish our work from prior studies. If the reviewer could identify any specific parts that remain unclear, we would be happy to revise them further.

W2, Q2: Multiscale comparison

Could the authors compare GORI-CE with a symmetric non-cyclic multi-scale sampler that has a similar negative-sample distance distribution, under the same candidate budget and average RMS displacement?

  • On Multiscale comparison: To investigate whether the gains of GORI-CE arise solely from multi-scale negative sampling, we leverage our graph-based Local NCE framework to construct a symmetric, non-cyclic multi-scale sampler based on a hierarchical Gaussian anchor (Appendix E.1, Figures 4–6 of the revised paper). The sampler generates negatives at three distance scales while preserving the energy-only NCE objective. We match the candidate budget J=19J=19 and overall RMS displacement, with a negative-distance distribution comparable to that of GORI-CE.
  • On HEPMASS and MINIBOONE, the multi-scale anchor sampler outperforms Anchor at larger RMS distances, confirming that multi-scale sampling contributes to the improvement. However, GORI-CE consistently achieves higher mean test log-likelihood at these scales, suggesting that multi-scale distances alone do not fully explain its advantage and that the cyclic dependence structure may provide additional benefits.

Reference

[1] H Tjelmeland. Using all metropolis–hastings proposals to estimate mean values. norwegian university of science and technology, trondheim. Technical report, Norway. Tech Rep 4, 2004

[2] Andrew J Holbrook. Generating mcmc proposals by randomly rotating the regular simplex. Journal of multivariate analysis, 194:105106, 2023.

[3] Amanda Olmin, Jakob Lindqvist, Lennart Svensson, and Fredrik Lindsten. On the connection between noise-contrastive estimation and contrastive divergence. In International Conference on Artificial Intelligence and Statistics, pages 3016–3024. PMLR, 2024.

I thank the authors for their detailed responses and revisions. The theoretical clarifications and the additional multi-scale baseline have substantially addressed my technical concerns, and I will raise my soundness score to 3.

However, I still find the theoretical contributions relatively incremental. Moreover, the empirical advantages appear most pronounced at larger sampling distances, while the baselines remain competitive at well-performing sampling scales (e.g., v = 0.25). Therefore, I maintain my original overall rating.

Thanks for your thoughtful feedback.


I thank the authors for their detailed responses and revisions. The theoretical clarifications and the additional multi-scale baseline have substantially addressed my technical concerns, and I will raise my soundness score to 3.

  • Thank you for recognizing our efforts in revising the manuscript and for raising your soundness score. We appreciate your constructive feedback.

However, I still find the theoretical contributions relatively incremental.

  • While conventional NCE typically draws negative samples independently of the observed data point X0X_0, our local NCE framework introduces a time-scale parameter tt that controls the conditional sampling of negatives around X0X_0. We believe this establishes a distinct direction for NCE, contrasting with what may be called global NCE, rather than merely providing an incremental improvement to existing methods. We hope the reviewer will take this conceptual distinction into account when assessing the significance of our theoretical contribution.

Moreover, the empirical advantages appear most pronounced at larger sampling distances, while the baselines remain competitive at well-performing sampling scales (e.g., v = 0.25). Therefore, I maintain my original overall rating.

  • On RMS Robustness: We would like to clarify why robustness to vv remains important, even if v=0.25v=0.25 appears to be a sweet spot in several experiments.
  • As established in the ED paper (Theorem 2), the ideal ED objective converges to the maximum-likelihood objective (up to an additive constant) as v→∞v\to\infty. However, with a finite number of negative samples, increasing vv introduces estimation difficulties, leading to a trade-off explicitly discussed in the ED paper (Appendix D.6). Therefore, under a fixed negative-sample budget JJ, the practical goal is to choose the largest contrastive scale vv that can be effectively supported by the available negatives.
  • <u>In this sense, there is no universal sweet spot:</u> the optimal vv depends on the dataset, the negative-sample budget, and the sampling method. This fundamental trade-off is shared by kernel-based contrastive methods, including CNCE (Ceylan and Gutmann, 2018, Section 2). The trade-off in vv is illustrated in Appendix E.5 of the revised manuscript, showing that the optimal RMS scale varies with the dimensionality dd and the number of negatives JJ.
  • Importantly, our results show that while ED performs slightly better than GORI-CE in a narrow region of small vv on a few datasets, GORI-CE achieves substantially better performance over a much broader range of larger vv. Therefore, when the appropriate contrastive scale is uncertain, GORI-CE provides a valuable alternative to ED, offering better performance across a wider range of scales rather than relying on a narrowly tuned choice of vv.
  • Performance advantages on certain datasets: We also investigate the types of datasets on which GORI-CE performs particularly well. By generating negatives at multiple spatial scales, GORI-CE can be advantageous for multimodal distributions with heterogeneous component widths, as illustrated in Section E.5, Figure 13 of the revised paper. Furthermore, GORI-CE also exhibits performance advantages for distributions with well-separated modes, as shown in Section E.5, Figure 14. This further highlights the flexibility of the local NCE framework, where different sampling geometries can be chosen to match the characteristics of the target distribution. The robustness of GORI-CE to the choice of RMS is also demonstrated in Appendix E.5 in the revised paper.
Reviewer_PSKyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Training energy based model via NCE (noise contrastive estimation) has been limited to pairs, as drawing multiple negatives around a data point leaks geometric info. That is to say, training by comparing real data to fake negative samples usually only works with one negative at a time. if there are multiple negatives around a real data point, the real data point is usually in the center of these data, allowing the model to cheat using geometry and exploit this shortcut rather than learning the actual data distribution. here authors show arranging all the samples onto a symmetric graph hides the sample that is real. This removes the ability to take a geometric shortcut.The method creates both nearby and far away negative samples and avoid a narrow unhelpful distance. The loop prevents what we call cheating in machine learning and appears to be better than some other standard methods.

Critical strengths

Eliminating cheating: shows that graph symmetry stops the model from using the candidate centroid as a shortcut Outperforms standard baselines across synthetic tests and real world tabular benchmarks This is a new framework using graph theory to formalize and design local negative sample for energy-based models multiscale sampling includes near and far sampling, avoiding sample clustering

Critical weaknesses

Missing recent baselines. The comparison on NCE and energy discrepancy variants but not benchmarking against score-based or diffusion models. Mixed results on images, showing that advantages can be dataset dependent Involves only Gaussian perturbations and relies on manual tuning the noise scale Testing only of small, fixed number of negative samples and does not look at the question of infinite number of negatives Limited scope of what was compared - nce, energy discrepancy

Questions for authors

Can you look at speed, sample quality, stability against modern score-based/diffusion models? How sensitive are the results to the choice of noise scale? Can non-Gaussian noise, discrete data be used with this framework? What happens when negative counts scale to large numbers?

4 comments

We thank the reviewer for their thoughtful comments and constructive feedback. We address each point below, with the corresponding changes incorporated into the revised manuscript and highlighted in blue.


W1, Q1: Sample quality and cost against recent baseline

Can you look at speed, sample quality, stability against modern score-based/diffusion models?

  • On recent baseline: We thank the reviewer for suggesting comparisons with additional EBM training objectives. We have added denoising score matching (DSM) and contrastive divergence (CD-1, CD-5, and CD-10) baselines for image generation. Please see Section 6.2, Appendix E.3, and Appendix F.3 in the revised paper.
  • To isolate the effect of the prior-learning objective, all methods share the same frozen generator, latent training data, energy architecture, and number of training updates. Across SVHN, CIFAR-10, and CelebA64, GORI-CE achieves competitive FID relative to DSM and CD, while exhibiting substantially greater robustness to the sampling RMS distance. Furthermore, GORI-CE requires only 2.1 ms per training update, compared with 3.2 ms for DSM and 4.1–12.3 ms for CD.
  • These comparisons focus on the effectiveness and computational efficiency of learning a latent EBM prior, rather than end-to-end image generation performance against diffusion models.
  • On sample quality and cost: We additionally evaluate sample quality and diversity across the Local NCE methods and CD/DSM baselines using the improved precision and recall metrics [1] (Appendix E.3, Figure 10 of the revised paper). We also compare wall-clock time per training update (Appendix E.3, Figure 11 of the revised paper).
  • The results reveal dataset-dependent quality–diversity trade-offs. In particular, GORI-CE achieves one of the highest precision values on CelebA64 and remains competitive on SVHN and CIFAR-10, although no method consistently dominates both metrics. Local NCE methods require only 2.0–2.3 ms per update, substantially less than DSM (3.2 ms) and CD-1/5/10 (4.1/7.8/12.3 ms).
  • Together, these results highlight the practical value of the Local NCE framework, which supports diverse sampling constructions while maintaining competitive generation quality and low computational cost.

W2, Q2: Ablation on RMS vv, negative counts JJ

Mixed results on images… relies on manual tuning the noise scale Testing only of small, fixed number of negative samples. How sensitive are the results to the choice of noise scale? What happens when negative counts scale to large numbers?

  • On RMS distance vv: We perform an ablation study on the RMS distance v∈0.1,0.25,0.5,0.75,1.0v\in\\{0.1,0.25,0.5,0.75,1.0\\} across five density-estimation benchmarks and three image datasets. Please see Section 6.2, Figure 3; Appendix E.2, Figure 7 of the revised paper. While the methods perform comparably at smaller distances, GORI-CE exhibits substantially less performance degradation as v increases, demonstrating greater robustness to the sampling scale.

  • FID vs. RMS negative distance vv (CelebA 64; mean ± SD over 3 prior seeds; FID against the test set. Lower is better.)

    Methodvv = 0.10.250.50.751
    Anchor (ED)61.0 ± 6.946.7 ± 2.346.8 ± 0.954.3 ± 2.058.7 ± 2.3
    Cycle (GORI)60.4 ± 7.745.8 ± 0.646.6 ± 0.348.8 ± 0.451.2 ± 0.7
    CA 2×1061.5 ± 8.346.3 ± 0.746.8 ± 0.651.8 ± 0.957.3 ± 0.8
    CNCE63.2 ± 8.846.8 ± 0.848.1 ± 1.055.7 ± 3.660.1 ± 1.9
    Baseline (no scale)CD-1CD-5CD-10DSM
    FID65.1 ± 7.148.8 ± 1.448.3 ± 2.751.4 ± 0.1
  • On Negative count JJ: We chose the default J=19J=19 to use a negative-sample budget comparable to the M=16M=16 samples employed in the image generation experiments of ED [2]. We additionally conducted an ablation with J∈3,19,49,79,99J\in\\{3,19,49,79,99\\} on HEPMASS and MINIBOONE in Appendix E.2, Figure 8 in the revised version. Both Anchor and GORI-CE improve as JJ increases, but GORI-CE exhibits substantially larger gains, with a widening performance gap over Anchor.

  • Test log-likelihood vs. number of negatives JJ (HEPMASS and MINIBOONE; mean ± SD over seeds 0–2; J+1J+1 candidates per pool. Higher is better.)

    DatasetMethodJJ = 319497999
    HEPMASSAnchor (ED)-20.58 ± 0.04-19.43 ± 0.06-18.89 ± 0.07-18.61 ± 0.04-18.55 ± 0.01
    HEPMASSCycle (GORI)-20.47 ± 0.05-18.47 ± 0.06-17.68 ± 0.08-17.46 ± 0.11-17.33 ± 0.10
    MINIBOONEAnchor (ED)-43.59 ± 0.64-41.66 ± 0.65-39.89 ± 0.92-38.95 ± 0.34-38.74 ± 0.55
    MINIBOONECycle (GORI)-43.22 ± 1.30-32.48 ± 0.57-28.60 ± 0.85-27.30 ± 0.32-26.53 ± 0.63

W3, Q3: Generalization to non-Gaussian kernels and discrete data.

Mixed results on images, showing that advantages can be dataset dependent Involves only Gaussian perturbations… Can non-Gaussian noise, discrete data be used with this framework?

  • Our current construction focuses on Gaussian perturbations, following the continuous-data settings commonly studied in CNCE [4] and ED [2].
  • However, the kernel symmetry condition underlying our Local NCE framework (Definition 1) is not inherently restricted to Gaussian distributions. For instance, Gaussian edge kernels can be replaced with other symmetric kernels while preserving the required permutation symmetry.
  • Extending these constructions to non-Gaussian kernels and discrete state spaces requires further investigation. Notably, discrete extensions of ED have also been studied in separate work [3]. We have clarified these limitations and identified non-Gaussian kernels and discrete-data extensions as directions for future work in Section 8 in the revised paper.

Reference

[1] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.

[2] Tobias Schröder, Zijing Ou, Jen Lim, Yingzhen Li, Sebastian Vollmer, and Andrew Duncan. Energy discrepancies: a score-independent loss for energy-based models. Advances in Neural Information Processing Systems, 36:45300–45338, 2023.

[3] Tobias Schr¨oder, Zijing Ou, Yingzhen Li, and Andrew Duncan. Energy-based modelling for discrete and mixed data via heat equations on structured spaces. Advances in Neural Information Processing Systems, 37:79246–79281, 2024

[4] Ciwan Ceylan and Michael U Gutmann. Conditional noise-contrastive estimation of unnormalised models. In International Conference on Machine Learning, pages 726–734. PMLR, 2018.

Reviewer_PSKyOpenReview ↗

Response to authors

I very much appreciate the thoughtful work in response. I will be keeping my score, which was still positive.

Thank you for your kind words and for taking the time to carefully consider our responses. We sincerely appreciate your constructive feedback throughout the review process.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair for overseeing the review process and all reviewers for their careful reading and valuable feedback. To facilitate the evaluation of our revised submission, we provide below a concise summary of the main concerns raised by the reviewers and the corresponding revisions. Detailed responses to each point are provided in the individual reviewer comments, and the revisions are highlighted in blue in the revised paper.


Reviewers PSKy, X9eZ, 38yT: Sensitivity to sampling distance, negative count, and anchor count.

  • We conducted ablations on RMS distance vv, negative count JJ, and anchor count KK. GORI-CE shows greater robustness to large sampling distances, larger gains from increasing JJ, and consistently outperforms cyclic anchoring across different anchor counts (Section 6.2, Figure 3; Appendix E.2, Figures 7–9).

Reviewers PSKy, X9eZ, 38yT: Additional baselines, generation quality, and computational efficiency.

  • We added DSM and CD-1/5/10 baselines, precision–recall evaluations, and wall-clock comparisons. Local NCE achieves competitive generation quality while requiring only 2.0–2.3 ms per update, compared with 3.2 ms for DSM and 4.1–12.3 ms for CD (Section 6.2; Appendix E.3, Figures 10–11; Appendix F.3).

Reviewers Q8a9, X9eZ: Theoretical novelty and clarity.

  • We clarified our contributions relative to prior multiproposal MCMC and NCE–CD work, emphasizing permutation-based kernel symmetry, a graph-theoretic sufficient condition, and the extension of the NCE–CD correspondence. We also improved the theoretical explanations and added a summary of contributions (Introduction; Sections 3.2, 4.1–4.3; Appendices C.1–C.2).

Reviewers Q8a9, 38yT: Multi-scale sampling versus cyclic structure.

  • We introduced a symmetric, non-cyclic multi-scale baseline with a comparable negative-distance distribution and matched sampling budget. It improves over Anchor at larger RMS distances but remains below GORI-CE, suggesting that multi-scale sampling alone does not fully explain GORI-CE's advantage (Appendix E.1; Figures 4–6).

Reviewer 38yT: Scalability and evaluation on a larger image dataset.

  • We added ImageNet64 experiments using a shared frozen autoencoder. Local NCE achieves FID comparable to CD-10, while GORI-CE achieves higher recall than DSM at comparable precision, although DSM obtains the lowest FID (Appendix E.4, Figure 12; Appendix F.4).

Reviewer PSKy: Generalization to non-Gaussian kernels and discrete data.

  • We clarified that kernel symmetry is not inherently Gaussian-specific, while our current constructions focus on continuous Gaussian settings. We discuss non-Gaussian and discrete extensions as future directions (Section 8).

We hope this summary provides a clear overview of how we have addressed the reviewers' concerns in terms of clarity and empirical support. We sincerely appreciate your time and consideration and hope that these revisions will be taken into account in the evaluation of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.