acceptodds
Withdrawn from ICLR 2027

EnergyViG: Decoupled Energy-guided Structure Refinement for Vision GNNs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision Graph Neural Networks (ViG) treats an image as a set of visual patches for graph representation learning and yields promising results across various computer vision tasks. However, most existing works primarily focus on static graph construction, ignoring the performance gains and noise reduction benefits of dynamic structure refinement. Meanwhile, generative models such as Energy-based Models (EBMs) are generally unsuitable for discriminative tasks and struggle with large-scale images. Our goal is to introduce a unified generative-discriminative paradigm for dynamically modeling relationships between visual patches, aiming to produce higher-quality representations for improving downstream tasks. Specifically, we propose Decoupled Energy Learning (DEL) that defines a joint distribution of sample pairs to approximate the target distribution. It decouples EBMs into energy matching and contrastive learning as a global loss function, which pulls similar pairs closer and pushes dissimilar pairs further apart in the representation space. For implementation, we develop an end-to-end framework, termed Decoupled Energy learning guided Structure Refinement for improving ViG (EnergyViG). Structure refinement is deployed within ViG architectures in a plug-and-play manner, dynamically adding or pruning edges based on similarity metrics with a relaxation strategy. Theoretical analyses demonstrate the effectiveness of EnergyViG in processing large datasets through graph operations. Empirical evaluations confirm that it outperforms state-of-the-art methods on three major benchmarks, achieving 84.3% Top-1 accuracy on ImageNet-1K, 46.4% mAP on MS COCO, and 50.9% mIoU on ADE20K.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Decoupled Energy‑based Models, Graph Structure Refinement, Vision Graph Neural Networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3vAtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EnergyViG, a Vision GNN framework that dynamically refines graph structure. Its Decoupled Energy Learning objective decomposes a joint energy model over augmented graph views into a discriminative InfoNCE term and a generative EBM term estimated with Langevin sampling. Pairwise similarities are converted into a relaxed adjacency matrix using Gumbel-Sigmoid. Experiments report improvements on ImageNet-1K, MS COCO, and ADE20K.

Critical strengths
  1. Dynamic graph refinement is relevant for Vision GNNs, where fixed k-NN structures may contain noisy or missing connections.
  2. The evaluation covers classification, detection, instance segmentation, and semantic segmentation.
  3. The paper reports ablations on loss components, edge addition and pruning, Langevin iterations, and integration with different ViG backbones.
Critical weaknesses
  1. The theoretical claims are overstated. In Eq. (4), the energy EθE_\theta is bounded because the representations are unit-normalized. Thus, over the unbounded sampling space implied by Eq. (3), the partition function in Eq. (4) may diverge; the same issue affects the marginal EBM in Eq. (7). Moreover, Appendix A.1 treats the contrastive-divergence estimator in Eq. (11) as an exact likelihood objective, ignoring the dependence of samples from pθp_\theta on θ\theta unless they are explicitly stop-gradient. Finally, the convergence claim following Eq. (3) is not justified by the five Langevin steps used in practice (Fig. 3 and Appendix C.3).
  2. The novelty over ECL-GSR is unclear. Zeng et al., Graph Structure Refinement with Energy-based Contrastive Learning , AAAI 2025, 39(21):22326–22335, DOI: 10.1609/aaai.v39i21.34388, already combine generative EBM training, contrastive learning, and similarity-based edge refinement. Several central equations and the overall pipeline closely resemble ECL-GSR. The paper does not clearly isolate DEL’s new contribution or provide a direct adaptation baseline.
  3. The motivation for unifying generative and discriminative learning is insufficient. All evaluated tasks are discriminative, while no generative capability is evaluated. The paper does not explain what limitation of InfoNCE-based structure refinement requires marginal generative modeling.
  4. The presentation lacks precision. For the motivation section, “Comprehensively enhancing performance” is not a meaningful scientific motivation. Describing MNIST and CIFAR as “toy datasets” is dismissive and does not clarify the methodological challenge.
Questions for authors

My questions and concerns have already been presented in the Critical Weaknesses section.

Reviewer_tSQSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EnergyViG, an energy-guided graph structure refinement framework for Vision Graph Neural Networks (ViGs). The proposed method introduces Decoupled Energy Learning (DEL), which decomposes an energy-based learning objective into a generative term and a discriminative contrastive learning term. The framework is integrated into a multi-stage Vision GNN architecture and evaluated on image classification, object detection, and semantic segmentation. The reported results are competitive, achieving 84.3% Top-1 accuracy on ImageNet-1K, 46.4% box mAP on MS COCO, and 50.9% mIoU on ADE20K.

Critical strengths

S1. The proposed method demonstrates competitive results on three widely used benchmarks, suggesting that the framework can generalize across different downstream vision tasks rather than being tailored solely to image classification.

S2. The integration of energy-based learning, contrastive learning, and differentiable graph structure refinement is practically appealing. Jointly optimizing node representations and graph connectivity is a reasonable approach to improving Vision GNNs.

S3. The paper investigates the contributions of generative and discriminative objectives, edge addition and pruning, relaxation parameters, and Langevin sampling steps. The experiments on WiGNet-S and GreedyViG-S also provide evidence of the framework's applicability to different graph-based backbones.

Critical weaknesses

W1. Energy-based contrastive representation learning has already been investigated by EBCLR [1]. More importantly, ECL-GSR [2] combines energy-based modeling and contrastive learning to jointly improve graph representations and graph structures. ECL-GSR also learns pairwise relationships and refines graph connectivity by adding and removing edges based on representation similarities. These design principles substantially overlap with those of EnergyViG. Although the present work extends this direction to Vision GNNs and large-scale visual benchmarks, the manuscript does not sufficiently clarify its methodological differences from ECL-GSR. The authors should explain whether DEL introduces a fundamentally different objective, optimization procedure, or theoretical property compared with existing energy-based contrastive graph refinement methods. Without such clarification, the contribution appears primarily to be an architectural adaptation and a large-scale empirical extension of existing approaches.

W2. The paper claims that graph-based processing reduces the computational and memory costs of energy-based learning on large-scale visual data. However, as image resolution increases, the number of graph nodes also grows, leading to a quadratic increase in the computational and memory costs of pairwise similarity computation. This raises concerns about the scalability of the proposed graph refinement mechanism to high-resolution images. Although the paper reports FLOPs and parameter counts, a more detailed analysis of computational cost, memory usage, and scalability across different image resolutions would help support the claimed efficiency.

[1] Kim, B., & Ye, J. C. (2022). Energy-Based Contrastive Learning of Visual Representations. Advances in Neural Information Processing Systems. [2] Zeng, X., Wang, Y., Sun, Y., Guo, G., Ding, W., & Zhang, B. (2025). Graph Structure Refinement with Energy-based Contrastive Learning. Proceedings of the AAAI Conference on Artificial Intelligence.

Questions for authors
  1. What are the fundamental mathematical and algorithmic differences between DEL and the energy-based contrastive learning framework introduced by ECL-GSR [2]? Could the authors provide a detailed comparison of their objectives?

  2. How does the computational and memory cost of the proposed graph refinement mechanism scale with increasing image resolution? Could the authors provide experiments at different resolutions to evaluate its scalability, particularly considering the quadratic complexity of pairwise similarity computation as the number of visual patches increases?

Reviewer_eiLXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EnergyViG, an energy-guided structure refinement framework for VGNN. The core component of decoupled energy learning decomposes joint energy-based modeling into a discriminative contrastive objective and a generative energy-matching objective. The framework incorporates a differentiable graph refinement module that adjusts connections between visual patches using representation similarity and Gumbel-Sigmoid relaxation. EnergyViG is integrated into multi-stage Vision GNN backbones and evaluated on image classification, object detection, and semantic segmentation. The reported results achieve better performance in different datasets.

Critical strengths
  1. The paper addresses an important problem in Vision GNNs. Graph construction is an important component of visual graph representation learning. The proposed framework explores learnable structure refinement rather than relying exclusively on fixed k-NN connections. This is a meaningful direction for improving the flexibility of Vision GNN architectures.
  2. The integration of energy-based modeling and contrastive learning is conceptually interesting. The DEL objective combines discriminative representation alignment with generative energy matching. This provides an alternative approach to jointly optimizing visual representations and graph structures, potentially exploiting complementary information from two learning paradigms.
Critical weaknesses
  1. The graph refinement mechanism introduces potentially substantial computational overhead that is not adequately characterized.
  2. The theoretical contribution of Decoupled Energy Learning is insufficiently established. The central methodological contribution is the DEL objective, which decomposes joint energy-based learning into discriminative and generative components. Since DEL is presented as the central theoretical contribution, its novelty should extend beyond an algebraic decomposition and a weighted combination of established objectives. The current analysis does not sufficiently establish a distinctive theoretical advance.
Questions for authors

Except the concerns raised in critical weakness, some questions the authors should be addressed.

  1. Theoretical novelty. Since Eq. (6) follows directly from the probability chain rule, what is the fundamental theoretical contribution of DEL beyond existing hybrid energy-based contrastive learning approaches? How should the weighted objective in Eq. (9) be interpreted when alpha\neq 1?
  2. Energy-guided refinement. How does the DEL objective directly influence graph structure optimization? Can the authors provide empirical evidence that energy matching leads to more informative graph connections rather than merely improving node representations?
  3. Computational efficiency. How does EnergyViG compare with corresponding ViG baselines in end-to-end training time, inference latency, and peak GPU memory consumption under identical hardware and batch-size settings?
  4. Optimization details are ambiguous. How is the replay buffer initialized and updated during Langevin sampling? How are generated samples represented, and how are gradients handled through the energy-based sampling procedure?
  5. Structural analysis. Can the authors visualize the learned graph structures and quantify changes in edge density, semantic consistency, and connectivity before and after refinement?

Comments on the paper

1 comment

Dear reviewers,

If you believe the authors have addressed your questions, please respond to the rebuttal.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.