- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes JEP-VGP, a parameter-efficient adaptation framework for pretrained Vision Graph Neural Networks (ViGs), which addresses the limitation of underutilizing cross-region dependencies and contextual constraints in existing methods. Inspired by Joint-Embedding Predictive Architectures (JEPA), JEP-VGP uses contextual prediction to provide direct supervision for prompt-induced changes in node representations, complementing conventional image-level classification supervision. Experiments and analyses demonstrate the effectiveness of JEP-VGP and its components, including PREP and TGPL.
Critical strengths
-
The paper introduces PREP, which predicts prompt-induced node representation residuals from contextual residuals, extending joint-embedding prediction to vision graph prompt learning in a targeted way. This provides a principled auxiliary objective beyond conventional classification supervision and is supported by ablations showing consistent improvements over the VGP baseline.
-
On 10 classification benchmarks, JEP-VGP improves average accuracy from 89.6% to 91.1%, outperforming full fine-tuning on 9 of 10 datasets while using 94.6% fewer trainable parameters. The auxiliary prediction components are removed at inference, keeping inference FLOPs unchanged relative to VGP. These results support the method’s practical value for parameter-efficient adaptation.
Critical weaknesses
- Although the ablations show improvements from PREP and TGPL, the evaluation does not sufficiently establish whether the gains arise specifically from predicting prompt-induced residuals and type-guided learning, rather than from the additional auxiliary training supervision itself. Stronger comparisons against alternative self-supervised or contextual prediction objectives would help substantiate the methodological novelty.
Questions for authors
-
How sensitive is JEP-VGP’s performance to the choice of the EMA teacher update rate and the masking ratio? Do the reported experiments establish that the method performs robustly across these settings?
-
How does cyclically selecting prompt types for auxiliary prediction compare with jointly optimizing all prompt types under an equivalent training budget? This comparison would help clarify whether cyclic type selection is essential to the reported improvements.
-
Could the proposed method benefit other vision architectures beyond ViGs, and what adaptations would be required to generalize it to those architectures?
5 comments
Response to reviewer's comments Part 2
Comment 2 Questions 1
How sensitive is JEP-VGP’s performance to the choice of the EMA teacher update rate and the masking ratio? Do the reported experiments establish that the method performs robustly across these settings?
Response:
Thank you for your question about the EMA update rate and masking ratio. To further evaluate JEP-VGP's robustness to these hyperparameters, we add sensitivity experiments on CUB.
Sensitivity to the EMA decay coefficient and target-block area-ratio interval on CUB. Only one hyperparameter is varied at a time; all other settings are held fixed. We report Top-1 accuracy (%), with default settings in bold.
| Top-1 (%) | Top-1 (%) | ||
|---|---|---|---|
| 89.4 | 88.9 | ||
| 89.5 | 89.3 | ||
| 89.6 | 89.6 | ||
| 89.5 | 89.3 |
As shown above, varying the EMA decay coefficient from to changes Top-1 accuracy by only percentage points, indicating stability with respect to the EMA update rate. In comparison, the maximum performance difference across the tested target-block area-ratio sampling intervals is percentage points, with the default interval performing best. All tested configurations outperform VGP's , indicating that JEP-VGP maintains a consistent performance advantage over the examined hyperparameter ranges.
Comment 3 Questions 2
How does cyclically selecting prompt types for auxiliary prediction compare with jointly optimizing all prompt types under an equivalent training budget? This comparison would help clarify whether cyclic type selection is essential to the reported improvements.
Response:
Thank you for your suggestions on prompt type selection. To further analyze cyclic selection, we compare mixed-residual prediction (Joint) with type-specific cyclic prediction (Cyclic). As shown below, both settings activate all prompts in the classification branch and compute one residual prediction target per auxiliary update. Joint uses all three prompt types simultaneously to construct mixed residuals, whereas Cyclic selects a single prompt type for residual prediction according to Eq. (23).
Comparison between mixed-residual prediction and type-specific cyclic prediction. Both settings activate Node, Edge, and Graph Prompts in the classification branch and compute one residual prediction target per auxiliary update. We report Top-1 accuracy (%).
| Setting | Auxiliary prediction | DTD | CUB | Dogs | Flowers |
|---|---|---|---|---|---|
| A1 | Joint (mixed residual) | 76.4 | 88.7 | 82.3 | 98.6 |
| A6 | Cyclic (type-specific residual) | 78.9 | 89.6 | 83.6 | 99.4 |
Compared with Joint, Cyclic improves accuracy by , , , and percentage points on DTD, CUB, Dogs, and Flowers, respectively. These results indicate that type-specific cyclic prediction achieves better classification performance under the current training settings, supporting TGPL's cyclic selection strategy.
Response to reviewer's comments Part 3
Comment 4 Questions 3
Could the proposed method benefit other vision architectures beyond ViGs, and what adaptations would be required to generalize it to those architectures?
Response:
Thank you for your suggestions on PREP's cross-architecture applicability, which is also of great interest to us. Although JEP-VGP's type-guided mechanism depends on ViG's graph structure, PREP's core idea—providing contextual supervision by predicting prompt-induced representation changes—is not limited to graph architectures. To evaluate its cross-architecture applicability, we introduce PREP into VPT [2] on ViT [1] and SVP [4] on Vision Mamba [3].
Specifically, we retain each method's backbone and original prompt structure and construct prompt residuals as the difference between the prompted pathway and its matched reference pathway at the final-layer patch representations. We then use contextual prompt residuals from the student's masked view to predict target prompt residuals from the EMA teacher's full-feature view. The auxiliary prediction loss constrains prompt-induced representation changes and is jointly optimized with the original classification loss.
Transfer of PREP to ViT and Vision Mamba architectures. We report Top-1 classification accuracy (%) on DTD and Dogs.
| Backbone | Method | DTD | Dogs |
|---|---|---|---|
| ViT-S | VPT | 64.6 | 88.6 |
| ViT-S | VPT + PREP (Ours) | 66.2 | 90.2 |
| Vim-S | SVP | 74.0 | 95.0 |
| Vim-S | SVP + PREP (Ours) | 74.6 | 95.3 |
As shown above, adding PREP improves VPT by percentage points on both DTD and Dogs, and SVP by and percentage points, respectively. These results provide initial evidence of PREP's applicability to vision architectures beyond ViG.
References
[1]DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale[C]//International Conference on Learning Representations. 2021.
[2]JIA M, TANG L, CHEN B C, et al. Visual Prompt Tuning[C]//Computer Vision – ECCV 2022. Cham: Springer, 2022: 709–727.
[3]ZHU L, LIAO B, ZHANG Q, et al. Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model[C]//Proceedings of the 41st International Conference on Machine Learning. PMLR, 2024, 235: 62429–62442.
[4]YAO Y, LIU Z, CUI Z, et al. Selective Visual Prompting in Vision Mamba[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2025, 39(21): 22083–22091.
Response to reviewer's comments Part 1
We sincerely thank you for your careful review and recognition of PREP's design, empirical performance, and parameter efficiency. Your suggestions regarding auxiliary supervision, hyperparameter sensitivity, prompt type selection, and cross-architecture applicability provide valuable perspectives for clarifying JEP-VGP's design and applicability. We have carefully considered these suggestions and respond to each below. The revised manuscript has been uploaded.
Comment 1
Response:
Thank you for your insightful suggestions on auxiliary supervision. We understand the importance of distinguishing gains from additional auxiliary supervision itself from those arising from the specific designs of PREP and TGPL. To this end, we introduce MAE-style masked residual reconstruction as a control (B1), using the same Pyramid ViG-M backbone and VGP training framework. This method reconstructs masked-node residuals with a Transformer decoder containing learnable mask tokens, using prompt-induced residuals as reconstruction targets and computing normalized mean squared error only at target locations. The classification pathway remains unchanged. B1 adopts the masked decoding mechanism of MAE [1], but reconstructs feature residuals rather than image pixels.
Classification performance of different auxiliary supervision strategies on three datasets. B1 uses MAE-style masked residual reconstruction, with prompt-induced feature residuals rather than image pixels as reconstruction targets. B2 uses joint prompt-residual prediction, and B3 is the complete JEP-VGP. We report Top-1 classification accuracy (%).
As shown above, both B1 and Joint PREP (B2) outperform VGP, indicating that additional auxiliary supervision can provide performance gains. Compared with B1, the complete JEP-VGP improves accuracy by 0.9, 0.9, and 0.8 percentage points on DTD, Dogs, and Flowers, respectively. Compared with B2, the gains are 2.5, 1.3, and 0.8 percentage points, respectively. These results show that the complete JEP-VGP achieves better classification performance among the compared auxiliary supervision settings, supporting the combined design of PREP and TGPL.
References
[1]HE K, CHEN X, XIE S, et al. Masked Autoencoders Are Scalable Vision Learners[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 16000-16009.