acceptodds
Under review as a conference paper at ICLR 2027

JEP-VGP: Learning Vision Graph Prompts via Joint-Embedding Prediction of Prompt Residuals

Abstract

Vision Graph Neural Networks (ViGs) model irregular semantic relationships in images through dynamic visual graphs, while vision graph prompting enables parameter-efficient adaptation by introducing low-rank Graph, Edge, and Node Prompts with a frozen backbone. However, existing methods mainly rely on image-level classification supervision, providing limited contextual constraints on prompt-induced node representation changes and underutilizing cross-region dependencies in visual graphs. To address this issue, we propose JEP-VGP, a vision graph prompt learning framework inspired by the Joint-Embedding Predictive Architecture (JEPA), which takes prompt residuals as prediction targets to provide explicit contextual guidance. Specifically, we introduce Prompt-Residual Embedding Prediction (PREP), which extracts prompt-induced representation changes by comparing prompted and matched-reference representations under the same configuration, and predicts target residuals from contextual residuals through a lightweight predictor with a slowly updated prompt teacher. This transforms cross-region dependencies into fine-grained supervision for prompt adaptation. Furthermore, we develop Type-Guided Prompt Learning (TGPL) to impose type-specific residual prediction constraints on different prompt pathways through cyclic mode selection and type-dependent weighting, while preserving joint prompt optimization in classification learning. During training, the ViG backbone remains frozen, and only lightweight adaptation components are optimized; auxiliary components are removed during inference without introducing additional inference-time auxiliary computation. Experiments on 10 visual classification benchmarks show that JEP-VGP improves the average accuracy of VGP by 1.5 percentage points under the same experimental settings.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Vision Graph Neural Networks, Vision Graph Prompting, Parameter-Efficient Fine-Tuning, Joint-Embedding Prediction, Prompt Residual Prediction, Type-Guided Prompt Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_bLnDOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes JEP-VGP, a parameter-efficient adaptation framework for pretrained Vision Graph Neural Networks (ViGs), which addresses the limitation of underutilizing cross-region dependencies and contextual constraints in existing methods. Inspired by Joint-Embedding Predictive Architectures (JEPA), JEP-VGP uses contextual prediction to provide direct supervision for prompt-induced changes in node representations, complementing conventional image-level classification supervision. Experiments and analyses demonstrate the effectiveness of JEP-VGP and its components, including PREP and TGPL.

Critical strengths
  1. The paper introduces PREP, which predicts prompt-induced node representation residuals from contextual residuals, extending joint-embedding prediction to vision graph prompt learning in a targeted way. This provides a principled auxiliary objective beyond conventional classification supervision and is supported by ablations showing consistent improvements over the VGP baseline.

  2. On 10 classification benchmarks, JEP-VGP improves average accuracy from 89.6% to 91.1%, outperforming full fine-tuning on 9 of 10 datasets while using 94.6% fewer trainable parameters. The auxiliary prediction components are removed at inference, keeping inference FLOPs unchanged relative to VGP. These results support the method’s practical value for parameter-efficient adaptation.

Critical weaknesses
  1. Although the ablations show improvements from PREP and TGPL, the evaluation does not sufficiently establish whether the gains arise specifically from predicting prompt-induced residuals and type-guided learning, rather than from the additional auxiliary training supervision itself. Stronger comparisons against alternative self-supervised or contextual prediction objectives would help substantiate the methodological novelty.
Questions for authors
  1. How sensitive is JEP-VGP’s performance to the choice of the EMA teacher update rate and the masking ratio? Do the reported experiments establish that the method performs robustly across these settings?

  2. How does cyclically selecting prompt types for auxiliary prediction compare with jointly optimizing all prompt types under an equivalent training budget? This comparison would help clarify whether cyclic type selection is essential to the reported improvements.

  3. Could the proposed method benefit other vision architectures beyond ViGs, and what adaptations would be required to generalize it to those architectures?

5 comments
AuthorsOpenReview ↗

Response to reviewer's comments Part 1

We sincerely thank you for your careful review and recognition of PREP's design, empirical performance, and parameter efficiency. Your suggestions regarding auxiliary supervision, hyperparameter sensitivity, prompt type selection, and cross-architecture applicability provide valuable perspectives for clarifying JEP-VGP's design and applicability. We have carefully considered these suggestions and respond to each below. The revised manuscript has been uploaded.

Comment 1

Although the ablations show improvements from PREP and TGPL, the evaluation does not sufficiently establish whether the gains arise specifically from predicting prompt-induced residuals and type-guided learning, rather than from the additional auxiliary training supervision itself. Stronger comparisons against alternative self-supervised or contextual prediction objectives would help substantiate the methodological novelty.

Response:

Thank you for your insightful suggestions on auxiliary supervision. We understand the importance of distinguishing gains from additional auxiliary supervision itself from those arising from the specific designs of PREP and TGPL. To this end, we introduce MAE-style masked residual reconstruction as a control (B1), using the same Pyramid ViG-M backbone and VGP training framework. This method reconstructs masked-node residuals with a Transformer decoder containing learnable mask tokens, using prompt-induced residuals as reconstruction targets and computing normalized mean squared error only at target locations. The classification pathway remains unchanged. B1 adopts the masked decoding mechanism of MAE [1], but reconstructs feature residuals rather than image pixels.

Classification performance of different auxiliary supervision strategies on three datasets. B1 uses MAE-style masked residual reconstruction, with prompt-induced feature residuals rather than image pixels as reconstruction targets. B2 uses joint prompt-residual prediction, and B3 is the complete JEP-VGP. We report Top-1 classification accuracy (%).

SettingMethodDTDDogsFlowers
B0VGP74.881.798.2
B1VGP + MAE-style Residual78.082.798.6
B2VGP + PREP (Joint)76.482.398.6
B3JEP-VGP (Ours)78.983.699.4

As shown above, both B1 and Joint PREP (B2) outperform VGP, indicating that additional auxiliary supervision can provide performance gains. Compared with B1, the complete JEP-VGP improves accuracy by 0.90.9, 0.90.9, and 0.80.8 percentage points on DTD, Dogs, and Flowers, respectively. Compared with B2, the gains are 2.52.5, 1.31.3, and 0.80.8 percentage points, respectively. These results show that the complete JEP-VGP achieves better classification performance among the compared auxiliary supervision settings, supporting the combined design of PREP and TGPL.

References

[1]HE K, CHEN X, XIE S, et al. Masked Autoencoders Are Scalable Vision Learners[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 16000-16009.

AuthorsOpenReview ↗

Response to reviewer's comments Part 2

Comment 2 Questions 1

How sensitive is JEP-VGP’s performance to the choice of the EMA teacher update rate and the masking ratio? Do the reported experiments establish that the method performs robustly across these settings?

Response:

Thank you for your question about the EMA update rate and masking ratio. To further evaluate JEP-VGP's robustness to these hyperparameters, we add sensitivity experiments on CUB.

Sensitivity to the EMA decay coefficient μ\mu and target-block area-ratio interval [ρmin⁡,ρmax⁡][\rho_{\min},\rho_{\max}] on CUB. Only one hyperparameter is varied at a time; all other settings are held fixed. We report Top-1 accuracy (%), with default settings in bold.

μ\muTop-1 (%)[ρmin⁡,ρmax⁡][\rho_{\min},\rho_{\max}]Top-1 (%)
0.990.9989.4[0.025,0.05][0.025,0.05]88.9
0.9960.99689.5[0.05,0.10][0.05,0.10]89.3
0.999\mathbf{0.999}89.6[0.10,0.20]\mathbf{[0.10,0.20]}89.6
0.99950.999589.5[0.15,0.30][0.15,0.30]89.3

As shown above, varying the EMA decay coefficient μ\mu from 0.990.99 to 0.99950.9995 changes Top-1 accuracy by only 0.20.2 percentage points, indicating stability with respect to the EMA update rate. In comparison, the maximum performance difference across the tested target-block area-ratio sampling intervals is 0.70.7 percentage points, with the default interval [0.10,0.20][0.10,0.20] performing best. All tested configurations outperform VGP's 87.487.4\\%, indicating that JEP-VGP maintains a consistent performance advantage over the examined hyperparameter ranges.

Comment 3 Questions 2

How does cyclically selecting prompt types for auxiliary prediction compare with jointly optimizing all prompt types under an equivalent training budget? This comparison would help clarify whether cyclic type selection is essential to the reported improvements.

Response:

Thank you for your suggestions on prompt type selection. To further analyze cyclic selection, we compare mixed-residual prediction (Joint) with type-specific cyclic prediction (Cyclic). As shown below, both settings activate all prompts in the classification branch and compute one residual prediction target per auxiliary update. Joint uses all three prompt types simultaneously to construct mixed residuals, whereas Cyclic selects a single prompt type for residual prediction according to Eq. (23).

Comparison between mixed-residual prediction and type-specific cyclic prediction. Both settings activate Node, Edge, and Graph Prompts in the classification branch and compute one residual prediction target per auxiliary update. We report Top-1 accuracy (%).

SettingAuxiliary predictionDTDCUBDogsFlowers
A1Joint (mixed residual)76.488.782.398.6
A6Cyclic (type-specific residual)78.989.683.699.4

Compared with Joint, Cyclic improves accuracy by 2.52.5, 0.90.9, 1.31.3, and 0.80.8 percentage points on DTD, CUB, Dogs, and Flowers, respectively. These results indicate that type-specific cyclic prediction achieves better classification performance under the current training settings, supporting TGPL's cyclic selection strategy.

AuthorsOpenReview ↗

Response to reviewer's comments Part 3

Comment 4 Questions 3

Could the proposed method benefit other vision architectures beyond ViGs, and what adaptations would be required to generalize it to those architectures?

Response:

Thank you for your suggestions on PREP's cross-architecture applicability, which is also of great interest to us. Although JEP-VGP's type-guided mechanism depends on ViG's graph structure, PREP's core idea—providing contextual supervision by predicting prompt-induced representation changes—is not limited to graph architectures. To evaluate its cross-architecture applicability, we introduce PREP into VPT [2] on ViT [1] and SVP [4] on Vision Mamba [3].

Specifically, we retain each method's backbone and original prompt structure and construct prompt residuals as the difference between the prompted pathway and its matched reference pathway at the final-layer patch representations. We then use contextual prompt residuals from the student's masked view to predict target prompt residuals from the EMA teacher's full-feature view. The auxiliary prediction loss constrains prompt-induced representation changes and is jointly optimized with the original classification loss.

Transfer of PREP to ViT and Vision Mamba architectures. We report Top-1 classification accuracy (%) on DTD and Dogs.

BackboneMethodDTDDogs
ViT-SVPT64.688.6
ViT-SVPT + PREP (Ours)66.290.2
Vim-SSVP74.095.0
Vim-SSVP + PREP (Ours)74.695.3

As shown above, adding PREP improves VPT by 1.61.6 percentage points on both DTD and Dogs, and SVP by 0.60.6 and 0.30.3 percentage points, respectively. These results provide initial evidence of PREP's applicability to vision architectures beyond ViG.

References

[1]DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale[C]//International Conference on Learning Representations. 2021.

[2]JIA M, TANG L, CHEN B C, et al. Visual Prompt Tuning[C]//Computer Vision – ECCV 2022. Cham: Springer, 2022: 709–727.

[3]ZHU L, LIAO B, ZHANG Q, et al. Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model[C]//Proceedings of the 41st International Conference on Machine Learning. PMLR, 2024, 235: 62429–62442.

[4]YAO Y, LIU Z, CUI Z, et al. Selective Visual Prompting in Vision Mamba[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2025, 39(21): 22083–22091.

Reviewer_JicROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Inspired by the Joint-Embedding Predictive Architecture (JEPA), this paper proposes JEP-VGP, whose core component PREP (Prompt-Residual Embedding Prediction) explicitly defines prompt residuals and trains a lightweight predictor to predict target residuals at masked node regions from visible contextual residuals. A second component, TGPL, decomposes the auxiliary objective by prompt pathway through cyclic mode selection and type-dependent weights, jointly optimized with the classification loss under delayed activation and linear ramp-up. Evaluated on 10 classification benchmarks with a frozen Pyramid ViG-M, JEP-VGP improves average accuracy over VGP from 89.6% to 91.1%.

Critical strengths

-Defining prompt-induced representation changes as explicit prediction targets is conceptually elegant. The paper also correctly notes that its auxiliary components are removed at inference, so the inference pathway is unchanged.

-Beyond the main table, this work also includes more ablation studies to better understand the method design and the contributions of each component. The consistent sign of the gains across all 10 datasets supports the general claim.

Critical weaknesses

-The design of using prompt residuals for extra supervision is interesting. However, the motivation of leveraging prompt residuals has not been clearly mentioned. In addition, the reason why prompt residuals are beneficial to this task is not clear.

-The central claim that prediction "transforms cross-region dependencies into fine-grained supervision" is never directly evidenced: no visualization of learned residuals, no correlation between prediction-loss decrease and classification gain, no per-region analysis showing that the predictor's targets correspond to semantically meaningful cross-region dependencies.

-Each auxiliary update requires up to four full backbone forward passes (student masked prompted + student masked reference + teacher full prompted + teacher full reference), an EMA teacher, and the predictor. The paper reports only that inference overhead is zero, but does not mention the training burden. A fair comparison should consider both.

-For Node and Edge modes, the reference removes only the prompt adjustment term while keeping mixing coefficients; for Graph mode, the reference retains the same number of virtual nodes with zero features. Zeroing learnable virtual-node coefficients makes the Graph residual capture parameter-removal effects rather than the same "adjustment-off" semantics as the other modes, so the three residuals are not comparable under a single definition. This should be justified or unified.

-From Table 2, the performance improvements mainly come from A2 and A3, making PREP-Graph less significant. Additional, ablation study on two datasets is not enough. It is better to show the results on at least 4 datasets. In Table 3, the hyperparameters are manually set. Is it possible to learn an optimized combination?

Questions for authors

-The residual-based supervision design is interesting, but the paper does not clearly state why prompt residuals are the right prediction target, nor why predicting them should benefit prompt adaptation — please provide an explicit motivation and supporting analysis (e.g., ablating residual targets against direct prediction of prompted representations).

-The claim that prediction "transforms cross-region dependencies into fine-grained supervision" is not directly evidenced: there are no visualizations of learned residuals, no analysis correlating prediction-loss decrease with classification gain, and no per-region study showing the predictor's targets correspond to semantically meaningful cross-region dependencies.

-Each auxiliary update requires up to four full backbone forward passes (student masked prompted/reference + teacher full prompted/reference) plus the EMA teacher and predictor, yet only the zero inference overhead is reported. A fair comparison with VGP should quantify the training burden as well (wall-clock, memory, FLOPs per epoch).

-The reference pathway is defined differently across modes: Node/Edge remove only the adjustment term, while Graph retains virtual nodes with zeroed features — making the Graph residual capture parameter-removal effects rather than the same "adjustment-off" semantics. This should be justified or unified so the three residuals are comparable.

-Table 2 shows the gains come mainly from A2/A3, leaving PREP-Graph's contribution unclear, and ablating on only two datasets (CUB, Flowers) is insufficient — please extend to at least four datasets. Additionally, since the Table 3 hyperparameters are manually set, is it possible to learn an optimized configuration (e.g., via automatic hyperparameter search or end-to-end learnable weights)?

8 comments
AuthorsOpenReview ↗

Response to reviewer's comments Part 1

We sincerely thank you for your careful review and recognition of our prompt-residual prediction design and empirical results. Your comments provide valuable perspectives for clarifying JEP-VGP's motivation, mechanisms, and empirical performance. We have incorporated these suggestions and respond to each below. The revised manuscript has been uploaded.

Comment 1 / Questions 1

The residual-based supervision design is interesting, but the paper does not clearly state why prompt residuals are the right prediction target, nor why predicting them should benefit prompt adaptation — please provide an explicit motivation and supporting analysis (e.g., ablating residual targets against direct prediction of prompted representations).

Response:

Thank you for your suggestions on the prompt-residual prediction design. We understand your concern. PREP uses prompt residuals as prediction targets because complete prompted node representations contain both existing features of the pretrained backbone and prompt-induced representation changes. Directly predicting complete representations makes it difficult to impose targeted constraints on prompt-induced changes. Therefore, PREP defines prompt residuals as the representation difference between the prompted pathway and its matched reference pathway under the same prompt mode (Eq. 9). This shifts auxiliary supervision from predicting complete node representations to predicting prompt-induced changes, explicitly focusing it on prompt-induced node representation changes.

Specifically, PREP uses contextual residuals from the masked view to predict target residuals generated by the EMA teacher from the full-feature view (Eqs. 12–17). The prediction loss updates learnable prompt parameters through the student prompted pathway, imposing contextual prediction constraints on prompt-induced node representation changes to complement image-level classification supervision. We have clarified the motivation for prompt-residual prediction and its constraints on prompt adaptation in Section 2.2 of the revised manuscript (pp. 5–6), and added a comparison and analysis of the prediction formulations in Appendix C.3 (p. 16, Table 7).

To evaluate the effectiveness of prompt-residual prediction, we compare different auxiliary prediction formulations on four datasets.

Ablation of auxiliary prediction formulations (Top-1 accuracy, %). The two auxiliary formulations differ in both predictor inputs and target representations; all other settings are held fixed, including the backbone, prompt configuration, masking strategy, predictor architecture, TGPL, and training strategy. VGP uses no auxiliary prediction supervision.

Prediction FormulationDTD ↑\uparrowCUB ↑\uparrowDogs ↑\uparrowFlowers ↑\uparrow
None (VGP)74.887.481.798.2
Prompted Representation76.988.382.298.5
Prompt Residual (JEP-VGP)78.989.683.699.4

As shown above, both complete prompted-representation prediction and prompt-residual prediction outperform VGP, indicating that both auxiliary prediction formulations improve classification performance. Prompt-residual prediction further improves accuracy by 2.02.0, 1.31.3, 1.41.4, and 0.90.9 percentage points on DTD, CUB, Dogs, and Flowers, respectively, with an average gain of 1.41.4 percentage points. These results show that, under the same backbone and training settings, using prompt residuals as both predictor inputs and prediction targets achieves better downstream classification performance, supporting PREP's residual prediction design.

AuthorsOpenReview ↗

Response to reviewer's comments Part 2

Comment 2 / Questions 2

The claim that prediction "transforms cross-region dependencies into fine-grained supervision" is not directly evidenced: there are no visualizations of learned residuals, no analysis correlating prediction-loss decrease with classification gain, and no per-region study showing the predictor's targets correspond to semantically meaningful cross-region dependencies.

Response:

Thank you for your suggestions on representation visualization. We randomly select six test images each from Flowers and Dogs and add prompt-residual and Grad-CAM visualizations to analyze prompt-induced node representation changes and differences in classification responses, as shown in Figure 2 in Appendix C.2 of the revised manuscript (p. 15). Regarding residual distributions, Node Prompts mainly exhibit localized block-like or discrete node responses. Edge Prompts show responses at multiple locations in some samples, consistent with their neighborhood information aggregation mechanism. Graph Prompts exhibit regional responses of varying spatial extent, which are related to virtual-node interactions (Eqs. 3–5). Further Grad-CAM comparisons show that all three prompt adjustments can change the classification response distributions relative to their reference pathways, with high-response regions becoming more concentrated on class-discriminative object parts in some samples. For example, responses on Flowers focus more on flower centers and surrounding petals, while those on Dogs emphasize key regions such as the head and face. These results visually illustrate node representation changes induced by different prompt modes and the associated adjustments in class-related responses. The context intervention and visualization analyses have been added to Appendix C.1 (p. 14, Table 6) and Appendix C.2 (pp. 14–15; Figure 2 is on p. 15) of the revised manuscript, respectively.

To examine the relationship between decreasing prediction loss and classification performance, we further analyze training trajectories on five datasets.

Training-trajectory correlation between prediction loss and contemporaneous Top-1 classification accuracy on five datasets. Spearman ρ\rho denotes their rank correlation coefficient, and pp-value is the pp value of the correlation test. Reported Top-1 is the best classification accuracy (%) reported in the paper; correlations are computed from the selected training trajectories.

DatasetSpearman ρ\rhopp-valueReported Top-1 (%)
CIFAR100−0.941-0.9414.62×10−434.62\times10^{-43}90.3
DTD−0.696-0.6962.60×10−142.60\times10^{-14}78.9
Flowers−0.881-0.8812.56×10−302.56\times10^{-30}99.4
NABirds−0.929-0.9297.93×10−407.93\times10^{-40}84.7
Dogs−0.773-0.7734.63×10−194.63\times10^{-19}83.6

As shown above, prediction loss is significantly negatively correlated with contemporaneous Top-1 accuracy on CIFAR100, DTD, Flowers, NABirds, and Dogs. The Spearman correlation coefficients are −0.941-0.941, −0.696-0.696, −0.881-0.881, −0.929-0.929, and −0.773-0.773, respectively, with all pp values below 10−1310^{-13}. These results indicate that lower prediction loss generally corresponds to higher classification accuracy during training across these classification datasets, providing correlational evidence for the relationship between residual prediction supervision and downstream classification performance.

To examine the contribution of cross-region contextual information, we conduct a paired experiment on all 5,794 CUB test images.

Cosine error of target-residual prediction on all 5,794 CUB test images, defined as Ecos=1−cos⁡(Δ^im,Δf,im)∈[0,2]E_{\mathrm{cos}}=1-\cos(\widehat{\Delta}_i^m,\Delta_{f,i}^m)\in[0,2], where Δ^im\widehat{\Delta}_i^m and Δf,im\Delta_{f,i}^m denote the predicted residual and teacher target residual, respectively. Results are the mean ±\pm standard deviation across test images; lower is better.

MethodCosine error ↓\downarrow
Trained predictor0.4169±0.1097\boldsymbol{0.4169 \pm 0.1097}
Trained (shuffled context)0.6440±0.10910.6440 \pm 0.1091
Random predictor1.0067±0.02181.0067 \pm 0.0218

As shown above, the target-residual cosine error with the original context is 0.4169±0.10970.4169\pm0.1097. Keeping the predictor, teacher targets, and target locations fixed and shuffling only the spatial arrangement of contextual residuals at non-target locations increases the error to 0.6440±0.10910.6440\pm0.1091, a relative increase of 54.4754.47\\%. This result demonstrates that both contextual residual content and spatial arrangement contribute to target-residual prediction, supporting PREP's use of contextual information for node-level supervision.

AuthorsOpenReview ↗

Response to reviewer's comments Part 3

Comment 3 / Questions 3

Each auxiliary update requires up to four full backbone forward passes (student masked prompted/reference + teacher full prompted/reference) plus the EMA teacher and predictor, yet only the zero inference overhead is reported. A fair comparison with VGP should quantify the training burden as well (wall-clock, memory, FLOPs per epoch).

Response:

Thank you for highlighting the training overhead and helping us improve this evaluation. PREP's auxiliary update involves four forward computation pathways. Only the student prompted pathway requires backpropagation to optimize prompt parameters within the backbone; the reference and teacher pathways do not retain backward computation graphs. To comprehensively evaluate the computational overhead, we benchmark efficiency on five datasets using an RTX 3090 with a batch size of 32.

Training cost, inference efficiency, and classification accuracy of VGP and JEP-VGP across five datasets. Training time is measured over 100 epochs, FLOPs per epoch, and memory by peak GPU usage.

DatasetMethodTop-1 (%)Trainable Params (M)100-Epoch Train Time (h)Train FLOPs (PF/epoch)Peak Train Mem. (GiB)Inference (ms/img)Infer. FLOPs (G/img)Infer. Mem. (GiB)
CIFAR100VGP89.72.5310.321.147011.293.2759.221.441
JEP-VGP90.32.5517.222.603417.163.2789.221.441
DTDVGP74.82.470.380.043111.293.2749.221.441
JEP-VGP78.92.500.650.098417.163.2809.221.441
FlowersVGP98.22.530.220.023411.293.2749.221.441
JEP-VGP99.42.560.350.053417.163.2829.221.441
NABirdsVGP80.93.464.930.549011.313.2779.221.445
JEP-VGP84.73.498.221.246217.183.2799.221.445
DogsVGP81.72.552.470.275311.293.2759.221.441
JEP-VGP83.62.574.130.624617.163.2819.221.441

As shown above, JEP-VGP requires approximately 1.67×1.67\times the 100-epoch training time, 2.27×2.27\times the training FLOPs per epoch, and 1.52×1.52\times the peak GPU memory usage for training of VGP.

This comparison of training and inference costs, together with the accompanying analysis, has been added to Appendix B.1 of the revised manuscript (p. 13, Table 5).

The additional training computation in JEP-VGP is used for contextual prediction of prompt residuals, supplementing the original image-level classification supervision with guidance targeted at prompt-induced representation changes. With the same backbone and number of training epochs, JEP-VGP improves average Top-1 accuracy by 2.322.32 percentage points across the five datasets, while adding only 0.020.02–0.030.03 M trainable parameters. Importantly, the additional prediction modules are used only during training. Inference FLOPs and GPU memory usage remain essentially the same as VGP, with latency differences below 0.30.3\\%. Thus, JEP-VGP obtains better downstream classification performance through additional training computation while retaining the original inference efficiency.

We also agree that reducing PREP's training overhead is a worthwhile direction. In future work, we will explore reusing intermediate student and teacher features, reducing redundant computation in matched reference pathways, and optimizing auxiliary prediction frequency to lower computational and memory costs while preserving the benefits of residual prediction supervision.

Reviewer_wsYnOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JEP-VGP, a parameter-efficient adaptation framework for Vision Graph Neural Networks. The key component, Prompt-Residual Embedding Prediction (PREP), constructs prompt residuals by comparing prompted and matched-reference node representations and predicts target residuals from masked contextual residuals using an EMA teacher. Type-Guided Prompt Learning (TGPL) further applies this objective to Node, Edge, and Graph Prompt pathways through cyclic mode selection. Experiments on ten image classification benchmarks show that JEP-VGP improves the average accuracy of VGP by 1.5 percentage points while introducing few additional parameters and no additional inference-time computation. Treating prompt-residual prediction as an auxiliary self-supervised regularization objective is an interesting direction for parameter-efficient adaptation. The main weakness is that the paper does not provide sufficient analysis to explain why this objective improves contextual modeling or how it changes the learned representations.

Critical strengths
  1. The paper introduces a meaningful extension to vision graph prompting (VGP) by directly supervising prompt-induced representation changes rather than relying exclusively on image-level classification supervision.
  2. The empirical results are consistently positive. JEP-VGP improves over VGP across all ten datasets and achieves an average improvement of 1.5 percentage points.
  3. The idea of treating prompt-residual prediction as an auxiliary self-supervised regularization objective is interesting for parameter-efficient adaptation.
Critical weaknesses
  1. The connection between prompt-residual prediction and contextual understanding remains insufficiently explained. PREP can be interpreted as introducing a self-supervised or self-distillation regularizer during downstream adaptation. However, it is unclear why predicting teacher-generated prompt residuals necessarily improves the model’s sensitivity to cross-region context. The paper would benefit from a more precise explanation of this mechanism and experiments that directly verify the contribution of contextual information.
  2. The effect on learned representations is not analyzed. The central claim is that JEP-VGP improves the semantic representation of prompts and image patch tokens, but the paper only reports downstream classification accuracy. Visualization or statistical analysis of the learned embeddings would make this claim more convincing. Examples include feature-space visualization, class separability measurements, prompt-token alignment, attention or graph-neighborhood analysis, and comparisons of intra-class and inter-class distances before and after applying JEP-VGP.
  3. The diversity of downstream tasks is limited. Although the paper evaluates ten datasets, they are all image classification benchmarks using the same Pyramid ViG-M backbone. Additional experiments on other adaptation tasks, such as object detection or semantic segmentation, would better demonstrate the generality of the proposed framework.
Questions for authors

My questions and concerns have already been presented in the Critical Weaknesses section.

10 comments
AuthorsOpenReview ↗

Response to reviewer's comments Part 1

We sincerely thank you for your careful review and positive assessment of our research motivation and empirical results. Your suggestions regarding cross-region contextual modeling, representation analysis, and downstream task diversity have helped us improve the paper. We have conducted additional experiments and analyses to address these concerns and respond to each below. The revised manuscript has been uploaded.

Comment 1

The connection between prompt-residual prediction and contextual understanding remains insufficiently explained. PREP can be interpreted as introducing a self-supervised or self-distillation regularizer during downstream adaptation. However, it is unclear why predicting teacher-generated prompt residuals necessarily improves the model’s sensitivity to cross-region context. The paper would benefit from a more precise explanation of this mechanism and experiments that directly verify the contribution of contextual information.

Response:

Thank you for your suggestions on PREP's contextual modeling mechanism. We agree that PREP can be interpreted as a form of self-supervised regularization. Its contextual constraint, however, arises not only from teacher target residuals but also from the prediction task jointly constructed by input-feature masking and target-residual masking. Under the same prompt mode, PREP defines prompt residuals as the representation difference between the prompted pathway and its matched reference pathway, focusing the prediction target on prompt-induced node representation changes (Eq. 9).

Specifically, ViG models relationships between different image regions through dynamic graph construction and graph convolution. Prompt residuals can therefore include representation changes caused by prompt adjustments through cross-region graph interactions. The student receives a masked view that retains only the initial features of visible context nodes, while the teacher generates target residuals from the full-feature view, with its prompt parameters updated by EMA (Eqs. 12–15). After graph propagation, the student residuals at target locations are further replaced with a shared learnable mask vector (Eq. 16). This prevents the predictor from directly accessing target residuals and encourages prediction from contextual residuals at non-target locations. The prediction loss updates learnable prompt parameters through the student prompted pathway, imposing contextual prediction constraints on prompt-induced node representation changes to complement image-level classification supervision. We have clarified the motivation for residual prediction and the contextual constraint mechanism in Section 2.2 of the revised manuscript (pp. 5–6, around Eq. 9 and Eqs. 16–17), and added the context intervention experiment in Appendix C.1 (p. 14, Table 6).

To examine whether target-residual prediction depends on context, we conduct a controlled experiment on all 5,794 CUB test images.

Cosine error of target-residual prediction on all 5,794 CUB test images, defined as Ecos=1−cos⁡(Δ^im,Δf,im)∈[0,2]E_{\mathrm{cos}}=1-\cos(\widehat{\Delta}_i^m,\Delta_{f,i}^m)\in[0,2], where Δ^im\widehat{\Delta}_i^m and Δf,im\Delta_{f,i}^m denote the predicted residual and teacher target residual, respectively. Results are the mean ±\pm standard deviation across test images; lower is better.

MethodCosine error ↓\downarrow
Trained predictor0.4169±0.1097\boldsymbol{0.4169 \pm 0.1097}
Trained (shuffled context)0.6440±0.10910.6440 \pm 0.1091
Random predictor1.0067±0.02181.0067 \pm 0.0218

As shown above, the trained predictor achieves a cosine error of 0.41690.4169 with the original context, compared with 1.00671.0067 for a randomly initialized predictor. Keeping the trained model, input-feature masks, teacher targets, and target locations fixed, we randomly permute complete residual vectors among non-target locations within each image. The error increases to 0.64400.6440, a relative increase of approximately 54.4754.47\\%. This result indicates that the predictor uses the spatial arrangement of contextual residuals, providing experimental support for PREP's contextual prediction constraint.

AuthorsOpenReview ↗

Response to reviewer's comments Part 2

Comment 2

The effect on learned representations is not analyzed. The central claim is that JEP-VGP improves the semantic representation of prompts and image patch tokens, but the paper only reports downstream classification accuracy. Visualization or statistical analysis of the learned embeddings would make this claim more convincing. Examples include feature-space visualization, class separability measurements, prompt-token alignment, attention or graph-neighborhood analysis, and comparisons of intra-class and inter-class distances before and after applying JEP-VGP.

Response:

Thank you for your suggestions on representation analysis. We have added prompt-residual and Grad-CAM visualizations to examine prompt-induced representation changes and differences in classification responses. Specifically, we randomly select six test images each from Flowers and Dogs, visualize their prompt residuals, and compare Grad-CAM responses between the matched reference and prompted pathways, as shown in Figure 2 in Appendix C.2 of the revised manuscript (p. 15).

Regarding residual distributions, Node Prompts mainly exhibit localized block-like or discrete node responses. Edge Prompts show responses at multiple locations in some samples, consistent with their neighborhood information aggregation mechanism. Graph Prompts exhibit regional responses of varying spatial extent, which are related to virtual-node interactions (Eqs. 3–5). Further Grad-CAM comparisons show that all three prompt adjustments can improve the classification response distributions relative to their reference pathways, with high-response regions becoming more concentrated on class-discriminative object parts in some samples. For example, responses on Flowers focus more on flower centers and surrounding petals, while those on Dogs emphasize key regions such as the head and face. These results visually illustrate node representation changes induced by different prompt modes and the associated adjustments in class-related responses.

These visualizations and qualitative analyses have been added to Appendix C.2 of the revised manuscript (pp. 14–15; Figure 2 is on p. 15).

Comment 3

The diversity of downstream tasks is limited. Although the paper evaluates ten datasets, they are all image classification benchmarks using the same Pyramid ViG-M backbone. Additional experiments on other adaptation tasks, such as object detection or semantic segmentation, would better demonstrate the generality of the proposed framework.

Response:

To evaluate the applicability of our method to different downstream vision tasks, we have added semantic segmentation and object detection experiments on PASCAL VOC 2012 [1]. Object detection uses Faster R-CNN [3] with FPN [2], while semantic segmentation uses a hybrid head combining a pyramid pooling module [4] with a Mask2Former [5] query decoder.

Semantic segmentation and object detection performance of ViG, VGP, and JEP-VGP on PASCAL VOC 2012. We report segmentation mIoU and detection AP$_{50}$ (%), respectively; higher is better.

TaskMetricViGVGPJEP-VGP (Ours)
Semantic SegmentationmIoU (%)78.3677.9879.70
Object DetectionAP$_{50}$ (%)49.7846.7651.68

As shown above, JEP-VGP improves segmentation mIoU and detection AP$_{50}$ over VGP by 1.721.72 and 4.924.92 percentage points, respectively. Compared with fully fine-tuned ViG, the improvements are 1.341.34 and 1.901.90 percentage points, respectively. These results further demonstrate the effectiveness of our method across different downstream vision tasks.

References

[1]EVERINGHAM M, ESLAMI S M A, VAN GOOL L, et al. The Pascal Visual Object Classes Challenge: A Retrospective[J]. International Journal of Computer Vision, 2015, 111(1): 98–136.

[2]LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature Pyramid Networks for Object Detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 2117–2125.

[3]REN S, HE K, GIRSHICK R, et al. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks[C]//Advances in Neural Information Processing Systems. 2015, 28: 91–99.

[4]ZHAO H, SHI J, QI X, et al. Pyramid Scene Parsing Network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 2881–2890.

[5]CHENG B, MISRA I, SCHWING A G, et al. Masked-Attention Mask Transformer for Universal Image Segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 1290–1299.

Two concerns remain regarding the interpretation of the new results.

  1. The downstream results are encouraging, but the substantial detection gain requires further clarification. JEP-VGP improves over VGP by 1.72 percentage points in segmentation mIoU and 4.92 points in detection AP. However, VGP itself falls 3.02 AP points below fully fine-tuned ViG, whereas JEP-VGP exceeds ViG by 1.90 points. Could the authors explain why VGP underperforms substantially in detection, and establish that the improvement reflects the contribution of PREP under a well-controlled comparison? Please specify the dataset splits, exact AP definition, initialization, training schedules, hyperparameter selection, and which backbone, prompt, and task-head parameters are trainable for each method. Results across multiple seeds and a matched VGP versus VGP+PREP ablation would help establish the robustness and attribution of this gain.

  2. The added visualizations do not yet directly address the representation-level concern. Grad-CAM and spatial residual heatmaps show where responses occur on the image grid, but they do not reveal the graph connectivity or cross-region interactions underlying the proposed mechanism. In particular, visually localized or distributed responses alone do not establish that the prompts encode richer relational or semantic information. Moreover, comparisons between prompted and reference pathways do not isolate the effect of PREP. Could the authors directly compare VGP and JEP-VGP under the same prompt configuration, using quantitative representation measures such as class separability or intra-/inter-class distances, together with graph-aware analysis of neighborhood connections or virtual-node interactions? Such evidence would more directly substantiate the claimed improvement in contextual representations than the current qualitative heatmaps.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.