JEP-VGP: Learning Vision Graph Prompts via Joint-Embedding Prediction of Prompt Residuals

Vision Graph Neural Networks (ViGs) model irregular semantic relationships in images through dynamic visual graphs, while vision graph prompting enables parameter-efficient adaptation by introducing low-rank Graph, Edge, and Node Prompts with a frozen backbone. However, existing methods mainly rely on image-level classification supervision, providing limited contextual constraints on prompt-induced node representation changes and underutilizing cross-region dependencies in visual graphs. To address this issue, we propose JEP-VGP, a vision graph prompt learning framework inspired by the Joint-Embedding Predictive Architecture (JEPA), which takes prompt residuals as prediction targets to provide explicit contextual guidance. Specifically, we introduce Prompt-Residual Embedding Prediction (PREP), which extracts prompt-induced representation changes by comparing prompted and matched-reference representations under the same configuration, and predicts target residuals from contextual residuals through a lightweight predictor with a slowly updated prompt teacher. This transforms cross-region dependencies into fine-grained supervision for prompt adaptation. Furthermore, we develop Type-Guided Prompt Learning (TGPL) to impose type-specific residual prediction constraints on different prompt pathways through cyclic mode selection and type-dependent weighting, while preserving joint prompt optimization in classification learning. During training, the ViG backbone remains frozen, and only lightweight adaptation components are optimized; auxiliary components are removed during inference without introducing additional inference-time auxiliary computation. Experiments on 10 visual classification benchmarks show that JEP-VGP improves the average accuracy of VGP by 1.5 percentage points under the same experimental settings.