- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PSGR improves visual graph inputs for image–text retrieval through caption-guided LVLM refinement and RPCS sampling, followed by local matching and GPO-based global matching. It reports strong results on Flickr30K and MS-COCO.
Critical strengths
The method addresses a clear semantic mismatch in graph-based retrieval. The appendix provides substantial analysis and clearly discloses external supervision, transfer settings, and timing scope.
Critical weaknesses
The specific benefit of caption-guided supervision is not sufficiently isolated. Table 3 changes both the graph generator and its supervision, while Table 2 mainly studies subsequent reward choices. Comparing generic graph supervision with caption-guided supervision using the same LVLM, training budget, and graph budget would test the central design more directly.
The connection between reward improvements and the quality of the retained grounded graph remains indirect. Equation 26 rewards both main and extra triplets, although default retrieval uses only main triplets. Multiplying raw label cosines can also reward semantically mismatched combinations. Figure 7 measures repetition and relation composition rather than relation correctness or grounding accuracy, leaving the source of the improvement unclear.
As a secondary point, the choice of 22K SFT and 7K GRPO samples is not justified or tested for allocation sensitivity (Section 3.1 and Appendix D.1). Table 2 compares training strategies rather than data allocations. Please explain the choice and compare a small number of allocations with the total supervision size fixed and compute matched as closely as possible.
Questions for authors
As weaknesses above.
4 comments
Response to Reviewer sxFx (2/3)
[W2] Reward improvements and the retained grounded graph
Concern: Reward improvements may arise from extra triplets or sign cancellation, while the quality of the grounded relations retained for retrieval remains insufficiently measured.
Main and extra reward contributions. We decomposed the cross-modal score on the same 7,000 GRPO training images using their existing caption triplets. For each response, we recomputed the original reward using main candidates alone and using the union of main and extra candidates. Main candidates here are the parsed grounded output before RPCS.
| Model | Main-only reward | Main + extra reward | Extra marginal contribution | Extra wins (%) |
|---|---|---|---|---|
| SFT only | 0.411380 | 0.417786 | 0.006406 | 3.89 |
| GRPO without cross-modal reward | 0.483473 | 0.483562 | 0.000090 | 0.04 |
| Full GRPO | 0.547269 | 0.556449 | 0.009180 | 9.62 |
Full GRPO improves the main-only score from 0.411380 after SFT to 0.547269. Extra candidates add 0.009180, approximately 1.65% of the complete mean score. The score increase is therefore also present in the grounded main output. This decomposition measures caption matching before sampling, while the independent evaluation below measures the final retained graph.
Direct evaluation of retained PSGs. We evaluated 100 randomly sampled Flickr30K test images and their 500 captions. References were constructed using GPT-6 Luna with human feedback, manually checked, and locked without access to evaluated predictions. The reference contains 373 caption-grounded relations. Each method contributes 1,800 predictions under the same 18-relation budget.
Precision and recall measure one-to-one semantic matches to the reference. Grounding accuracy is the proportion of these matches for which both endpoint boxes achieve IoU . Hallucination judgments use image and caption evidence, with uncertain cases recorded separately.
| Graph source | Matched triplets | Caption-grounded precision | Caption-grounded recall | Grounding accuracy | Hallucination count | Hallucination rate | Uncertain count |
|---|---|---|---|---|---|---|---|
| Generic VSG (Neural Motifs) | 122 | 6.78 | 32.71 | 13.11 | 390 | 21.67 | 21 |
| SFT + RPCS | 256 | 14.22 | 68.63 | 16.80 | 187 | 10.39 | 23 |
| GRPO + RPCS | 278 | 15.44 | 74.53 | 19.06 | 145 | 8.06 | 51 |
Compared with SFT + RPCS, GRPO + RPCS increases caption-grounded precision by 1.22 percentage points, recall by 5.90 points, and grounding accuracy among semantic matches by 2.26 points, while reducing the confirmed hallucination rate by 2.33 points. These measurements assess the actual graph inputs used for retrieval, adding direct semantic, grounding, and visual-support evidence to the repetition and relation-composition analysis.
Raw cosine products. We also sampled eight responses for each of the 7,000 images, giving 56,000 outputs per model, and removed positive products formed from two negative label cosines. We then reran content matching and recomputed cross-modal maxima for Full GRPO.
| Diagnostic | Affected / total evaluations | Frequency | Mean reward decrease after removal |
|---|---|---|---|
| Output with a two-negative pair in its final content assignment | 1 / 56,000 outputs | 0.00179% | |
| Caption-triplet score affected in cross-modal matching | 1 / 499,104 caption-triplet evaluations | 0.000200% |
We additionally retrained with label-wise cosine clipping to in both semantic rewards and positive-score filtering for the order reward. Assignments must exceed for order scoring, and fewer than two valid assignments receive zero order reward. Under this combined modification, adding the cross-modal reward still improves ensemble Rsum by 3.60 on Flickr30K, 1.84 on COCO 5-fold 1K, and 4.96 on COCO 5K. The full modified-reward results are 558.54, 540.78, and 461.46, only 0.36, 0.12, and 0.54 below the original results (Appendix A.7.2, Table 18). Together with the measured occurrence frequencies and reward changes, these results suggest that sign cancellation was not a critical factor driving retrieval performance in these experiments.
Extra triplets in the matching input. We focus on graph purification rather than matching-network design, and therefore use SEMScene’s established node-level and graph-level networks as the local and global matchers. We tested extra-triplet integration by applying the same preprocessing and embedding procedure as textual triplets, then concatenating their embeddings with the image-side feature sequence before visual GPO pooling. This reduced Flickr30K ensemble Rsum from 558.9 to 544.5 (Appendix A.8, Table 19). Our default input therefore uses grounded visual triplets, while dedicated integration of text-only extra semantics is left to future work.
Response to Reviewer sxFx (3/3)
[W3] Allocation of SFT and GRPO supervision
Concern: The 22K SFT / 7K GRPO split requires a rationale and an allocation comparison at a fixed total supervision size.
The 22K/7K design assigns most supervision to learning grounded, caption-aligned generation through SFT, while reserving a smaller subset for the more expensive reward-based optimization. SFT establishes the generation content and grounding interface, and GRPO further optimizes semantic fidelity and caption alignment. The 7K GRPO subset is selected by output-length bin sampling, as described in Section 3.1 and Appendix D.1.
Following your suggestion, we added two allocations around the default split, 20K/9K and 24K/5K, keeping the total CG-VSG supervision size fixed at 29,000 samples. We evaluate all three allocations using the same retrieval architecture, DINOv2 features, 36-object/18-relation graph budget, and min-max fusion. Local and Global matchers are trained for 30 epochs and selected by validation performance. The evaluated GRPO checkpoints are step 5,625 for 20K/9K, 4,375 for 22K/7K, and 3,125 for 24K/5K, making the optimization-step counts explicit alongside the fixed sample total.
Appendix A.13.1, Table 24. The manuscript also reports all Local and Global metrics.
Flickr30K 1K
| Branch | SFT / GRPO samples | I2T R@1 | R@5 | R@10 | T2I R@1 | R@5 | R@10 | Rsum |
|---|---|---|---|---|---|---|---|---|
| Local | 20K / 9K | 84.8 | 98.3 | 99.3 | 73.7 | 93.6 | 96.5 | 546.16 |
| Local | 22K / 7K | 85.9 | 98.6 | 99.4 | 73.9 | 93.3 | 96.8 | 547.90 |
| Local | 24K / 5K | 84.1 | 98.3 | 99.4 | 72.6 | 92.6 | 96.1 | 543.08 |
| Global | 20K / 9K | 86.6 | 98.6 | 99.8 | 74.6 | 93.0 | 95.8 | 548.46 |
| Global | 22K / 7K | 87.4 | 98.2 | 99.5 | 74.3 | 93.0 | 96.3 | 548.70 |
| Global | 24K / 5K | 86.6 | 98.8 | 99.6 | 73.3 | 93.0 | 96.1 | 547.42 |
| Ensemble | 20K / 9K | 89.1 | 98.9 | 99.7 | 77.9 | 94.8 | 97.3 | 557.68 |
| Ensemble | 22K / 7K | 90.3 | 99.0 | 99.5 | 78.1 | 94.7 | 97.3 | 558.90 |
| Ensemble | 24K / 5K | 88.3 | 98.6 | 99.9 | 76.4 | 94.5 | 97.3 | 554.92 |
MS-COCO 5-fold 1K
| Branch | SFT / GRPO samples | I2T R@1 | R@5 | R@10 | T2I R@1 | R@5 | R@10 | Rsum |
|---|---|---|---|---|---|---|---|---|
| Local | 20K / 9K | 79.0 | 96.6 | 98.9 | 68.4 | 92.8 | 97.0 | 532.64 |
| Local | 22K / 7K | 79.1 | 96.6 | 99.2 | 68.5 | 92.8 | 97.0 | 533.20 |
| Local | 24K / 5K | 78.7 | 96.8 | 98.9 | 68.1 | 92.5 | 96.8 | 531.70 |
| Global | 20K / 9K | 80.7 | 97.2 | 99.2 | 67.6 | 91.9 | 96.1 | 532.80 |
| Global | 22K / 7K | 81.6 | 97.0 | 99.1 | 67.9 | 92.0 | 96.1 | 533.60 |
| Global | 24K / 5K | 80.0 | 97.0 | 99.2 | 67.4 | 91.8 | 96.3 | 531.76 |
| Ensemble | 20K / 9K | 82.1 | 97.8 | 99.3 | 70.7 | 93.3 | 97.3 | 540.51 |
| Ensemble | 22K / 7K | 82.7 | 97.5 | 99.3 | 71.0 | 93.3 | 97.2 | 540.90 |
| Ensemble | 24K / 5K | 81.8 | 97.4 | 99.1 | 70.4 | 93.2 | 97.1 | 538.98 |
MS-COCO 5K
| Branch | SFT / GRPO samples | I2T R@1 | R@5 | R@10 | T2I R@1 | R@5 | R@10 | Rsum |
|---|---|---|---|---|---|---|---|---|
| Local | 20K / 9K | 57.5 | 85.0 | 92.0 | 47.4 | 75.7 | 85.0 | 442.65 |
| Local | 22K / 7K | 58.1 | 86.1 | 92.8 | 47.4 | 76.0 | 85.0 | 445.30 |
| Local | 24K / 5K | 56.4 | 85.4 | 92.5 | 47.1 | 75.5 | 84.9 | 441.90 |
| Global | 20K / 9K | 60.6 | 86.8 | 93.0 | 46.3 | 75.2 | 84.5 | 446.44 |
| Global | 22K / 7K | 61.3 | 87.1 | 93.4 | 46.9 | 75.5 | 84.7 | 448.90 |
| Global | 24K / 5K | 59.0 | 85.9 | 93.2 | 45.9 | 74.9 | 84.2 | 443.08 |
| Ensemble | 20K / 9K | 63.2 | 87.8 | 94.0 | 50.2 | 78.1 | 86.6 | 459.96 |
| Ensemble | 22K / 7K | 63.9 | 88.3 | 94.4 | 50.7 | 78.0 | 86.7 | 462.00 |
| Ensemble | 24K / 5K | 61.9 | 87.1 | 94.0 | 49.4 | 77.9 | 86.4 | 456.74 |
Among the tested allocations, 22K/7K achieves the highest Rsum for Local, Global, and Ensemble under all three evaluation protocols. Its ensemble gains over 20K/9K are 1.22, 0.39, and 2.04 on Flickr30K, COCO 5-fold 1K, and COCO 5K, respectively. Compared with 24K/5K, the gains are 3.98, 1.92, and 5.26. Flickr30K benefits more strongly in the Local branch, while both COCO branches improve. The 22K/7K split also achieves the highest ensemble R@1 in both retrieval directions across all three protocols.
Individual cutoffs occasionally favor other allocations. For example, 24K/5K achieves higher Flickr30K ensemble I2T R@10, at 99.9% versus 99.5%, but lower Rsum. Overall, moving 2,000 samples from either SFT to GRPO or GRPO to SFT reduces ensemble performance. This pattern is consistent with the complementary roles of supervised initialization and reward-based refinement, with 22K/7K providing the strongest overall performance among the tested splits.
Thanks for the detailed response, and I have no further questions.
Response to Reviewer sxFx (1/3)
Thank you for recognizing the semantic mismatch addressed by PSGR and the detailed analyses in our appendix. We have added experiments and clarified the supervision design in response to your comments.
[W1] Caption-guided supervision and the GRPO ablation
Concern: The benefit of caption guidance should be examined using the same LVLM and comparable training and graph budgets.
Generic supervision in the current input setting. Our LVLM already receives the generic VSG as part of its prompt. Using that same graph as the training target would largely teach Qwen to reproduce information supplied in its input. This would compare graph reproduction with caption-oriented refinement, rather than compare two supervision sets with equivalent refinement quality.
Caption guidance enters at two stages. During CG-VSG construction, GPT-5 uses the image and its human-written captions to generate grounded targets that prioritize the entities and relations emphasized in human descriptions. These targets supervise SFT and provide the reference for the GRPO content-order reward. Caption information therefore enters the training targets before the cross-modal reward is introduced. During GRPO, the cross-modal reward additionally aligns generated triplets with caption-derived scene graphs, explicitly optimizing caption-level semantics.
Removing caption guidance throughout training would consequently require rebuilding the CG-VSG targets without captions and repeating LVLM adaptation and retrieval evaluation. This involves substantial data-construction time and cost. Generating the existing 29,000 Flickr30K targets alone cost over USD 350 in GPT-5 tokens, including failed generations and regeneration, followed by structured-output cleaning and SFT/GRPO training.
Given these practical considerations, we evaluated the additional caption-alignment signal at the GRPO stage. Both variants start from the same caption-guided SFT model, use the same GRPO data and content-order and format rewards, and differ only in the cross-modal reward. The retrieval architecture, DINOv2 features, and 36-object/18-relation graph budget remain unchanged.
Revised manuscript: Section 4.3.1, Table 2. Results below are min-max ensemble recalls and Rsum. Recall values are percentages.
Adding the cross-modal reward improves ensemble Rsum by 5.6 on Flickr30K, 3.9 on COCO 5K, and 0.7 on COCO 5-fold 1K. Without it, GRPO changes Rsum relative to SFT by -1.4, +0.4, and +0.8, respectively. These results show the additional benefit of explicit caption alignment during reinforcement learning, on top of the caption guidance already present in the CG-VSG targets and SFT initialization.
Consistency under modified rewards. The same benefit persists when we jointly apply label-wise cosine clipping and positive-score order filtering. Adding the cross-modal reward then improves ensemble Rsum by 3.60 on Flickr30K, 4.96 on COCO 5K, and 1.84 on COCO 5-fold 1K (Appendix A.7.2, Table 18). Explicit caption alignment therefore improves retrieval under both reward implementations.
Direct graph-quality evidence. We also evaluate the resulting graphs on 100 randomly sampled Flickr30K test images using independently constructed, manually checked caption-grounded references and the same 18-relation budget (Section 4.4, Table 4, and Appendix A.5). Before GRPO, SFT + RPCS already increases caption-grounded precision from 6.78% to 14.22% and recall from 32.71% to 68.63%, compared with generic VSGs. Grounding accuracy among semantic matches increases from 13.11% to 16.80%, while the confirmed hallucination rate decreases from 21.67% to 10.39%. Full GRPO further reaches 15.44% precision, 74.53% recall, and 19.06% grounding accuracy, with an 8.06% hallucination rate.
Tables 2 and 18 assess the explicit caption-alignment reward during GRPO, while Table 4 measures caption coverage, grounding, and unsupported predictions in the retained graphs. The supervision-construction and adaptation stages are described in Section 3.1 and Appendix D.