acceptodds
Under review as a conference paper at ICLR 2027

Purifying Generic Visual Scene Graph to Enable Caption-Aligned Image-Text Retrieval

Abstract

Scene graph matching methods for image-text retrieval represent images and captions with structured entities and relations, enabling fine-grained cross-modal matching beyond isolated region-word alignment. However, existing methods typically rely on generic scene graph generators, which produce Visual Scene Graphs (VSGs) containing caption-irrelevant semantics. This creates a semantic gap with the sparse Textual Scene Graphs (TSGs) parsed from captions. To bridge this gap, we propose PSGR, a retrieval-oriented framework that purifies generic visual scene graphs prior to matching. PSGR uses GPT-5-generated caption-guided supervision to adapt a Large Vision-Language Model (LVLM) with SFT and GRPO, then applies Rank-Preserved Candidate Sampling (RPCS) to select informative and diverse triplets from refined and generic graph candidates. The resulting Purified Scene Graphs (PSGs) are compared with TSGs through local matching and global GPO-based graph matching. The LVLM refiner is adapted using Flickr30K training supervision and then frozen to generate offline graph inputs, while the retrieval matchers are trained separately for each benchmark. Under this protocol, PSGR obtains 558.9 and 462.0 Rsum on Flickr30K and MS-COCO 5K, respectively, outperforming the existing scene graph matching methods. With the visual backbone and matching architecture held fixed, LVLM-based graph refinement and RPCS jointly improve Rsum by 13.0 points on Flickr30K and 6.6 points on MS-COCO 5K over generic VSG top-rank sampling. Our curated data and code are available at https://anonymous.4open.science/r/PSGR-DA6D

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image-Text Retrieval, Scene Graph Matching, Graph Purification, LVLM, Reinforce Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sxFxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PSGR improves visual graph inputs for image–text retrieval through caption-guided LVLM refinement and RPCS sampling, followed by local matching and GPO-based global matching. It reports strong results on Flickr30K and MS-COCO.

Critical strengths

The method addresses a clear semantic mismatch in graph-based retrieval. The appendix provides substantial analysis and clearly discloses external supervision, transfer settings, and timing scope.

Critical weaknesses

The specific benefit of caption-guided supervision is not sufficiently isolated. Table 3 changes both the graph generator and its supervision, while Table 2 mainly studies subsequent reward choices. Comparing generic graph supervision with caption-guided supervision using the same LVLM, training budget, and graph budget would test the central design more directly.

The connection between reward improvements and the quality of the retained grounded graph remains indirect. Equation 26 rewards both main and extra triplets, although default retrieval uses only main triplets. Multiplying raw label cosines can also reward semantically mismatched combinations. Figure 7 measures repetition and relation composition rather than relation correctness or grounding accuracy, leaving the source of the improvement unclear.

As a secondary point, the choice of 22K SFT and 7K GRPO samples is not justified or tested for allocation sensitivity (Section 3.1 and Appendix D.1). Table 2 compares training strategies rather than data allocations. Please explain the choice and compare a small number of allocations with the total supervision size fixed and compute matched as closely as possible.

Questions for authors

As weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer sxFx (1/3)

Thank you for recognizing the semantic mismatch addressed by PSGR and the detailed analyses in our appendix. We have added experiments and clarified the supervision design in response to your comments.

[W1] Caption-guided supervision and the GRPO ablation

Concern: The benefit of caption guidance should be examined using the same LVLM and comparable training and graph budgets.

Generic supervision in the current input setting. Our LVLM already receives the generic VSG as part of its prompt. Using that same graph as the training target would largely teach Qwen to reproduce information supplied in its input. This would compare graph reproduction with caption-oriented refinement, rather than compare two supervision sets with equivalent refinement quality.

Caption guidance enters at two stages. During CG-VSG construction, GPT-5 uses the image and its human-written captions to generate grounded targets that prioritize the entities and relations emphasized in human descriptions. These targets supervise SFT and provide the reference for the GRPO content-order reward. Caption information therefore enters the training targets before the cross-modal reward is introduced. During GRPO, the cross-modal reward additionally aligns generated triplets with caption-derived scene graphs, explicitly optimizing caption-level semantics.

Removing caption guidance throughout training would consequently require rebuilding the CG-VSG targets without captions and repeating LVLM adaptation and retrieval evaluation. This involves substantial data-construction time and cost. Generating the existing 29,000 Flickr30K targets alone cost over USD 350 in GPT-5 tokens, including failed generations and regeneration, followed by structured-output cleaning and SFT/GRPO training.

Given these practical considerations, we evaluated the additional caption-alignment signal at the GRPO stage. Both variants start from the same caption-guided SFT model, use the same GRPO data and content-order and format rewards, and differ only in the cross-modal reward. The retrieval architecture, DINOv2 features, and 36-object/18-relation graph budget remain unchanged.

Revised manuscript: Section 4.3.1, Table 2. Results below are min-max ensemble recalls and Rsum. Recall values are percentages.

Training strategyFlickr I2T R@1Flickr T2I R@1Flickr RsumCOCO 5K I2T R@1COCO 5K T2I R@1COCO 5K RsumCOCO 5-fold 1K Rsum
SFT only88.476.9554.762.349.8457.7539.4
SFT + GRPO without cross-modal reward87.776.0553.362.449.9458.1540.2
SFT + full GRPO90.378.1558.963.950.7462.0540.9

Adding the cross-modal reward improves ensemble Rsum by 5.6 on Flickr30K, 3.9 on COCO 5K, and 0.7 on COCO 5-fold 1K. Without it, GRPO changes Rsum relative to SFT by -1.4, +0.4, and +0.8, respectively. These results show the additional benefit of explicit caption alignment during reinforcement learning, on top of the caption guidance already present in the CG-VSG targets and SFT initialization.

Consistency under modified rewards. The same benefit persists when we jointly apply label-wise cosine clipping and positive-score order filtering. Adding the cross-modal reward then improves ensemble Rsum by 3.60 on Flickr30K, 4.96 on COCO 5K, and 1.84 on COCO 5-fold 1K (Appendix A.7.2, Table 18). Explicit caption alignment therefore improves retrieval under both reward implementations.

Direct graph-quality evidence. We also evaluate the resulting graphs on 100 randomly sampled Flickr30K test images using independently constructed, manually checked caption-grounded references and the same 18-relation budget (Section 4.4, Table 4, and Appendix A.5). Before GRPO, SFT + RPCS already increases caption-grounded precision from 6.78% to 14.22% and recall from 32.71% to 68.63%, compared with generic VSGs. Grounding accuracy among semantic matches increases from 13.11% to 16.80%, while the confirmed hallucination rate decreases from 21.67% to 10.39%. Full GRPO further reaches 15.44% precision, 74.53% recall, and 19.06% grounding accuracy, with an 8.06% hallucination rate.

Tables 2 and 18 assess the explicit caption-alignment reward during GRPO, while Table 4 measures caption coverage, grounding, and unsupported predictions in the retained graphs. The supervision-construction and adaptation stages are described in Section 3.1 and Appendix D.

AuthorsOpenReview ↗

Response to Reviewer sxFx (2/3)

[W2] Reward improvements and the retained grounded graph

Concern: Reward improvements may arise from extra triplets or sign cancellation, while the quality of the grounded relations retained for retrieval remains insufficiently measured.

Main and extra reward contributions. We decomposed the cross-modal score on the same 7,000 GRPO training images using their existing caption triplets. For each response, we recomputed the original reward using main candidates alone and using the union of main and extra candidates. Main candidates here are the parsed grounded output before RPCS.

ModelMain-only rewardMain + extra rewardExtra marginal contributionExtra wins (%)
SFT only0.4113800.4177860.0064063.89
GRPO without cross-modal reward0.4834730.4835620.0000900.04
Full GRPO0.5472690.5564490.0091809.62

Full GRPO improves the main-only score from 0.411380 after SFT to 0.547269. Extra candidates add 0.009180, approximately 1.65% of the complete mean score. The score increase is therefore also present in the grounded main output. This decomposition measures caption matching before sampling, while the independent evaluation below measures the final retained graph.

Direct evaluation of retained PSGs. We evaluated 100 randomly sampled Flickr30K test images and their 500 captions. References were constructed using GPT-6 Luna with human feedback, manually checked, and locked without access to evaluated predictions. The reference contains 373 caption-grounded relations. Each method contributes 1,800 predictions under the same 18-relation budget.

Precision and recall measure one-to-one semantic matches to the reference. Grounding accuracy is the proportion of these matches for which both endpoint boxes achieve IoU ≥0.5\geq0.5. Hallucination judgments use image and caption evidence, with uncertain cases recorded separately.

Graph sourceMatched triplets ↑\uparrowCaption-grounded precision ↑\uparrowCaption-grounded recall ↑\uparrowGrounding accuracy ↑\uparrowHallucination count ↓\downarrowHallucination rate ↓\downarrowUncertain count
Generic VSG (Neural Motifs)1226.7832.7113.1139021.6721
SFT + RPCS25614.2268.6316.8018710.3923
GRPO + RPCS27815.4474.5319.061458.0651

Compared with SFT + RPCS, GRPO + RPCS increases caption-grounded precision by 1.22 percentage points, recall by 5.90 points, and grounding accuracy among semantic matches by 2.26 points, while reducing the confirmed hallucination rate by 2.33 points. These measurements assess the actual graph inputs used for retrieval, adding direct semantic, grounding, and visual-support evidence to the repetition and relation-composition analysis.

Raw cosine products. We also sampled eight responses for each of the 7,000 images, giving 56,000 outputs per model, and removed positive products formed from two negative label cosines. We then reran content matching and recomputed cross-modal maxima for Full GRPO.

DiagnosticAffected / total evaluationsFrequencyMean reward decrease after removal
Output with a two-negative pair in its final content assignment1 / 56,000 outputs0.00179%1.40×10−101.40\times10^{-10}
Caption-triplet score affected in cross-modal matching1 / 499,104 caption-triplet evaluations0.000200%9.85×10−99.85\times10^{-9}

We additionally retrained with label-wise cosine clipping to [0,1][0,1] in both semantic rewards and positive-score filtering for the order reward. Assignments must exceed 10−810^{-8} for order scoring, and fewer than two valid assignments receive zero order reward. Under this combined modification, adding the cross-modal reward still improves ensemble Rsum by 3.60 on Flickr30K, 1.84 on COCO 5-fold 1K, and 4.96 on COCO 5K. The full modified-reward results are 558.54, 540.78, and 461.46, only 0.36, 0.12, and 0.54 below the original results (Appendix A.7.2, Table 18). Together with the measured occurrence frequencies and reward changes, these results suggest that sign cancellation was not a critical factor driving retrieval performance in these experiments.

Extra triplets in the matching input. We focus on graph purification rather than matching-network design, and therefore use SEMScene’s established node-level and graph-level networks as the local and global matchers. We tested extra-triplet integration by applying the same preprocessing and embedding procedure as textual triplets, then concatenating their embeddings with the image-side feature sequence before visual GPO pooling. This reduced Flickr30K ensemble Rsum from 558.9 to 544.5 (Appendix A.8, Table 19). Our default input therefore uses grounded visual triplets, while dedicated integration of text-only extra semantics is left to future work.

AuthorsOpenReview ↗

Response to Reviewer sxFx (3/3)

[W3] Allocation of SFT and GRPO supervision

Concern: The 22K SFT / 7K GRPO split requires a rationale and an allocation comparison at a fixed total supervision size.

The 22K/7K design assigns most supervision to learning grounded, caption-aligned generation through SFT, while reserving a smaller subset for the more expensive reward-based optimization. SFT establishes the generation content and grounding interface, and GRPO further optimizes semantic fidelity and caption alignment. The 7K GRPO subset is selected by output-length bin sampling, as described in Section 3.1 and Appendix D.1.

Following your suggestion, we added two allocations around the default split, 20K/9K and 24K/5K, keeping the total CG-VSG supervision size fixed at 29,000 samples. We evaluate all three allocations using the same retrieval architecture, DINOv2 features, 36-object/18-relation graph budget, and min-max fusion. Local and Global matchers are trained for 30 epochs and selected by validation performance. The evaluated GRPO checkpoints are step 5,625 for 20K/9K, 4,375 for 22K/7K, and 3,125 for 24K/5K, making the optimization-step counts explicit alongside the fixed sample total.

Appendix A.13.1, Table 24. The manuscript also reports all Local and Global metrics.

Flickr30K 1K

BranchSFT / GRPO samplesI2T R@1R@5R@10T2I R@1R@5R@10Rsum
Local20K / 9K84.898.399.373.793.696.5546.16
Local22K / 7K85.998.699.473.993.396.8547.90
Local24K / 5K84.198.399.472.692.696.1543.08
Global20K / 9K86.698.699.874.693.095.8548.46
Global22K / 7K87.498.299.574.393.096.3548.70
Global24K / 5K86.698.899.673.393.096.1547.42
Ensemble20K / 9K89.198.999.777.994.897.3557.68
Ensemble22K / 7K90.399.099.578.194.797.3558.90
Ensemble24K / 5K88.398.699.976.494.597.3554.92

MS-COCO 5-fold 1K

BranchSFT / GRPO samplesI2T R@1R@5R@10T2I R@1R@5R@10Rsum
Local20K / 9K79.096.698.968.492.897.0532.64
Local22K / 7K79.196.699.268.592.897.0533.20
Local24K / 5K78.796.898.968.192.596.8531.70
Global20K / 9K80.797.299.267.691.996.1532.80
Global22K / 7K81.697.099.167.992.096.1533.60
Global24K / 5K80.097.099.267.491.896.3531.76
Ensemble20K / 9K82.197.899.370.793.397.3540.51
Ensemble22K / 7K82.797.599.371.093.397.2540.90
Ensemble24K / 5K81.897.499.170.493.297.1538.98

MS-COCO 5K

BranchSFT / GRPO samplesI2T R@1R@5R@10T2I R@1R@5R@10Rsum
Local20K / 9K57.585.092.047.475.785.0442.65
Local22K / 7K58.186.192.847.476.085.0445.30
Local24K / 5K56.485.492.547.175.584.9441.90
Global20K / 9K60.686.893.046.375.284.5446.44
Global22K / 7K61.387.193.446.975.584.7448.90
Global24K / 5K59.085.993.245.974.984.2443.08
Ensemble20K / 9K63.287.894.050.278.186.6459.96
Ensemble22K / 7K63.988.394.450.778.086.7462.00
Ensemble24K / 5K61.987.194.049.477.986.4456.74

Among the tested allocations, 22K/7K achieves the highest Rsum for Local, Global, and Ensemble under all three evaluation protocols. Its ensemble gains over 20K/9K are 1.22, 0.39, and 2.04 on Flickr30K, COCO 5-fold 1K, and COCO 5K, respectively. Compared with 24K/5K, the gains are 3.98, 1.92, and 5.26. Flickr30K benefits more strongly in the Local branch, while both COCO branches improve. The 22K/7K split also achieves the highest ensemble R@1 in both retrieval directions across all three protocols.

Individual cutoffs occasionally favor other allocations. For example, 24K/5K achieves higher Flickr30K ensemble I2T R@10, at 99.9% versus 99.5%, but lower Rsum. Overall, moving 2,000 samples from either SFT to GRPO or GRPO to SFT reduces ensemble performance. This pattern is consistent with the complementary roles of supervised initialization and reward-based refinement, with 22K/7K providing the strongest overall performance among the tested splits.

Thanks for the detailed response, and I have no further questions.

Reviewer_T9JdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces PSGR, a retrieval-oriented framework that reduces the semantic gap between generic visual scene graphs and caption-derived textual scene graphs. It uses GPT-5-generated caption-guided supervision to adapt an LVLM through SFT and GRPO, applies RPCS to construct purified scene graphs from refined and generic graph candidates, and combines fine-grained local matching with global GPO-based graph matching. PSGR achieves Rsum scores of 558.9 on Flickr30K and 462.0 on MS-COCO 5K; under a fixed DINOv2 backbone and retrieval matcher, graph refinement and RPCS improve Rsum over generic VSG sampling by 13.0 and 6.6 points, respectively.

Critical strengths
  • The figures are clear and make the overall pipeline easy to follow.
  • The results are strong, and Table 1 shows that PSGR outperforms all the compared methods.
Critical weaknesses
  • The overall pipeline is overly complex, and its practical efficiency remains unclear. The image branch sequentially relies on Neural Motifs, a Qwen refiner, bounding-box correction, RPCS, DINOv2/GloVe features, and a GCN, while the text branch further uses T5, BiGRU, and GloVe before combining local and global matchers. The appendix reports only matching-stage latency with graphs and features precomputed; full PSGR is already about 1.43× slower than SGM at this stage, while scene-graph generation, LVLM refinement, feature extraction, end-to-end latency, and caching costs are excluded.

  • The main comparison is not conducted under matched settings. PSGR benefits from GPT-5-generated supervision, Qwen2.5-VL, DINOv2 ViT-g/14, and a two-branch ensemble, whereas the external baselines use different backbones, pretraining data, and training protocols. DINOv2 alone improves Rsum by 14.4/20.0; although Table 3 confirms that graph purification still contributes 13.0/6.6 under a fixed PSGR backbone and matcher, external methods are not reevaluated with matched features, supervision, and compute budgets.

  • The methodological novelty and necessity of individual components are insufficiently established. Much of the gain comes from GPT-5-generated caption-guided supervision and the trained LVLM refiner, whereas RPCS adds only about 1.5/1.7 Rsum over RSG top-rank selection. Since GPT-5 is already used to construct CG-VSG targets, the paper should justify the full “generic VSG → CG-VSG → Qwen RSG → RPCS PSG” chain and compare it with a simpler approach that directly generates retrieval-oriented PSGs from images using a strong LVLM. Moreover, PSG quality is evaluated mainly through downstream retrieval, without direct measurements of factuality, grounding accuracy, relation precision/recall, or hallucination.

Questions for authors
  • What are the end-to-end latency, FLOPs, memory usage, and caching costs of the full pipeline, including generic VSG generation, Qwen refinement, DINOv2 feature extraction, and local/global matching? How do they compare with the main baselines?

  • Is the resulting PSG directly better than the generic VSG in terms of graph quality? It would be helpful to report triplet precision/recall, grounding accuracy, caption alignment, relation diversity, hallucination rate, or a human evaluation, rather than relying only on downstream Recall@K.

  • Since GPT-5 is already used to generate CG-VSG supervision, why is the full chain of generic VSG generation, Qwen refinement, and RPCS still necessary? How would it compare with a simpler baseline where GPT-5 or the same LVLM directly generates a fixed-budget PSG from the image?

5 comments
AuthorsOpenReview ↗

Response to Reviewer T9Jd (1/3)

Thank you for recognizing the clarity of our figures and the strong retrieval results. We have conducted additional experiments in response to your suggestions to further substantiate our claims.

[W1 and Q1] Full-pipeline efficiency and computational costs

Concern: The original efficiency analysis covers only graph matching and excludes the upstream processing required by the complete pipeline.

We have added a full-pipeline evaluation on 1,000 Flickr30K test images and their 5,000 captions, using a single NVIDIA H800 GPU with 80GB of memory. The evaluation includes generic VSG generation, Qwen refinement, RPCS, visual feature extraction, text processing, and graph encoding, matching, and ranking.

Revised manuscript: Section 4.6, Table 6.

MethodEnd-to-end runtime (s/test set)FLOPs (TFLOPs/image)GPU memory (GB)Image cache (MB/image)Preprocessing throughput (images/s)
SGM552.682.68169.950.5091.856
LGSGM553.372.68189.950.5031.856
SEMScene571.692.733619.650.5091.856
PSGR1588.1583.419866.030.3360.636

PSGR requires 1,588.15 seconds for the complete test set, approximately 2.87 times SGM's runtime and 2.78 times SEMScene's. Its GPU memory usage is 66.03 GB, preprocessing throughput is 0.636 images/s, and image-side cache storage is 0.336 MB per image, or approximately 336 MB for the 1,000-image gallery.

Revised manuscript: Appendix A.12.2, Table 23. Stage runtimes are in seconds.

StageSGMLGSGMSEMScenePSGRPSGR amortized time (s/image)
Generic VSG generation325.20325.20325.20325.200.32520
LVLM refinement------983.500.98350
RPCS------1.600.00160
Visual feature extraction180.10180.10180.10107.950.10795
Text processing33.5033.5033.50154.010.15401
Graph encoding, matching, and ranking13.8914.5732.9015.900.01590
End-to-end total552.68553.37571.691588.151.58815

LVLM refinement is the principal overhead, taking 983.50 seconds, or 61.9% of PSGR's total runtime. Text processing also adds cost because PSGR uses FlanT5-SG, whereas the baselines use a CPU-based parser. RPCS takes 1.60 seconds, approximately 0.1% of the total, while graph encoding, matching, and ranking take 15.90 seconds. These measurements identify the upstream stages responsible for most of the additional runtime.

The four methods use Neural Motifs on the same images. The baselines use EfficientNet-B5 with 456×456456\times456 crops in FP32, while PSGR uses DINOv2 ViT-g/14 with 224×224224\times224 crops under BF16 autocasting. All use crop batch size 32. The evaluated graph budgets are 36 objects and 25 relations for the baselines and 36 objects and 18 relations for PSGR. Full inference settings are given in Appendix E.1.2.

The per-image time of 1.58815 seconds is the complete test-set runtime divided by 1,000, including processing the 5,000 captions. It is an amortized measurement for this batched workload.

Although PSGR incurs higher full-pipeline costs than the main baselines, the additional runtime is concentrated in LVLM refinement, which accounts for 61.9% of the total. Under the evaluated batched workload, complete processing takes an amortized 1.59 seconds per image, while graph encoding, matching, and ranking contribute only 0.016 seconds per image. These results demonstrate a practical trade-off between retrieval accuracy and computational cost.

AuthorsOpenReview ↗

Response to Reviewer T9Jd (2/3)

[W2] Comparisons under the same visual backbone

Concern: Differences in visual features, supervision, matching architectures, and computational costs complicate attribution of the reported gains.

We have explicitly distinguished the improvements from the visual backbone and graph purification in the revised Introduction and Experiments. Replacing EfficientNet-B5 with DINOv2 improves ensemble Rsum by 14.4 on Flickr30K and 20.0 on MS-COCO 5K. Separately, with the same DINOv2 backbone and matching architecture, LVLM refinement and RPCS jointly improve Rsum by 13.0 and 6.6 on the two benchmarks.

Matched-backbone comparisons. We evaluate three open-source scene graph matching methods, SGM, LGSGM, and SEMScene, under EfficientNet-B5 and DINOv2 ViT-g/14, retaining each method's matching architecture. The following Flickr30K results report all six recalls and Rsum.

Revised manuscript: Appendix A.4.1, Table 10. Recall values are percentages.

BackboneMethodI2T R@1R@5R@10T2I R@1R@5R@10Rsum
EfficientNet-B5SGM71.8091.7095.5053.5079.6086.50478.60
EfficientNet-B5LGSGM71.0091.9096.1057.4084.1090.20490.70
EfficientNet-B5SEMScene77.8095.1097.7063.2087.6092.60514.00
EfficientNet-B5PSGR86.0097.7099.1073.4092.5095.80544.50
DINOv2 ViT-g/14SGM80.1098.2099.5069.8092.1095.90535.60
DINOv2 ViT-g/14LGSGM80.7098.2099.7070.6892.4895.78537.54
DINOv2 ViT-g/14SEMScene84.3099.1099.9073.4893.7896.54547.10
DINOv2 ViT-g/14PSGR90.3099.0099.5078.1094.7097.30558.90

PSGR achieves the highest Rsum under both visual backbones, exceeding SEMScene by 30.50 with EfficientNet-B5 and 11.80 with DINOv2.

PSG inputs in existing matching networks. We additionally compare generic VSG and PSG inputs within each matching architecture using the same DINOv2 backbone. Parentheses indicate changes relative to the corresponding generic-VSG setting.

Revised manuscript: Section 4.5, Table 5, and Appendix A.4.2.

MethodGraph inputI2T R@1R@5R@10T2I R@1R@5R@10Rsum
SGMGeneric VSG80.1098.2099.5069.8092.1095.90535.60
SGMPSG83.60 (+3.50)98.00 (-0.20)99.10 (-0.40)72.70 (+2.90)92.20 (+0.10)96.00 (+0.10)541.60 (+6.00)
LGSGMGeneric VSG80.7098.2099.7070.6892.4895.78537.54
LGSGMPSG83.50 (+2.80)98.50 (+0.30)99.50 (-0.20)75.20 (+4.52)94.00 (+1.52)96.70 (+0.92)547.40 (+9.86)
SEMSceneGeneric VSG84.3099.1099.9073.4893.7896.54547.10
SEMScenePSG88.80 (+4.50)99.00 (-0.10)99.60 (-0.30)77.00 (+3.52)94.70 (+0.92)97.30 (+0.76)556.40 (+9.30)

The default generic VSG input contains 36 object slots and 25 relation slots, while PSG uses 36 object slots and 18 relation slots. PSG therefore retains the object budget while reducing the relation budget by seven slots, or 28%. The internal sampling ablations in Table 3 separately use a common budget of 36 objects and 18 relations for all variants.

Replacing generic VSGs with PSGs improves Rsum by 6.00 for SGM, 9.86 for LGSGM, and 9.30 for SEMScene, with higher R@1 in both retrieval directions for all three methods. These results show that the purified graph representation benefits existing matching networks.

Supervision and computational settings. The LVLM uses 29,000 Flickr30K training samples, with 22,000 for SFT and 7,000 for GRPO, and is then frozen to generate graph inputs for both benchmarks. Retrieval matchers are trained separately on each benchmark's training pairs (Section 4.1). LVLM adaptation takes 16.3 hours for SFT and 82.3 hours for GRPO on eight 80GB A100 GPUs (Appendix E.1.1). Full-pipeline inference and storage measurements are provided in Part 1 and Section 4.6, Table 6.

AuthorsOpenReview ↗

Response to Reviewer T9Jd (3/3)

[W3, Q2, and Q3] Component roles and direct graph-quality evaluation

Concern: The contribution of RPCS is smaller than that of LVLM refinement, the full generation chain needs justification, and direct graph-quality evidence is missing.

RPCS complements LVLM refinement through static sampling. We agree that RPCS's incremental gain is smaller than the generic VSG-to-RSG improvement. RPCS is a lightweight, training-free step that adds no further LVLM inference. It diversifies the retained RSG relations and, when the relation budget is underfilled, supplements them with generic VSG candidates selected for diversity and reduced redundancy. This broadens the available semantic coverage within a compact graph input. Table 3 reports additional gains of 1.5/1.7 Rsum on Flickr30K/MS-COCO 5K over RSG top-rank selection.

Direct graph-quality evaluation. We evaluated 100 randomly sampled Flickr30K test images and their 500 captions. References were constructed with GPT-6 Luna and human feedback, manually checked, and locked independently of the evaluated predictions, yielding 373 caption-grounded relations. Neural Motifs, SFT + RPCS, and GRPO + RPCS use the same images and an 18-relation budget, with 1,800 predictions per method.

Precision and recall measure one-to-one semantic matches against the reference. Grounding accuracy is evaluated among these matches and requires both endpoint boxes to satisfy IoU ≥0.5\geq0.5. Hallucination judgments use image and caption evidence, with uncertain predictions recorded separately. Visually supported relations absent from the captions are not automatically counted as hallucinations.

Revised manuscript: Section 4.4, Table 4. Rates are percentages. The full protocol appears in Appendix A.5.

Graph sourceMatched triplets ↑\uparrowCaption-grounded precision ↑\uparrowCaption-grounded recall ↑\uparrowGrounding accuracy ↑\uparrowHallucination count ↓\downarrowHallucination rate ↓\downarrowUncertain count
Generic VSG (Neural Motifs)1226.7832.7113.1139021.6721
SFT + RPCS25614.2268.6316.8018710.3923
GRPO + RPCS27815.4474.5319.061458.0651

SFT substantially improves caption alignment over Neural Motifs. GRPO further increases caption-grounded recall by 5.90 percentage points, improves grounding accuracy among semantic matches by 2.26 points, and reduces the confirmed hallucination rate by 2.33 points. These measurements directly assess caption alignment, localization, and unsupported predictions in the graph inputs, complementing downstream Recall@K.

Why the full chain is used. Our design separates caption-guided supervision construction, caption-free graph refinement, and fixed-budget graph selection. CG-VSG construction is a training-time process, not an additional generation stage performed for every test image.

Direct GPT-5 graph generation across both benchmarks would incur substantial costs. Flickr30K and MS-COCO together contain approximately 154,000 images across the evaluated splits. Constructing targets individually for the 29,000 Flickr30K training images already cost over USD 350 in GPT-5 tokens, including failed generations and regeneration. Scaling generation would also require structured-output cleaning and validation of graph parseability and grounding. We therefore construct a limited GPT-5 supervision set and use it to adapt a locally deployed 7B LVLM.

Reviewer_2oPROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the semantic gap in scene-graph-based image-text retrieval caused by generic scene graph generators producing visual scene graphs (VSGs) with caption-irrelevant or overly coarse semantics, and proposes PSGR, a retrieval-oriented framework that purifies generic VSGs prior to cross-modal matching. Under an evaluation protocol where the LVLM refiner is adapted on Flickr30K and kept frozen to generate offline graph inputs while the retrieval matchers are trained independently per benchmark, PSGR achieves 558.9 Rsum on Flickr30K and 462.0 Rsum on MS-COCO 5K, outperforming existing region-based, patch-based, and scene-graph-based baselines.

Critical strengths

S1: The paper effectively identifies a fundamental bottleneck in existing scene-graph-based image-text retrieval methods, which predominantly focus on designing more intricate matching networks over given scene graphs and thus inevitably inherit the generic, noisy, and caption-irrelevant semantic limitations of input Visual Scene Graphs (VSGs). By reformulating scene graph purification as a retrieval-oriented input construction problem, the proposed PSGR framework directly targets the semantic gap between generic VSGs and sparse Textual Scene Graphs (TSGs).

S2: The paper exhibits commendable transparency and rigor in disclosing its experimental protocol, engineering implementation details, and methodological limitations. In Section 4.1 as well as Appendices A.2, E, and F, the authors candidly clarify the scope of data usage and empirical constraints, explicitly noting that the MS-COCO evaluation incorporates external Flickr30K supervision during the graph-refinement stage and that the cross-dataset experiment in Appendix A.7 isolates matcher transfer rather than serving as an end-to-end zero-shot evaluation.

Critical weaknesses

W1: The fairness of the primary comparisons in Tables 1 and 4 is substantially undermined by visual backbone discrepancies, while external supervision and backbone interactions remain insufficiently decoupled. Specifically, the visual extractor ablation in Table 5 shows that replacing EfficientNet-B5 with DINOv2 ViT-g/14 yields massive ensemble Rsum gains. When constrained to EfficientNet-B5, PSGR's Rsum drops to 531.0 on MS-COCO 5-fold 1K and 442.0 on MS-COCO 5K, underperforming multiple existing baselines in Tables 1 and 4, including HREM (2023), CHAN (2023), CORA (2024), RVSE++ (2025), and D2S-VSE (2025).

Furthermore, the MS-COCO main experiments and the MS-COCO-to-Flickr30K cross-dataset evaluation in Table 9 rely on a frozen LVLM refiner adapted on 29,000 Flickr30K training samples to generate RSG inputs. This setup not only breaks the strict in-dataset training setting for MS-COCO, but also means that the upstream refiner in Table 9 had direct access to Flickr30K image-caption supervision, creating an unfair protocol disparity against baselines (SCAN, HREM, CHAN) that received no Flickr30K exposure.

W2: The proposed RPCS strategy demonstrates marginal empirical gains over simple top-rank truncation and relies on rigid, potentially flawed heuristic rules. As shown in the ablation study in Table 3, the vast majority of the purification gain is already captured by LVLM refinement alone. In contrast, introducing the complex weighted sampling mechanics of Full RPCS yields only marginal incremental improvements. Given that Appendix A.4 explicitly acknowledges that variability across random seeds was not measured, these minimal margins risk falling within stochastic experimental variance.

In fallback sampling for under-complete RSGs (Section 3.2, Eq. 4), the sampling weights πj(r)\pi_j^{(r)} are computed strictly relative to selected fallback candidates and independently of the retained RSG, which inherently fails to suppress relational or subject-predicate redundancy between the fallback generic triplets and the already retained RSG triplets. Furthermore, Appendix E.2 reveals that the coarse relation categorization (attribute, spatial, action, semantic) relies entirely on hard-coded string matching against static word lists without lemmatization or learned classifiers.

W3: The framework incurs heavy training and feature extraction computational overheads, while its inference efficiency analysis largely omits the most time-consuming upstream stages. According to Appendix E.1.1, fine-tuning Qwen2.5-VL-7B on eight 80GB NVIDIA A100 GPUs requires 16.3 hours for SFT and 82.3 hours for GRPO, yet Table 2 indicates that the computationally intensive GRPO stage yields modest ensemble Rsum gains. Furthermore, the accuracy-speed trade-off in Appendix A.9 and Figure 5 evaluates only the final matching stage while excluding generic VSG generation, LVLM refinement, and visual feature extraction, while novel queries or un-cached gallery additions still require full upstream processing, the paper leaves end-to-end latency, preprocessing throughput, and cache storage unmeasured.

W4: There are several technical vulnerabilities and inconsistencies in the GRPO reward formulation, the extra triplet design, and the textual encoder substitution. Specifically, Appendix D.2.2 reveals that the cross-modal reward rcrossr_{cross} (Eq. 26) calculates document similarities via spaCy en_core_web_sm and computes c(q,v^)c(q, \hat{v}) without per-label nonnegative clipping. Because the zero floor acts only after multiplication, two negative similarities can erroneously yield a positive product. Additionally, the order reward rorderr_{order} (Eq. 24) retains zero-score Hungarian assignments without filtering them by a positive score threshold. Furthermore, although 99.6% of the CG-VSG samples contain ungrounded extra triplets that the LVLM is explicitly trained and rewarded to generate during both SFT and GRPO, the primary PSGR model discards them when constructing visual inputs.

Questions for authors

Q1: Appendix A.9 explicitly notes that end-to-end latency, preprocessing throughput, and cache storage are not measured, please provide a concrete breakdown of the wall-clock runtime, memory footprint, and disk storage overhead when processing a single image across all four sequential stages: (1) generic VSG generation using Neural Motifs, (2) RSG generation via the frozen Qwen2.5-VL-7B refiner, (3) independent visual feature extraction using DINOv2 ViT-g/14, and (4) the final local and global graph matching. Presenting these empirical measurements is necessary to clarify the true computational and preprocessing costs of PSGR when handling novel query images or newly ingested gallery instances in realistic retrieval deployments.

Q2: As shown in Table 3, Full RPCS offers modest gains of only 0.7 to 1.7 Rsum points over RSG Top-rank, while Appendix A.4 explicitly notes that variability across random seeds was not measured. Therefore, please report the mean and standard deviation across multiple random seeds for the retrieval matcher to substantiate whether the performance margin of RPCS over RSG Top-rank is statistically significant. Additionally, since Eq. 4 currently evaluates fallback sampling weights relative to selected fallback candidates independently of the retained RSG, please clarify whether calculating redundancy penalties and diversity weights with respect to the joint set of retained RSG triplets and selected fallback triplets would further improve retrieval accuracy.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 2oPR (1/4)

Thank you for your thoughtful comments. We have conducted additional experiments in response to your suggestions.

[W1] Visual backbones and training protocols

Concern: Backbone differences complicate the primary comparisons. The Flickr30K-adapted refiner also introduces external supervision into MS-COCO experiments and target-dataset exposure into the matcher-transfer evaluation.

Matched-backbone comparisons. We have explicitly distinguished the improvements from the visual backbone and graph purification in the revised Introduction and Experiments. Replacing EfficientNet-B5 with DINOv2 improves ensemble Rsum by 14.4 on Flickr30K and 20.0 on MS-COCO 5K. Separately, using the same DINOv2 backbone and matching architecture, LVLM refinement and RPCS jointly improve Rsum by 13.0 and 6.6 on the two benchmarks. The revised text reports these contributions separately, supported by the feature-extractor and graph-sampling ablations.

Fine-grained retrieval methods are designed for their visual input representations. HREM, CHAN, CORA, and RVSE++ use BUTD region features, while the selected D2S-VSE configuration uses ViT-Base-224 patch features. PSGR instead uses entity and relation regions associated with structured scene graphs. Replacing one feature extractor with another across these architectures is therefore not a standalone backbone substitution, because their input interfaces and representation structures also differ.

We consequently evaluate the publicly available scene graph matching methods SGM, LGSGM, and SEMScene under the same visual backbone. Each retains its matching architecture. The following results are included in Appendix A.4.1, Table 10. Recall values are percentages.

Revised manuscript: Appendix A.4.1, Table 10.

BackboneMethodI2T R@1R@5R@10T2I R@1R@5R@10Rsum
EfficientNet-B5SGM71.8091.7095.5053.5079.6086.50478.60
EfficientNet-B5LGSGM71.0091.9096.1057.4084.1090.20490.70
EfficientNet-B5SEMScene77.8095.1097.7063.2087.6092.60514.00
EfficientNet-B5PSGR86.0097.7099.1073.4092.5095.80544.50
DINOv2 ViT-g/14SGM80.1098.2099.5069.8092.1095.90535.60
DINOv2 ViT-g/14LGSGM80.7098.2099.7070.6892.4895.78537.54
DINOv2 ViT-g/14SEMScene84.3099.1099.9073.4893.7896.54547.10
DINOv2 ViT-g/14PSGR90.3099.0099.5078.1094.7097.30558.90

PSGR obtains the highest Rsum in both backbone groups. Relative to SEMScene, the differences are 30.50 with EfficientNet-B5 and 11.80 with DINOv2. Under DINOv2, SEMScene retains higher I2T R@5 and R@10, while PSGR achieves higher I2T R@1 and all three T2I recalls.

Changing graph inputs within existing matchers. We additionally evaluate each of these three matching networks with generic VSG and PSG inputs, keeping the DINOv2 visual backbone and the corresponding matching architecture unchanged. This directly tests the benefit of the purified graph representation in existing networks. Section 4.5, Table 5, and Appendix A.4.2 report the results. Parentheses below show changes from each method's generic-VSG result in the revised manuscript's Table 10.

Revised manuscript: Section 4.5, Table 5, and Appendix A.4.2.

MatcherGraph inputI2T R@1R@5R@10T2I R@1R@5R@10Rsum
SGMPSG83.60 (+3.50)98.00 (-0.20)99.10 (-0.40)72.70 (+2.90)92.20 (+0.10)96.00 (+0.10)541.60 (+6.00)
LGSGMPSG83.50 (+2.80)98.50 (+0.30)99.50 (-0.20)75.20 (+4.52)94.00 (+1.52)96.70 (+0.92)547.40 (+9.86)
SEMScenePSG88.80 (+4.50)99.00 (-0.10)99.60 (-0.30)77.00 (+3.52)94.70 (+0.92)97.30 (+0.76)556.40 (+9.30)

PSG inputs improve Rsum and R@1 in both directions for all three matchers. These experiments complement the within-PSGR ablations by showing improvements with the visual backbone fixed in other scene graph matching networks.

Flickr30K supervision and the MS-COCO protocol. We acknowledge that the complete MS-COCO pipeline is not strictly in-dataset. We constructed CG-VSG supervision only from Flickr30K because of data-construction time and financial costs. Generating targets individually for 29,000 Flickr30K images already cost over USD 350 in GPT-5 tokens, including failed generations and regeneration, and required additional structured-output cleaning. We estimated that constructing comparable supervision for MS-COCO containing 113,287 images would cost at least four times as much, followed by further LVLM fine-tuning costs. For these practical reasons, we used the Flickr30K-adapted refiner for both benchmarks, while training the retrieval matchers separately on each benchmark.

AuthorsOpenReview ↗

Response to Reviewer 2oPR (2/4)

[W2 and Q2] RPCS gains, random seeds, fallback references, and coarse relation types

Concern: RPCS adds relatively small improvements over RSG top-rank selection, without measurements of training variability. Its fallback reference and rule-based relation categories also require validation.

The role and cost of RPCS. We agree that LVLM refinement accounts for the larger part of the purification gain. In Table 3, generic VSG top-rank to RSG top-rank increases Rsum by 11.5 on Flickr30K and 4.9 on MS-COCO 5K. RPCS then adds 1.5 and 1.7 points in the original runs. This additional step uses static sampling of the available graph candidates, without another learned model or training stage. Its measured runtime is 1.60 seconds for 1,000 images, approximately 0.1% of the complete PSGR runtime.

Five-run evaluation. We repeated retrieval training with seeds 42, 123, 2026, and 3407 for both RSG top-rank and Full RPCS, and combined these with the original reported run. Precomputed graph inputs, DINOv2 features, T5 textual inputs, and training settings are fixed. Local and Global branches from the same seed are paired for min-max ensemble. Appendix A.6.2, Tables 11 and 12, reports all six recalls and Rsum. The Rsum summary is below, using means and sample standard deviations over five runs.

Benchmark / protocolBranchRSG top-rankFull RPCSMean gain
Flickr30K 1KLocal546.00±0.90546.00 \pm 0.90547.90±1.51547.90 \pm 1.51+1.90
Flickr30K 1KGlobal548.68±1.06548.68 \pm 1.06548.70±0.41548.70 \pm 0.41+0.02
Flickr30K 1KEnsemble557.75±0.35557.75 \pm 0.35558.90±0.88558.90 \pm 0.88+1.15
MS-COCO 5-fold 1KLocal532.74±0.75532.74 \pm 0.75533.20±0.51533.20 \pm 0.51+0.46
MS-COCO 5-fold 1KGlobal532.91±0.31532.91 \pm 0.31533.60±0.46533.60 \pm 0.46+0.69
MS-COCO 5-fold 1KEnsemble539.98±0.35539.98 \pm 0.35540.98±0.22540.98 \pm 0.22+1.00
MS-COCO 5KLocal444.43±0.90444.43 \pm 0.90445.30±0.39445.30 \pm 0.39+0.87
MS-COCO 5KGlobal446.61±0.49446.61 \pm 0.49448.90±0.52448.90 \pm 0.52+2.29
MS-COCO 5KEnsemble459.69±0.65459.69 \pm 0.65462.00±0.50462.00 \pm 0.50+2.31

The ensemble gain recurs in three of the four additional Flickr30K runs and all four additional runs under each COCO protocol. Its mean increase is 1.15, 1.00, and 2.31 points on the three protocols. Branch-level effects differ, particularly on Flickr30K, where the Global means differ by only 0.02 and the Local branch gains 1.90. We report these repeated-run measurements rather than inferring statistical significance from a single-run margin.

Why the original fallback uses a fallback-only reference. Neural Motifs is trained for generic scene graph prediction and supplies broad object and relation candidates, whereas the LVLM is adapted to prioritize caption-aligned graph content. When a nonempty RSG has fewer than 18 valid relations, we first retain its available relations and then use generic candidates to supplement the remaining slots. The default reference promotes diversity within this supplementary selection. A generic relation that shares a predicate or coarse type with an RSG relation may still connect different entities and add scene context, so shared labels alone need not indicate redundant visual evidence.

Joint-reference experiment. Following your suggestion, we include the retained RSG relations in the fallback redundancy and diversity statistics, changing only the reference set. All other sampling parameters remain unchanged. Partial fallback is rare under the nonempty-but-under-complete condition, as summarized in Appendix A.6.3, Table 13.

DatasetTotal imagesPartial fallbackPercentageFinal graphs changed by joint reference
Flickr30K31,014390.126%25
MS-COCO123,2873600.292%168

These counts include only nonempty RSGs with fewer than 18 valid relations, excluding missing RSGs and zero-relation whole-graph fallback. A qualifying image need not change its selected graph when the reference changes. Table 14 reports the retrieval results, with ensemble Rsum summarized below alongside the separate coarse-type ablation.

Coarse-type experiment. Regarding the rule-based categories, we use them only as a soft sampling preference, not as replacement relation labels or hard candidate filters. We test their contribution by setting only λt=0\lambda_t=0.

SettingFlickr30K 1KMS-COCO 5-fold 1KMS-COCO 5K
Joint RSG-fallback reference557.4540.7460.6
Without coarse-type diversity, λt=0\lambda_t=0556.3540.5459.3
Full RPCS, fallback-only reference558.9540.9462.0

Joint-reference sampling does not improve ensemble Rsum in these runs, changing it by -1.5, -0.2, and -1.4 points. Disabling coarse-type diversity reduces Rsum by 2.6, 0.4, and 2.7 points. These experiments separately measure the effects of the reference choice and the coarse-type sampling preference.

AuthorsOpenReview ↗

Response to Reviewer 2oPR (3/4)

[W3 and Q1] Full-pipeline computational and storage costs

Concern: The original efficiency analysis excludes upstream processing, and LVLM adaptation is expensive relative to its incremental retrieval gains.

We agree that matching-stage timing alone does not characterize the full computational cost. We now measure the complete pipeline on 1,000 Flickr30K test images and their 5,000 captions, using one NVIDIA H800 GPU with 80GB memory. The measurements include Neural Motifs VSG generation, frozen Qwen refinement, RPCS, visual feature extraction, text processing, and final graph encoding, matching, and ranking. Results appear in Section 4.6, Table 6, with stage-wise measurements in Appendix A.12, Table 23.

Section 4.6, Table 6.

MethodEnd-to-end runtime (s/test set)FLOPs (TFLOPs/image)GPU memory (GB)Image cache (MB/image)Preprocessing throughput (images/s)
SGM552.682.68169.950.5091.856
LGSGM553.372.68189.950.5031.856
SEMScene571.692.733619.650.5091.856
PSGR1588.1583.419866.030.3360.636

Appendix A.12.2, Table 23.

StageSGM (s)LGSGM (s)SEMScene (s)PSGR (s)PSGR amortized time (s/image)
Generic VSG generation325.20325.20325.20325.200.32520
LVLM refinement------983.500.98350
RPCS------1.600.00160
Visual feature extraction180.10180.10180.10107.950.10795
Text processing33.5033.5033.50154.010.15401
Graph encoding, matching, and ranking13.8914.5732.9015.900.01590
End-to-end total552.68553.37571.691588.151.58815

All four methods use Neural Motifs on the same images. SGM, LGSGM, and SEMScene use their generic VSG inputs with 36 objects and 25 relations. PSGR uses the frozen Qwen2.5-VL-7B GRPO checkpoint at step 4,375 and constructs 36-object, 18-relation PSG inputs. Qwen inference uses vLLM with tensor and pipeline parallelism both set to one, at most 16 concurrent sequences, and a maximum generation length of 2,048.

The baselines use EfficientNet-B5 with 456×456456\times456 crops in FP32. PSGR uses DINOv2 ViT-g/14 with 224×224224\times224 crops under BF16 autocasting, storing its features in FP32. Crop batch size is 32 for all methods. Baseline text processing uses the CPU scene graph parser, while PSGR uses FlanT5-SG. These configurations and their input budgets are specified in Appendix E.1.2.

The largest component is LVLM refinement, accounting for 983.50 seconds, or 61.9% of PSGR's total runtime. RPCS accounts for approximately 0.1%, and final graph encoding, matching, and ranking account for approximately 1.0%. PSGR's full runtime is approximately 2.87 times SGM's and 2.78 times SEMScene's. Thus, its additional cost is concentrated upstream rather than in the final matcher.

The table reports 66.03 GB of GPU memory usage for PSGR's evaluated pipeline and an image-side stored representation of 0.336 MB per image, approximately 336 MB for the 1,000-image gallery. Preprocessing throughput is computed before final graph encoding and retrieval. The per-image times divide each complete test-set stage by 1,000, including the workload of processing 5,000 captions.

Training cost and the GRPO increment. The adaptation cost remains 16.3 hours for SFT and 82.3 hours for GRPO on eight 80GB A100 GPUs, as reported in Appendix E.1.1. Table 2 measures GRPO's incremental retrieval benefit over SFT: +4.2 Rsum on Flickr30K, +1.5 on COCO 5-fold 1K, and +4.3 on COCO 5K. The new direct graph evaluation also shows a change in the retained graph outputs: relative to SFT + RPCS, GRPO + RPCS increases caption-grounded recall from 68.63% to 74.53%, increases grounding accuracy among semantic matches from 16.80% to 19.06%, and reduces the confirmed hallucination rate from 10.39% to 8.06% (Section 4.4, Table 4). The revised paper reports these improvements alongside the measured training and full-pipeline inference costs.

** All rates are percentages, and changes are percentage points. Grounding accuracy is evaluated among semantic matches.**

Graph sourceMatched triplets ↑\uparrowCaption-grounded precision ↑\uparrowCaption-grounded recall ↑\uparrowGrounding accuracy ↑\uparrowHallucination count ↓\downarrowHallucination rate ↓\downarrowUncertain count
Neural Motifs1226.7832.7113.1139021.6721
SFT + RPCS25614.2268.6316.8018710.3923
GRPO + RPCS27815.4474.5319.061458.0651

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response, Shared concerns and new evidence

We thank the AC, PC, and all three reviewers for their careful assessment and constructive suggestions. We appreciate their recognition of the following strengths:

  • Problem and formulation: PSGR addresses the semantic mismatch between generic visual scene graphs and caption-derived graphs through retrieval-oriented graph purification (2oPR, sxFx).
  • Results and presentation: the retrieval results are strong, and the figures clearly explain the pipeline (T9Jd).
  • Analysis and transparency: the appendix provides substantial analysis and clearly describes external supervision, transfer settings, and the scope of timing measurements (2oPR, sxFx).

We have provided a self-contained response to each reviewer and expanded the revised manuscript with the experiments summarized below. Our contribution remains improving visual scene graph representations for retrieval, using established matching networks.

Shared concerns and new evidence

  1. Separating graph improvements from the visual backbone. (2oPR-W1, T9Jd-W2)

We evaluated SGM, LGSGM, and SEMScene with the same DINOv2 backbone and compared generic VSG and PSG inputs within each matching architecture. PSG inputs improve Flickr30K Rsum by 6.00, 9.86, and 9.30, respectively, with higher R@1 in both directions for all three methods. We also report backbone-grouped comparisons under EfficientNet-B5 and DINOv2. The Introduction and Experiments distinguish backbone gains from graph-purification gains. Section 4.5, Table 5, and Appendix A.4, Table 10.

  1. Directly measuring the quality of the retained graphs. (T9Jd-W3/Q2, sxFx-W2)

We added an evaluation on 100 randomly sampled Flickr30K test images and 500 captions, using independently constructed, manually checked references and a common 18-relation budget. Compared with generic VSGs, Full GRPO + RPCS increases caption-grounded recall from 32.71% to 74.53% and grounding accuracy among semantic matches from 13.11% to 19.06%, while reducing the confirmed hallucination rate from 21.67% to 8.06%. The evaluation reports precision, recall, grounding, hallucinations, and uncertain cases, providing direct graph evidence alongside retrieval results. Section 4.4, Table 4, and Appendix A.5.

  1. Measuring RPCS gains across repeated runs and testing its design choices. (2oPR-W2/Q2)

We combined four additional retrieval-training seeds with the original run, reporting means and sample standard deviations over five runs with graph inputs and features fixed. Full RPCS improves mean ensemble Rsum over RSG top-rank by 1.15 on Flickr30K, 1.00 on COCO 5-fold 1K, and 2.31 on COCO 5K. We also tested joint RSG-fallback references and disabled coarse-type diversity independently. The default fallback reference gives higher ensemble Rsum in the evaluated runs, while removing coarse-type diversity reduces it by 2.6, 0.4, and 2.7 on the three protocols. Table 3 and Appendix A.6, Tables 11-15.

  1. Checking reward behavior and its connection to graph refinement. (2oPR-W4, sxFx-W1/W2)

We sampled eight responses for each of 7,000 images, yielding 56,000 outputs per model, and measured the frequency and reward contribution of two-negative cosine products. We additionally retrained with label-wise cosine clipping and positive-score order filtering. The modified full-reward results differ from the original by only 0.12-0.54 Rsum, and adding cross-modal alignment remains beneficial under both reward implementations. Our response also decomposes main and extra scores: the main-only score rises from 0.411380 after SFT to 0.547269 after Full GRPO, complementing the independent retained-graph evaluation. Appendix A.7, Tables 16-18, and the response to sxFx-W2.

  1. Reporting the complete computational cost. (2oPR-W3/Q1, T9Jd-W1/Q1)

We profiled all stages on 1,000 Flickr30K images and 5,000 captions using one 80GB H800 GPU, reporting runtime, FLOPs, GPU memory, preprocessing throughput, and image-cache storage for PSGR, SGM, LGSGM, and SEMScene. PSGR takes 1,588.15 seconds, or an amortized 1.59 seconds per image under this batched workload. LVLM refinement accounts for 61.9% of runtime, while RPCS accounts for approximately 0.1%. These measurements quantify the full-pipeline accuracy-cost trade-off beyond matching-stage timing. Section 4.6, Table 6, and Appendix A.12, Table 23.

AuthorsOpenReview ↗

Reviewer-specific clarifications, Rebuttal revisions

Reviewer-specific clarifications

  • Training and transfer protocol (2oPR-W1). We explicitly distinguish the Flickr30K-adapted, frozen LVLM from the retrieval matchers trained separately per benchmark. MS-COCO graph generation therefore uses external Flickr30K training supervision, and the cross-dataset experiment evaluates matcher transfer. Section 4.1 and Appendices A.2 and A.10.

  • Caption guidance and component roles (sxFx-W1, T9Jd-W3/Q3). We clarify that CG-VSG construction is a training-time process. Generic VSGs provide grounded structural anchors, caption-guided targets specify semantic priorities, Qwen applies the learned refinement without captions, and RPCS constructs fixed-budget inputs through static sampling. The cross-modal reward ablation starts from the same caption-guided SFT model and tests the additional alignment signal during GRPO. Section 3, Table 2, and Appendices D and E.

  • Supervision allocation (sxFx-W3). We added 20K/9K and 24K/5K SFT/GRPO allocations around the default 22K/7K split, fixing the total supervision at 29,000 samples and reporting the actual GRPO optimization steps. Among the tested allocations, 22K/7K achieves the highest Rsum for Local, Global, and Ensemble under all three evaluation protocols. Appendix A.13.1, Table 24.

  • Order reward and extra triplets (2oPR-W4, sxFx-W2). We clarify the relative-order objective and its maximum 0.1 contribution to the main-triplet reward. We retain extra triplets in supervision and generated outputs, and report their direct-concatenation experiment before visual GPO pooling. Dedicated fusion of text-only extra semantics remains a future direction. Appendices D.2.1 and A.8, Table 19.

Rebuttal revisions

The manuscript now includes direct graph-quality evaluation (Section 4.4), PSG inputs in existing matchers (Section 4.5), and full-pipeline computational costs (Section 4.6). The appendix adds the annotation protocol, backbone comparisons, repeated-run statistics, fallback and coarse-type tests, reward diagnostics and retraining, stage-wise profiling, and supervision-allocation experiments. We have also clarified the training/inference roles, graph budgets, and data-usage protocol. We respectfully ask that the revised manuscript and additional evidence be considered alongside our detailed responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.