- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper tries to address two major problems in scene graph generation: 1) the long-tail issue, which means a few predicates dominate the training set, resulting in a long tail of underrepresented predicates, and 2) models struggling with new triplet combinations, where a triplet means the (subject, predicate, object) combination that a scene graph predicts. To overcome these issues they propose a Dual-Cue Relation Modeling framework, which first filters candidate object pairs and then predicts each relationship with two branches: a visual-spatial branch that uses the conventional SGG inputs (detector appearance features, box geometry and union-region features), excluding the object label information, which goes to the second branch; and a semantic-context branch that uses object category information and features from a frozen pre-trained vision-language model (BLIP). They then use a predicate-adaptive router that combines the two branches for each candidate relationship. During training, the semantic branch is aligned with descriptions of each predicate generated by GPT and encoded by BLIP, and a contrastive loss separates predicates within each branch. Additionally, to address the long-tail issue they propose Synthetic Cue Augmentation (SCA), which creates extra training examples for rare predicates by transferring the variations seen in similar common predicates to real rare-predicate examples. They evaluate on VG150, OIV6 and GQA200, where SCA improves mean recall at some cost to overall recall.
Critical strengths
-
SCA addresses the long-tail problem on VG150 dataset: Across all 3 tasks for VG150 dataset SCA raises mean recall and F@k over the base model. This is crucial because the long-tail problem is the paper's main target.
-
Effective integration of pre-trained VLM features: The paper provides a practical way to integrate vision-language knowledge into a standard two-stage SGG pipeline.
Critical weaknesses
-
Comparison with prior methods not fair because of BLIP: The models used features from pre-trained VLM (BLIP), while the baselines in Table 1 do not use pre-trained knowledge. Also Table 4 shows that without BLIP the models mR@50/100 is 27.4/30.3 (2nd row from the last) which is below PE-NEt's 31.4/33.5 in Table 1. This is a major concern because the paper's advantage over prior methods might largely come from BLIP rather than the dual-branch design. Additionally, testing BLIP and SCA on conventional two-stage models such as Motifs or VCTree, as done for the methods in Table 1's "Model-Agnostic" section, or comparing against SGG methods that also use vision-language models, would clarify where the gains come from.
-
Zero-shot claim is overstated: Generalization to unseen triplet compositions is one of the main challenges the paper aims to address (abstract), but the evidence is limited. Zero-shot results are reported only on VG150, and only for "Ours," so it is unclear whether SCA is included (Table 2). Moreover, the visual-only branch has higher zero-shot recall than the full model (Table 4: 20.4/23.9 vs. 18.4/21.9), suggesting the proposed combination reduces rather than improves compositional generalization.
Questions for authors
-
BLIP fairness: Could the authors add BLIP features and SCA to one or two conventional two-stage models, such as Motifs or VCTree, as done for the methods in Table 1's "Model-Agnostic" section? Alternatively, could they compare against SGG methods that also use pretrained vision-language features (e.g., FMSGG)? If the proposed model still clearly outperforms these under equal conditions, the contribution of the dual-branch design would be established.
-
Zero-shot: Does Table 2 report results with or without SCA? Could the authors report zero-shot recall for both, and on GQA200 or OIV6 as well? Given that the visual-only branch has higher zero-shot recall than the full model (Table 4), how do the authors support the claim of improved zero-shot generalization? If the evidence doesn't hold, I'd ask that the claim be narrowed.
4 comments
Thank you for the detailed response and the new experiments. The matched BLIP comparison with Motifs and VCTree partly addresses my first concern, although with SCA applied to all models the margin becomes small; a comparison with SGG methods that already use vision-language features would have strengthened it. The zero-shot evidence also remains weak: the full model has lower zero-shot recall than the visual-only branch, and the new GQA200 and OIV6 comparisons use baselines without pretrained vision-language features. I therefore maintain my score.
We thank the reviewer for the careful follow-up. We appreciate the acknowledgment that our matched BLIP comparisons partly address this concern. To further strengthen the evidence, we have expanded Table 11 with additional SGG methods leveraging pretrained vision-language knowledge, including CaCao, AMCA+GCL, HIERCOM, ST-SGG+APT, MLLMP, FMSGG, and BiRef.
Table 1. VG150 results (%). All metrics are reported as @50/100. B: BLIP; S: SCA.
| Method | PredCls R | PredCls mR | PredCls F | SGCls R | SGCls mR | SGCls F | SGDet R | SGDet mR | SGDet F |
|---|---|---|---|---|---|---|---|---|---|
| Motifs | 65.5/67.2 | 16.5/17.8 | 26.4/28.1 | 39.0/39.7 | 8.7/9.3 | 14.2/15.1 | 32.1/36.9 | 5.7/6.6 | 9.7/11.2 |
| + B | 67.8/70.3 | 20.4/23.3 | 31.4/35.0 | 42.0/43.4 | 13.7/15.4 | 20.7/22.7 | 29.1/33.7 | 8.0/9.4 | 12.5/14.7 |
| + B + S | 55.5/60.7 | 32.0/36.0 | 40.6/45.2 | 31.2/34.9 | 19.9/22.8 | 24.3/27.6 | 23.4/27.4 | 11.2/13.6 | 15.1/18.2 |
| + AMCA + GCL | 46.1/48.1 | 41.4/43.8 | 43.6/45.8 | 29.3/30.1 | 24.4/26.2 | 26.6/28.0 | 18.1/21.8 | 17.1/19.8 | 17.6/20.8 |
| + CaCao | — | 37.1/38.9 | — | — | 23.3/24.4 | — | — | 17.1/20.0 | — |
| + HIERCOM | 69.5/75.6 | 25.1/30.6 | 36.9/43.6 | 41.6/44.8 | 14.6/17.3 | 21.6/25.0 | 28.3/33.9 | 8.5/10.6 | 13.1/16.2 |
| + ST-SGG + APT | 58.7/62.3 | 31.3/34.6 | 40.8/44.5 | 36.6/38.4 | 20.3/21.5 | 26.1/27.6 | 30.2/34.4 | 14.8/18.1 | 19.9/23.7 |
| + MLLMP | 60.4/62.3 | 28.9/30.6 | 39.1/41.0 | 37.2/38.2 | 17.3/18.1 | 23.6/24.6 | 29.6/34.3 | 13.2/15.7 | 18.3/21.5 |
| VCTree | 66.2/68.1 | 15.4/16.6 | 25.0/26.7 | 40.5/41.4 | 7.5/8.0 | 12.7/13.4 | 31.5/36.2 | 6.6/7.7 | 10.9/12.7 |
| + B | 68.0/70.3 | 21.5/24.2 | 32.7/36.0 | 46.2/47.4 | 13.8/15.1 | 21.3/22.9 | 29.8/33.9 | 8.1/9.4 | 12.7/14.7 |
| + B + S | 60.1/63.6 | 29.8/33.1 | 39.8/43.5 | 33.1/35.8 | 20.3/23.4 | 25.2/28.3 | 23.6/27.8 | 11.4/13.4 | 15.4/18.1 |
| + AMCA + GCL | 44.6/46.4 | 42.8/44.5 | 43.7/45.4 | 33.6/34.5 | 28.5/29.2 | 30.8/31.6 | 19.3/20.3 | 17.0/18.8 | 18.1/19.5 |
| + CaCao | — | 39.0/40.8 | — | — | 27.5/28.7 | — | — | 16.4/19.1 | — |
| + HIERCOM | 69.8/75.8 | 26.3/31.8 | 38.2/44.8 | 46.7/50.7 | 16.9/20.0 | 24.8/28.7 | 32.2/37.0 | 8.7/11.1 | 13.7/17.1 |
| + MLLMP | 60.8/62.9 | 29.1/30.6 | 39.4/41.2 | 43.2/44.5 | 20.8/21.8 | 28.1/29.3 | 28.7/33.1 | 12.4/14.6 | 17.3/20.3 |
| FMSGG (w/o debiasing) | 67.3/69.5 | 33.2/34.3 | 44.5/45.9 | 42.0/43.2 | 20.2/21.4 | 27.3/28.6 | 36.6/39.8 | 12.4/13.7 | 18.5/20.4 |
| FMSGG (Full) | 63.8/65.4 | 37.7/39.4 | 47.4/49.2 | 39.8/40.4 | 25.4/26.6 | 31.0/32.1 | 34.8/36.4 | 19.6/22.2 | 25.1/27.6 |
| BiRef | 61.0/63.4 | 37.7/40.4 | 46.6/49.4 | 44.9/46.3 | 24.9/26.7 | 32.0/33.9 | 29.5/33.6 | 15.1/17.6 | 20.0/23.1 |
| DCRM+B | 66.0/68.5 | 30.1/33.3 | 41.3/44.8 | 41.5/43.0 | 18.0/19.8 | 25.1/27.1 | 26.7/30.5 | 12.3/14.2 | 16.8/19.4 |
| DCRM+B+S | 60.0/63.4 | 34.3/38.2 | 43.6/47.6 | 34.8/36.3 | 23.1/25.0 | 27.8/29.6 | 24.5/28.4 | 13.9/16.4 | 17.7/20.8 |
Matched BLIP/SCA comparisons share the frozen detector, BLIP representations, text-alignment supervision, dataset splits, and evaluation protocol. External results follow their original settings.
Matched comparisons. Incorporating BLIP into Motifs and VCTree improves their performance, while SCA further enhances class-balanced recognition. Under identical BLIP supervision, DCRM consistently outperforms both enhanced baselines in mR and F across all three tasks. This advantage persists when SCA is equally applied to all frameworks, demonstrating that the improvements cannot be attributed solely to pretrained BLIP knowledge or synthetic augmentation.
Broader comparisons. DCRM also demonstrates competitive performance among SGG methods leveraging pretrained vision-language knowledge, including independent frameworks such as FMSGG and BiRef. Notably, BiRef additionally employs Relation NMS during PredCls and SGCls inference, which promotes more diverse predicate predictions and may benefit class-balanced recall. Overall, these comparisons further demonstrate the competitiveness of DCRM among existing vision-language-enhanced SGG methods.
We thank the reviewer again for encouraging these more comprehensive comparisons.
We thank the reviewer for the continued scrutiny of our zero-shot evidence. We have clarified our claims and expanded zR comparisons across VG150, GQA200, and OIV6.
Clarification of the visual-only result. We have revised our claims to distinguish the contributions of different components: DCRM improves the balance between overall and class-balanced recall; pretrained BLIP knowledge supports zero-shot generalization within the dual-branch framework; SCA primarily addresses long-tailed predicates. Accordingly, the full model's zR not consistently exceeding that of the visual-only branch does not contradict these revised claims, which do not require the full model to outperform every individual branch on every metric. While we regret that the full model does not achieve the best performance on every metric, we faithfully report and explicitly discuss these results to ensure that the limitations of our model remain transparent.
1. VG150 Zero-Shot Comparisons
We have expanded the VG150 zR comparison to include CaCao, AMCA+GCL, HIERCOM, FMSGG, BiRef, and additional matched BLIP/SCA baselines.
Table 1. VG150 zero-shot recall, zR@50/100 (%).
| Method | PredCls | SGCls | SGDet |
|---|---|---|---|
| Motifs | 7.3/9.4 | 0.7/1.7 | 0.0/0.3 |
| + BLIP | 15.6/18.3 | 3.5/4.8 | 1.5/2.0 |
| + BLIP+SCA | 15.5/18.4 | 3.3/4.7 | 1.4/2.0 |
| + CaCao | 12.0/13.1 | 5.1/5.8 | 3.6/3.9 |
| + HIERCOM | 18.4/25.5 | 4.9/6.4 | 2.1/3.1 |
| + AMCA+GCL | 3.7/4.7 | 1.4/2.3 | 0.6/1.5 |
| VCTree | 7.7/10.3 | 0.7/2.1 | 0.0/0.5 |
| + BLIP | 16.3/19.7 | 3.6/5.1 | 1.2/1.9 |
| + BLIP+SCA | 16.5/19.3 | 3.1/4.9 | 1.4/1.9 |
| + CaCao | 13.6/14.9 | 6.5/7.2 | 3.3/5.2 |
| + TDE+HIERCOM | 13.7/20.2 | 4.3/6.2 | 1.7/2.5 |
| + AMCA+GCL | 4.3/5.2 | 2.0/2.9 | 0.8/1.6 |
| FMSGG (w/o debiasing) | 14.8/16.9 | 4.8/7.4 | 3.6/5.7 |
| FMSGG (Full) | 21.2/23.8 | 8.8/11.6 | 5.3/7.7 |
| BiRef | 11.4/13.1 | 2.8/4.6 | 1.4/2.3 |
| DCRM+B | 18.4/21.9 | 6.6/7.7 | 2.0/2.6 |
| DCRM+B+S | 18.6/21.4 | 6.5/7.4 | 2.2/2.9 |
Under matched BLIP settings, DCRM+B consistently outperforms Motifs+BLIP and VCTree+BLIP in zR across all three tasks. This advantage persists under matched BLIP+SCA settings. The expanded comparisons further demonstrate that DCRM achieves competitive zero-shot performance among existing SGG methods leveraging pretrained vision-language knowledge.
2. GQA200 Zero-Shot Comparisons
We further extended our zero-shot comparisons to GQA200 by evaluating BLIP-enhanced Motifs/VCTree, AMCA+GCL variants, Motifs+CaCao, and BiRef.
Table 2. GQA200 zero-shot recall, zR@50/100 (%).
| Method | PredCls | SGCls | SGDet |
|---|---|---|---|
| SHA | 12.6/14.9 | 4.1/4.7 | 1.0/1.1 |
| + GCL | 1.6/2.3 | 0.4/0.5 | 0.2/0.2 |
| PE-Net | 8.1/9.9 | 1.3/1.5 | 0.2/0.3 |
| + RA-SGG | 9.3/11.3 | 3.3/4.0 | 0.9/1.4 |
| Motifs | |||
| + AMCA+GCL | 3.1/4.1 | 1.1/2.5 | 0.4/1.2 |
| + BLIP | 9.7/11.9 | 3.2/3.9 | 0.8/1.5 |
| + CaCao | 10.9/13.4 | 3.7/5.2 | 1.7/2.2 |
| VCTree | |||
| + AMCA+GCL | 3.4/4.6 | 1.3/2.7 | 0.6/1.8 |
| + BLIP | 10.1/12.2 | 3.0/3.8 | 0.9/1.5 |
| BiRef | 10.3/12.6 | 3.7/4.2 | 0.8/1.1 |
| DCRM | 13.0/16.4 | 4.6/5.8 | 1.5/2.1 |
| DCRM+SCA | 12.5/16.4 | 4.2/5.9 | 1.6/2.0 |
DCRM achieves higher zR@100 than both BLIP-enhanced Motifs and VCTree across all three tasks. The additional AMCA+GCL, CaCao, and BiRef results further broaden the comparisons with methods incorporating pretrained vision-language knowledge. These results demonstrate that DCRM maintains competitive zero-shot performance on GQA200.
3. OIV6 Zero-Shot Comparisons
We also supplemented the OIV6 evaluation with BLIP-enhanced Motifs and VCTree.
Table 3. OIV6 zero-shot recall, zR@50/100 (%).
| Method | zR@50/100 |
|---|---|
| RelTR | 16.5/16.5 |
| BGNN | 8.3/13.8 |
| SQUAT | 11.9/13.8 |
| PE-Net | 14.9/16.5 |
| Motifs+BLIP | 16.1/17.4 |
| VCTree+BLIP | 16.9/18.3 |
| DCRM | 20.2/23.0 |
| DCRM+SCA | 22.0/25.7 |
DCRM achieves competitive zero-shot performance compared with existing methods, including BLIP-enhanced Motifs and VCTree. These results provide further evidence of the framework's generalization ability across different datasets.
4. Reproduction Challenges
As noted in our previous response, very few existing SGG methods report zero-shot recall on GQA200 and OIV6. We therefore attempted to extend the VG150 comparisons to these datasets. However, several practical limitations remain: some methods were not originally evaluated on these datasets; complete implementations or checkpoints could not be located for FMSGG and MLLMP; and the released APT repository lacks essential model, loss, and configuration components.
Overall, these additional experiments provide broader evidence for our revised claims while transparently acknowledging the trade-off observed in the visual-only branch. We thank the reviewer for motivating these improvements.
We thank the reviewer for the constructive feedback. We have added controlled comparisons and extended zero-shot evaluations to clarify the respective contributions of DCRM, pretrained semantic knowledge, and SCA.
Q1. Comparison Fairness and the Contribution of DCRM
We equip Motifs and VCTree with the same frozen BLIP representations, predicate text prototypes, and text-alignment supervision as DCRM. We then evaluate all three frameworks under matched BLIP (B) and BLIP + SCA (B+S) settings.
Table R1. VG150 comparisons (F@100; B / B+S).
DCRM achieves the highest F@100 across all three tasks in both matched settings, supporting the contributions of heterogeneous cue modeling and adaptive coordination to class-balanced relation recognition. Complete results appear in Appendix B.9 (Table 11).
Q2. Zero-Shot Recall and Scope of Compositional Generalization
We agree with the reviewer's observation that the full model has lower zero-shot recall than the visual-only branch on VG150 PredCls (21.9 vs. 23.9 zR@100). Their relative strengths are distinct: the full model improves F@100 from 36.5 to 44.8, emphasizing balanced predicate recognition. We have clarified this distinction in the revised manuscript.
The zero-shot improvement attributable to pretrained semantic knowledge is established by the controlled BLIP comparison: within the same adaptive dual-cue architecture, BLIP components improve zR@50/100 from 14.5/18.3 to 18.4/21.9. Accordingly, our claims distinguish BLIP's contribution to compositional generalization from DCRM's and SCA's primary focus on balanced and long-tailed predicate recognition. VG150 zero-shot results before and after SCA are reported in Table 2.
To our knowledge, existing SGG studies primarily report zero-shot recall on VG, and we found no established zR comparisons on GQA200 or OIV6. Following the reviewer's suggestion, we extended the VG-style zero-shot evaluation protocol to both datasets and evaluated representative baselines under the same protocol.
Table R2. Zero-shot recall on GQA200 (zR@50/100).
Table R3. Zero-shot recall on OIV6 (SGDet).
On GQA200, DCRM achieves higher zR@50/100 than the compared methods across all three tasks. The OIV6 comparison further extends the evaluation, although its zero-shot subset contains only 109 relation instances, so these results are treated as supplementary evidence. Details appear in Appendix C.1 (Table 12) and Appendix C.2 (Table 14).