acceptodds
Under review as a conference paper at ICLR 2027

Dual-Cue Relation Modeling with Synthetic Cue Augmentation for Unbiased Scene Graph Generation

Abstract

Scene Graph Generation (SGG) represents visual scenes as structured subject-predicate-object triplets, yet relation prediction remains challenging due to semantic ambiguity, limited generalization to unseen triplet compositions, and long-tailed predicate distributions. To address these challenges of relational evidence in SGG, we propose a framework of Dual-Cue Relation Modeling (DCRM), which models visual-spatial and semantic-context evidence in separate relation branches. To enhance semantic modeling and compositional generalization, we further incorporate pretrained encoders into this dual-cue framework, leveraging its cross-modal image representations and encoded predicate descriptions to provide semantic guidance and fine-grained predicate prototypes. An adaptive router dynamically coordinates the predictions of the two branches, enabling effective integration of the resulting heterogeneous cues. To address the long-tailed distribution, we further introduce Synthetic Cue Augmentation (SCA), which performs semantically guided local variation transfer from data-rich to underrepresented predicates in both cue spaces. Extensive experiments on VG150, OIV6, and GQA200 demonstrate competitive overall and balanced performance of the proposed framework, with improved long-tailed predicate recognition and zero-shot generalization.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Scene Graph Generation, Relation Representation and Modeling, Cue Augmentation, Long-tailed Predicate Distributions

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_kP8XOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tries to address two major problems in scene graph generation: 1) the long-tail issue, which means a few predicates dominate the training set, resulting in a long tail of underrepresented predicates, and 2) models struggling with new triplet combinations, where a triplet means the (subject, predicate, object) combination that a scene graph predicts. To overcome these issues they propose a Dual-Cue Relation Modeling framework, which first filters candidate object pairs and then predicts each relationship with two branches: a visual-spatial branch that uses the conventional SGG inputs (detector appearance features, box geometry and union-region features), excluding the object label information, which goes to the second branch; and a semantic-context branch that uses object category information and features from a frozen pre-trained vision-language model (BLIP). They then use a predicate-adaptive router that combines the two branches for each candidate relationship. During training, the semantic branch is aligned with descriptions of each predicate generated by GPT and encoded by BLIP, and a contrastive loss separates predicates within each branch. Additionally, to address the long-tail issue they propose Synthetic Cue Augmentation (SCA), which creates extra training examples for rare predicates by transferring the variations seen in similar common predicates to real rare-predicate examples. They evaluate on VG150, OIV6 and GQA200, where SCA improves mean recall at some cost to overall recall.

Critical strengths
  1. SCA addresses the long-tail problem on VG150 dataset: Across all 3 tasks for VG150 dataset SCA raises mean recall and F@k over the base model. This is crucial because the long-tail problem is the paper's main target.

  2. Effective integration of pre-trained VLM features: The paper provides a practical way to integrate vision-language knowledge into a standard two-stage SGG pipeline.

Critical weaknesses
  1. Comparison with prior methods not fair because of BLIP: The models used features from pre-trained VLM (BLIP), while the baselines in Table 1 do not use pre-trained knowledge. Also Table 4 shows that without BLIP the models mR@50/100 is 27.4/30.3 (2nd row from the last) which is below PE-NEt's 31.4/33.5 in Table 1. This is a major concern because the paper's advantage over prior methods might largely come from BLIP rather than the dual-branch design. Additionally, testing BLIP and SCA on conventional two-stage models such as Motifs or VCTree, as done for the methods in Table 1's "Model-Agnostic" section, or comparing against SGG methods that also use vision-language models, would clarify where the gains come from.

  2. Zero-shot claim is overstated: Generalization to unseen triplet compositions is one of the main challenges the paper aims to address (abstract), but the evidence is limited. Zero-shot results are reported only on VG150, and only for "Ours," so it is unclear whether SCA is included (Table 2). Moreover, the visual-only branch has higher zero-shot recall than the full model (Table 4: 20.4/23.9 vs. 18.4/21.9), suggesting the proposed combination reduces rather than improves compositional generalization.

Questions for authors
  1. BLIP fairness: Could the authors add BLIP features and SCA to one or two conventional two-stage models, such as Motifs or VCTree, as done for the methods in Table 1's "Model-Agnostic" section? Alternatively, could they compare against SGG methods that also use pretrained vision-language features (e.g., FMSGG)? If the proposed model still clearly outperforms these under equal conditions, the contribution of the dual-branch design would be established.

  2. Zero-shot: Does Table 2 report results with or without SCA? Could the authors report zero-shot recall for both, and on GQA200 or OIV6 as well? Given that the visual-only branch has higher zero-shot recall than the full model (Table 4), how do the authors support the claim of improved zero-shot generalization? If the evidence doesn't hold, I'd ask that the claim be narrowed.

4 comments

We thank the reviewer for the constructive feedback. We have added controlled comparisons and extended zero-shot evaluations to clarify the respective contributions of DCRM, pretrained semantic knowledge, and SCA.

Q1. Comparison Fairness and the Contribution of DCRM

We equip Motifs and VCTree with the same frozen BLIP representations, predicate text prototypes, and text-alignment supervision as DCRM. We then evaluate all three frameworks under matched BLIP (B) and BLIP + SCA (B+S) settings.

Table R1. VG150 comparisons (F@100; B / B+S).

MethodPredClsSGClsSGDet
Motifs35.0 / 45.222.7 / 27.514.7 / 18.2
VCTree36.1 / 43.522.9 / 28.314.7 / 18.1
DCRM44.8 / 47.627.1 / 29.619.4 / 20.8

DCRM achieves the highest F@100 across all three tasks in both matched settings, supporting the contributions of heterogeneous cue modeling and adaptive coordination to class-balanced relation recognition. Complete results appear in Appendix B.9 (Table 11).

Q2. Zero-Shot Recall and Scope of Compositional Generalization

We agree with the reviewer's observation that the full model has lower zero-shot recall than the visual-only branch on VG150 PredCls (21.9 vs. 23.9 zR@100). Their relative strengths are distinct: the full model improves F@100 from 36.5 to 44.8, emphasizing balanced predicate recognition. We have clarified this distinction in the revised manuscript.

The zero-shot improvement attributable to pretrained semantic knowledge is established by the controlled BLIP comparison: within the same adaptive dual-cue architecture, BLIP components improve zR@50/100 from 14.5/18.3 to 18.4/21.9. Accordingly, our claims distinguish BLIP's contribution to compositional generalization from DCRM's and SCA's primary focus on balanced and long-tailed predicate recognition. VG150 zero-shot results before and after SCA are reported in Table 2.

To our knowledge, existing SGG studies primarily report zero-shot recall on VG, and we found no established zR comparisons on GQA200 or OIV6. Following the reviewer's suggestion, we extended the VG-style zero-shot evaluation protocol to both datasets and evaluated representative baselines under the same protocol.

Table R2. Zero-shot recall on GQA200 (zR@50/100).

MethodPredClsSGClsSGDet
SHA12.6 / 14.94.1 / 4.71.0 / 1.1
SHA + GCL1.6 / 2.30.4 / 0.50.2 / 0.2
PE-Net8.1 / 9.91.3 / 1.50.2 / 0.3
PE-Net + RA-SGG9.3 / 11.33.3 / 4.00.9 / 1.4
DCRM13.0 / 16.44.6 / 5.81.5 / 2.1
DCRM + SCA12.5 / 16.44.2 / 5.91.6 / 2.0

Table R3. Zero-shot recall on OIV6 (SGDet).

MethodzR@50zR@100
RelTR16.516.5
BGNN8.313.8
SQUAT11.913.8
PE-Net14.916.5
DCRM20.223.0
DCRM + SCA22.025.7

On GQA200, DCRM achieves higher zR@50/100 than the compared methods across all three tasks. The OIV6 comparison further extends the evaluation, although its zero-shot subset contains only 109 relation instances, so these results are treated as supplementary evidence. Details appear in Appendix C.1 (Table 12) and Appendix C.2 (Table 14).

Thank you for the detailed response and the new experiments. The matched BLIP comparison with Motifs and VCTree partly addresses my first concern, although with SCA applied to all models the margin becomes small; a comparison with SGG methods that already use vision-language features would have strengthened it. The zero-shot evidence also remains weak: the full model has lower zero-shot recall than the visual-only branch, and the new GQA200 and OIV6 comparisons use baselines without pretrained vision-language features. I therefore maintain my score.

We thank the reviewer for the careful follow-up. We appreciate the acknowledgment that our matched BLIP comparisons partly address this concern. To further strengthen the evidence, we have expanded Table 11 with additional SGG methods leveraging pretrained vision-language knowledge, including CaCao, AMCA+GCL, HIERCOM, ST-SGG+APT, MLLMP, FMSGG, and BiRef.

Table 1. VG150 results (%). All metrics are reported as @50/100. B: BLIP; S: SCA.

MethodPredCls RPredCls mRPredCls FSGCls RSGCls mRSGCls FSGDet RSGDet mRSGDet F
Motifs65.5/67.216.5/17.826.4/28.139.0/39.78.7/9.314.2/15.132.1/36.95.7/6.69.7/11.2
 + B67.8/70.320.4/23.331.4/35.042.0/43.413.7/15.420.7/22.729.1/33.78.0/9.412.5/14.7
 + B + S55.5/60.732.0/36.040.6/45.231.2/34.919.9/22.824.3/27.623.4/27.411.2/13.615.1/18.2
 + AMCA + GCL46.1/48.141.4/43.843.6/45.829.3/30.124.4/26.226.6/28.018.1/21.817.1/19.817.6/20.8
 + CaCao—37.1/38.9——23.3/24.4——17.1/20.0—
 + HIERCOM69.5/75.625.1/30.636.9/43.641.6/44.814.6/17.321.6/25.028.3/33.98.5/10.613.1/16.2
 + ST-SGG + APT58.7/62.331.3/34.640.8/44.536.6/38.420.3/21.526.1/27.630.2/34.414.8/18.119.9/23.7
 + MLLMP60.4/62.328.9/30.639.1/41.037.2/38.217.3/18.123.6/24.629.6/34.313.2/15.718.3/21.5
VCTree66.2/68.115.4/16.625.0/26.740.5/41.47.5/8.012.7/13.431.5/36.26.6/7.710.9/12.7
 + B68.0/70.321.5/24.232.7/36.046.2/47.413.8/15.121.3/22.929.8/33.98.1/9.412.7/14.7
 + B + S60.1/63.629.8/33.139.8/43.533.1/35.820.3/23.425.2/28.323.6/27.811.4/13.415.4/18.1
 + AMCA + GCL44.6/46.442.8/44.543.7/45.433.6/34.528.5/29.230.8/31.619.3/20.317.0/18.818.1/19.5
 + CaCao—39.0/40.8——27.5/28.7——16.4/19.1—
 + HIERCOM69.8/75.826.3/31.838.2/44.846.7/50.716.9/20.024.8/28.732.2/37.08.7/11.113.7/17.1
 + MLLMP60.8/62.929.1/30.639.4/41.243.2/44.520.8/21.828.1/29.328.7/33.112.4/14.617.3/20.3
FMSGG (w/o debiasing)67.3/69.533.2/34.344.5/45.942.0/43.220.2/21.427.3/28.636.6/39.812.4/13.718.5/20.4
FMSGG (Full)63.8/65.437.7/39.447.4/49.239.8/40.425.4/26.631.0/32.134.8/36.419.6/22.225.1/27.6
BiRef61.0/63.437.7/40.446.6/49.444.9/46.324.9/26.732.0/33.929.5/33.615.1/17.620.0/23.1
DCRM+B66.0/68.530.1/33.341.3/44.841.5/43.018.0/19.825.1/27.126.7/30.512.3/14.216.8/19.4
DCRM+B+S60.0/63.434.3/38.243.6/47.634.8/36.323.1/25.027.8/29.624.5/28.413.9/16.417.7/20.8

Matched BLIP/SCA comparisons share the frozen detector, BLIP representations, text-alignment supervision, dataset splits, and evaluation protocol. External results follow their original settings.

Matched comparisons. Incorporating BLIP into Motifs and VCTree improves their performance, while SCA further enhances class-balanced recognition. Under identical BLIP supervision, DCRM consistently outperforms both enhanced baselines in mR and F across all three tasks. This advantage persists when SCA is equally applied to all frameworks, demonstrating that the improvements cannot be attributed solely to pretrained BLIP knowledge or synthetic augmentation.

Broader comparisons. DCRM also demonstrates competitive performance among SGG methods leveraging pretrained vision-language knowledge, including independent frameworks such as FMSGG and BiRef. Notably, BiRef additionally employs Relation NMS during PredCls and SGCls inference, which promotes more diverse predicate predictions and may benefit class-balanced recall. Overall, these comparisons further demonstrate the competitiveness of DCRM among existing vision-language-enhanced SGG methods.

We thank the reviewer again for encouraging these more comprehensive comparisons.

We thank the reviewer for the continued scrutiny of our zero-shot evidence. We have clarified our claims and expanded zR comparisons across VG150, GQA200, and OIV6.

Clarification of the visual-only result. We have revised our claims to distinguish the contributions of different components: DCRM improves the balance between overall and class-balanced recall; pretrained BLIP knowledge supports zero-shot generalization within the dual-branch framework; SCA primarily addresses long-tailed predicates. Accordingly, the full model's zR not consistently exceeding that of the visual-only branch does not contradict these revised claims, which do not require the full model to outperform every individual branch on every metric. While we regret that the full model does not achieve the best performance on every metric, we faithfully report and explicitly discuss these results to ensure that the limitations of our model remain transparent.

1. VG150 Zero-Shot Comparisons

We have expanded the VG150 zR comparison to include CaCao, AMCA+GCL, HIERCOM, FMSGG, BiRef, and additional matched BLIP/SCA baselines.

Table 1. VG150 zero-shot recall, zR@50/100 (%).

MethodPredClsSGClsSGDet
Motifs7.3/9.40.7/1.70.0/0.3
+ BLIP15.6/18.33.5/4.81.5/2.0
+ BLIP+SCA15.5/18.43.3/4.71.4/2.0
+ CaCao12.0/13.15.1/5.83.6/3.9
+ HIERCOM18.4/25.54.9/6.42.1/3.1
+ AMCA+GCL3.7/4.71.4/2.30.6/1.5
VCTree7.7/10.30.7/2.10.0/0.5
+ BLIP16.3/19.73.6/5.11.2/1.9
+ BLIP+SCA16.5/19.33.1/4.91.4/1.9
+ CaCao13.6/14.96.5/7.23.3/5.2
+ TDE+HIERCOM13.7/20.24.3/6.21.7/2.5
+ AMCA+GCL4.3/5.22.0/2.90.8/1.6
FMSGG (w/o debiasing)14.8/16.94.8/7.43.6/5.7
FMSGG (Full)21.2/23.88.8/11.65.3/7.7
BiRef11.4/13.12.8/4.61.4/2.3
DCRM+B18.4/21.96.6/7.72.0/2.6
DCRM+B+S18.6/21.46.5/7.42.2/2.9

Under matched BLIP settings, DCRM+B consistently outperforms Motifs+BLIP and VCTree+BLIP in zR across all three tasks. This advantage persists under matched BLIP+SCA settings. The expanded comparisons further demonstrate that DCRM achieves competitive zero-shot performance among existing SGG methods leveraging pretrained vision-language knowledge.

2. GQA200 Zero-Shot Comparisons

We further extended our zero-shot comparisons to GQA200 by evaluating BLIP-enhanced Motifs/VCTree, AMCA+GCL variants, Motifs+CaCao, and BiRef.

Table 2. GQA200 zero-shot recall, zR@50/100 (%).

MethodPredClsSGClsSGDet
SHA12.6/14.94.1/4.71.0/1.1
+ GCL1.6/2.30.4/0.50.2/0.2
PE-Net8.1/9.91.3/1.50.2/0.3
+ RA-SGG9.3/11.33.3/4.00.9/1.4
Motifs
+ AMCA+GCL3.1/4.11.1/2.50.4/1.2
+ BLIP9.7/11.93.2/3.90.8/1.5
+ CaCao10.9/13.43.7/5.21.7/2.2
VCTree
+ AMCA+GCL3.4/4.61.3/2.70.6/1.8
+ BLIP10.1/12.23.0/3.80.9/1.5
BiRef10.3/12.63.7/4.20.8/1.1
DCRM13.0/16.44.6/5.81.5/2.1
DCRM+SCA12.5/16.44.2/5.91.6/2.0

DCRM achieves higher zR@100 than both BLIP-enhanced Motifs and VCTree across all three tasks. The additional AMCA+GCL, CaCao, and BiRef results further broaden the comparisons with methods incorporating pretrained vision-language knowledge. These results demonstrate that DCRM maintains competitive zero-shot performance on GQA200.

3. OIV6 Zero-Shot Comparisons

We also supplemented the OIV6 evaluation with BLIP-enhanced Motifs and VCTree.

Table 3. OIV6 zero-shot recall, zR@50/100 (%).

MethodzR@50/100
RelTR16.5/16.5
BGNN8.3/13.8
SQUAT11.9/13.8
PE-Net14.9/16.5
Motifs+BLIP16.1/17.4
VCTree+BLIP16.9/18.3
DCRM20.2/23.0
DCRM+SCA22.0/25.7

DCRM achieves competitive zero-shot performance compared with existing methods, including BLIP-enhanced Motifs and VCTree. These results provide further evidence of the framework's generalization ability across different datasets.

4. Reproduction Challenges

As noted in our previous response, very few existing SGG methods report zero-shot recall on GQA200 and OIV6. We therefore attempted to extend the VG150 comparisons to these datasets. However, several practical limitations remain: some methods were not originally evaluated on these datasets; complete implementations or checkpoints could not be located for FMSGG and MLLMP; and the released APT repository lacks essential model, loss, and configuration components.

Overall, these additional experiments provide broader evidence for our revised claims while transparently acknowledging the trade-off observed in the visual-only branch. We thank the reviewer for motivating these improvements.

Reviewer_dkGoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for unbiased scene graph generation that combines DCRM with SCA. DCRM models visual-spatial and semantic-context information in two separate branches. The visual branch uses appearance and geometric features, while the semantic branch incorporates object semantics, pretrained vision-language representations, and text-based predicate prototypes. An adaptive router combines the predictions from both branches. To address the long-tailed predicate distribution, SCA transfers local feature variations from frequent predicates to semantically related underrepresented predicates. The synthesized features are then used to fine-tune the predicate classifiers and router. Experiments on VG150, OIV6, and GQA200 show improvements in class-balanced relation recognition.

Critical strengths
  1. The motivation is reasonable. Visual appearance, relative geometry, and semantic context provide different information for relation prediction. Modeling these cues separately is a reasonable approach, particularly when distinguishing visually similar predicates.
  2. The combination of pretrained vision-language features and predicate descriptions is useful. The semantic branch benefits from additional information beyond conventional detector features, and the ablations suggest that these representations improve relation classification.
  3. SCA improves class-balanced recognition without adding inference-time modules. The gains in mR@K and F@K across VG150 and GQA200 are encouraging. The results on OIV6 are also competitive, particularly in weighted mAP and the official evaluation score.
Critical weaknesses
  1. Appendix B.6 shows that SCA works better than simple resampling, and Appendix B.5 studies the number of donors and submodes. However, it is still unclear whether semantic donor selection and local clustering are actually necessary. For example, how would the method perform with randomly selected donors or without clustering?
  2. The stronger tail performance of the semantic branch in Figure 4 could therefore arise from its loss rather than its information source. Table 4 compares input components and fusion strategies, but does not use identical losses across branches or swap their losses.
  3. Eq. (15) selects the best donor mode, but Appendix B.3 says that three submodes are selected for sampling. Which procedure is actually used? The authors should also specify the number of clusters, how many synthetic features are generated per sample, and the ratio of real to synthetic features during training.
  4. Figure 5 reports 47.4 for PredCls F@100, while Tables 1 and 6 report 47.6. The text also states that the best checkpoint is at iteration 1,130, but this point is not shown in Figure 6.
  5. Why the fused model performs worse than the visual-only branch on zero-shot relations and whether the adaptive router relies too heavily on category-level semantic priors.
Questions for authors

Please refer to the weaknesses.

1 comment

We thank the reviewer for the careful assessment. We have added targeted ablations, clarified SCA implementation, and corrected the figure annotations.

Q1. Semantic Donor Selection and Local Clustering

We added a three-seed PredCls ablation examining the contributions of donor selection and local clustering.

Table R1. SCA component ablation on VG150 PredCls (@100; mean ± SD).

VariantRmRF
Without clustering59.80 ± 2.8238.20 ± 0.4446.60 ± 0.56
Random donors63.20 ± 0.3638.00 ± 0.1047.46 ± 0.14
Full SCA63.27 ± 0.2338.27 ± 0.1247.69 ± 0.02

Semantic donor selection gives a modest F@100 improvement over random selection (47.69 vs. 47.46), whereas local clustering has a larger effect (47.69 vs. 46.60 without clustering). The three-seed results also show lower variation for Full SCA. These findings support local-mode modeling, with a more modest contribution from donor selection (Appendix B.5, Table 7).

Q2. Cue Complementarity versus CE/WCE Supervision

To distinguish the contributions of heterogeneous cues from differentiated classification supervision, we compare V/V and V/S configurations under matched and exchanged CE/WCE objectives.

Table R2. Ablation of branch types and supervision on VG150 (F@100).

BranchSupervisionPredClsSGClsSGDet
V/VCE/CE37.219.615.8
V/SCE/CE42.826.219.1
V/VCE/WCE39.123.918.0
V/SCE/WCE44.827.119.4
V/SWCE/CE41.824.618.9

Under identical CE supervision, V/S improves F@100 over V/V by 5.6, 6.6, and 3.3 points across PredCls, SGCls, and SGDet. The gains persist under CE/WCE supervision (5.7, 3.2, and 1.4 points), confirming that the benefits of heterogeneous cues extend beyond differentiated classification losses.

Furthermore, Figure 9 (Appendix B.10) shows distinct predicate-level strengths of the visual and semantic branches under identical CE supervision.

Q3. SCA Sampling and Implementation

We have clarified the SCA procedure in Appendix B.3. For each tail predicate, SCA selects three semantically related donor predicates and identifies local variation modes independently in the two cue spaces. Each synthetic relation is generated by transferring a compatible, bounded residual to the target features. The real-to-synthetic ratio is not fixed but determined by class-dependent sampling and retention probabilities. During adaptation, the cue representations remain frozen, while the classifiers and router are fine-tuned.

Q4. Figure Consistency and Checkpoint Selection

We corrected Figure 5 to show 47.6 PredCls F@100 for DCRM+SCA, consistent with the main results. Figure 6 now explicitly includes iteration 1,130, which corresponds to the selected checkpoint.

The adaptation curve shows validation performance, while the main tables report test-set results from the validation-selected checkpoint (Appendices B.4–B.5).

Q5. Zero-Shot Recall and Router Reliance

We agree that the full model has lower VG150 PredCls zR@100 than the visual-only branch (21.9 vs. 23.9). The full model instead improves F@100 from 36.5 to 44.8.

In a controlled comparison within the same adaptive dual-cue architecture, incorporating BLIP improves zR@50/100 from 14.5/18.3 to 18.4/21.9, identifying the contribution of pretrained semantic knowledge.

The router does not rely solely on class priors: instance-level reliability also contributes to its performance. Under PredCls, class-only routing achieves 44.0 F@100, compared with 44.8 for the full router; under SGDet, the corresponding values are 18.9 and 19.4 (Appendix B.8, Table 10).

We have clarified that DCRM and SCA primarily target balanced and long-tailed predicate recognition, rather than uniform improvements in zero-shot recall.

Reviewer_JuQMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes separate visual-spatial and semantic-context branches for long-tailed scene graph generation, combining their predictions through adaptive routing. Pretrained vision-language representations supply semantic guidance. Synthetic Cue Augmentation (SCA) transfers local feature residuals from related frequent predicates to underrepresented predicates. Experiments evaluate recognition and unseen triplet compositions on VG150, OIV6, and GQA200.

Critical strengths
  1. Useful performance in several settings. The method achieves the best listed PredCls/SGCls F@100 scores, 47.6/29.6, and OIV6 weighted score, 47.3, supporting its practical value.
  2. Practical design and informative ablations. Table 4 examines branches, pretrained components, and fusion strategies. SCA leaves the inference architecture unchanged, making its training-time improvements potentially useful.
Critical weaknesses
  1. The methodological advance remains insufficiently demonstrated. Dual-branch learning, pretrained semantic guidance, and feature augmentation have precedents in DHL, FMSGG, and CFA. Potential novelty lies in cue-specific organization and compatible residual transfer, but their distinctive value is not isolated under SGDet. Table 4 covers only PredCls/SGCls, with different CE/WCE supervision across branches.
  2. VG150 SGDet shows no clear advantage over strong comparators. +SCA obtains R@100/mR@100/F@100 of 28.4/16.4/20.8, versus SQUAT’s 28.9/16.5/21.0. Reported SGDet zR@50/100 is 2.0/2.6, below PE-Net’s 2.3/3.6, without a separate +SCA result. Evidence for improved detection and compositional generalization therefore remains incomplete.
Questions for authors
  1. The component-wise ablations in Table 4 are limited to PredCls and SGCls. Could you provide the complete ablation study on SGDet?
  2. Does the benefit of separating visual-spatial and semantic-context inputs persist when the two branches use matched classification supervision?
  3. Keeping the original CE/WCE supervision unchanged, how does SGDet performance change when the router in Equation (8) uses only the class prior, without the relation-level reliability descriptor?
3 comments

We thank the reviewer for the constructive assessment. We clarified methodological distinctions and added controlled experiments, particularly for SGDet.

Methodological Novelty

Our contribution lies in how heterogeneous relation evidence and local feature variations are modeled, rather than in using two branches, pretrained semantics, or augmentation alone.

DHL organizes coarse-/fine-grained branches for long-tailed learning via curriculum learning and distillation. DCRM instead learns visual-spatial and semantic-context relation representations separately; both branches operate at inference and are coordinated by instance reliability and predicate-level priors.

FMSGG extracts pairwise relation embeddings from CLIP using relation prompts. DCRM instead integrates region-conditioned vision-language features into an independently learned semantic-context decoder, jointly trained with a visual-spatial decoder and fused at the prediction level.

CFA augments triplets through entity-feature replacement and feature mixing. SCA transfers bounded, locally compatible residual variations from semantically related data-rich predicates to tail predicates, independently in both cue spaces while preserving target labels.

Tables 5, 7, 10, and 11 provide supporting cue, SCA, routing, and matched BLIP/SCA analyses.

Q1. Component Contributions under SGDet

We apologize that we could not complete the full Table 4 component-wise SGDet ablation within the revision period. We instead prioritized three complementary SGDet analyses: branch/supervision ablations (Table 5), routing ablations (Table 10), and matched BLIP/SCA comparisons with Motifs and VCTree (Table 11).

Under the same frozen detector, BLIP features, text prototypes, and alignment supervision, SGDet F@100 is 19.4 for DCRM versus 14.7/14.7 for Motifs/VCTree. After SCA is applied to all three, the values are 20.8/18.2/18.1. Together, these experiments substantiate the cue-modeling and routing contributions on SGDet.

Q2. Heterogeneous Cues versus Classification Supervision

Table R1. VG150 F@100 by branch type and supervision.

BranchLossPredClsSGClsSGDet
V/VCE/CE37.219.615.8
V/SCE/CE42.826.219.1
V/VCE/WCE39.123.918.0
V/SCE/WCE44.827.119.4
V/SWCE/CE41.824.618.9

Under identical CE/CE supervision, V/S improves F@100 over V/V by 5.6/6.6/3.3 points across PredCls/SGCls/SGDet; under CE/WCE, gains are 5.7/3.2/1.4. V/S with CE/WCE has the highest mR and F among tested settings. Figure 9 (Appendix B.10) further shows complementary predicate-level strengths under identical CE supervision.

Q3. Class Priors and Instance Reliability

We change only the router, keeping V/S, CE/WCE, and no SCA fixed.

Table R2. VG150 routing ablations (@100; R/mR/F).

RouterPredClsSGDet
Per-class67.3 / 32.7 / 44.030.3 / 13.6 / 18.7
Instance-only68.9 / 31.5 / 43.231.0 / 13.2 / 18.5
Class-only68.1 / 32.5 / 44.030.4 / 13.7 / 18.9
Full68.5 / 33.3 / 44.830.5 / 14.2 / 19.4

Instance-only gives the highest R; class-only improves mR over instance-only. The full router yields the highest mR and F in both tasks, improving F@100 over class-only by 0.8 (PredCls) and 0.5 (SGDet) points (Table 10).

SGDet Performance and Zero-Shot Generalization

We agree that DCRM+SCA achieves 20.8 F@100 on VG150 SGDet, below SQUAT's 21.0; its 2.9 zR@100 is also below PE-Net's 3.6. We have narrowed the claims accordingly. Within the same dual-cue architecture, BLIP improves VG150 PredCls zR@50/100 from 14.5/18.3 to 18.4/21.9. DCRM and SCA primarily target balanced and long-tailed predicate recognition.

We also extended zero-shot evaluation to GQA200 and OIV6:

Table R3. GQA200 zR@50/100.

MethodPredClsSGClsSGDet
SHA12.6/14.94.1/4.71.0/1.1
SHA+GCL1.6/2.30.4/0.50.2/0.2
PE-Net8.1/9.91.3/1.50.2/0.3
PE-Net+RA-SGG9.3/11.33.3/4.00.9/1.4
DCRM13.0/16.44.6/5.81.5/2.1
DCRM+SCA12.5/16.44.2/5.91.6/2.0

Table R4. OIV6 SGDet zR@50/100.

MethodzR@50/100
RelTR16.5/16.5
BGNN8.3/13.8
SQUAT11.9/13.8
PE-Net14.9/16.5
DCRM20.2/23.0
DCRM+SCA22.0/25.7

These are supplementary cross-dataset results (Appendix C.1, Table 12; C.2, Table 14). OIV6 contains only 109 zero-shot relation instances under our protocol, so we do not draw broad generalization conclusions from it.

Thank you for the clarifications and additional experiments.

  1. DRM baseline comparison. The DRM row in Table 1 matches the original paper’s GQA200 results across all three tasks, although Table 1 reports results on VG150. Using DRM’s published VG150 results, DCRM + SCA matches DRM on PredCls F@100 (47.6 for both), leads by 0.8 points on SGCls (29.6 vs. 28.8), and trails by 2.7 points on SGDet (20.8 vs. 23.5). The PredCls tie also illustrates a trade-off: DCRM + SCA has higher R@100 (63.4 vs. 45.8), and DRM has higher mR@100 (49.6 vs. 38.2). Please correct the DRM row and update Figure 5 accordingly. On OIV6, Table 3 reports a weighted score of 47.3, compared with 47.9 reported for DRM without DKT. Please include this comparison or qualify the “highest score” claim in Section 4.1.

  2. Interpretation of the zero-shot results. In Table 5, with cross-entropy supervision on both branches, replacing the second visual branch with a semantic branch improves F@100 by 5.6, 6.6, and 3.3 points on PredCls, SGCls, and SGDet, respectively. In that same comparison, PredCls zR@100 falls from 23.6 to 22.0. Table 4 isolates a different effect: adding BLIP to the adaptive dual-branch model raises PredCls zR@100 from 18.3 to 21.9. I suggest making these conclusions explicit. The heterogeneous branches improve the balance between overall and mean recall across the three tasks, and BLIP improves zero-shot recall within the dual-branch model on VG150 PredCls.

We sincerely thank the reviewer for the careful and insightful follow-up. We greatly appreciate the time and effort you have devoted to examining our work. Your comments have helped us identify important issues and improve the accuracy and clarity of our paper. All corresponding revisions are highlighted in blue in the updated manuscript. Thank you again for your thoughtful feedback.

Reviewer_4JZcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses long-tailed scene graph generation through separate visual–spatial and semantic–context relation branches. The first uses detector-region and geometric evidence. The second incorporates object context and frozen BLIP image/text representations, including text prototypes obtained from predicate descriptions. A router uses both instance-level reliability descriptors and class priors to coordinate their predictions. The experiments are sound.

Critical strengths
  1. Local visual/geometric and semantic/contextual signals offer different information. Decision-level coordination and training-time tail adaptation form a plausible engineering solution.
  2. Informative experiments illustrates the effectiveness of proposed model, along with each component.
  3. Trade-offs and limitations are disclosed, which deeply explained the effectiveness and limitations of the model.
Critical weaknesses
  1. The theoretical support of the model is not sufficient. The motivation of module designment is not fully illustrated, which causes slight logic breaks.
  2. Section 3.2 uses CE for the visual branch and WCE for the semantic branch. Consequently, more balanced semantic-branch predictions cannot be attributed to evidence type alone. Table 4 provides useful representation/module ablations, but does not fully separate cue complementarity from the effects of different losses.
Questions for authors
  1. With capacity and training budget held fixed, how do identical or exchanged CE/WCE objectives affect the dual-cue R/mR/zR gains? Which predicates or instances demonstrate complementary errors?
  2. How much benefit comes from router class priors versus instance reliability? Does the full router reliably outperform a simpler learnable per-class fusion?
  3. Can the complete training/adaptation configuration, exact predicate descriptions, WCE/temperature/clustering rules and synthesis-sampling settings be provided to make these comparisons reproducible?
2 comments

We thank the reviewer for the constructive comments and positive assessment of our work. We have conducted additional controlled experiments and expanded the implementation details to clarify the contributions of heterogeneous cues, adaptive routing, and classification supervision.

Q1. Cue Complementarity and Classification Supervision

To distinguish the effects of branch types from differentiated CE/WCE supervision, we compare visual–visual (V/V) and visual–semantic (V/S) configurations under identical or exchanged classification objectives.

Table R1. Ablation of branch types and supervision on VG150 (F@100).

BranchSupervisionPredClsSGClsSGDet
V/VCE/CE37.219.615.8
V/SCE/CE42.826.219.1
V/VCE/WCE39.123.918.0
V/SCE/WCE44.827.119.4
V/SWCE/CE41.824.618.9

Under identical CE supervision, V/S improves F@100 over V/V by 5.6, 6.6, and 3.3 points across PredCls, SGCls, and SGDet, respectively. The same trend holds under CE/WCE supervision, with corresponding gains of 5.7, 3.2, and 1.4 points. Furthermore, assigning CE to the visual branch and WCE to the semantic branch consistently yields the highest mR and F among the tested supervision configurations.

To further examine predicate-level complementarity, we evaluate both branches using the V/S + CE/CE configuration. The visual branch performs better on walking on, laying on, and watching, whereas the semantic branch achieves higher accuracy on carrying, eating, and looking at. These results reveal distinct predicate-level strengths under identical classification supervision.

Complete results are provided in Table 5 and Appendix B.10 (Figure 9) of the revised manuscript.

Q2. Contributions of Class Priors and Instance Reliability

We further compare four routing variants under PredCls and SGDet to examine the contributions of predicate-level class priors and instance-level reliability. All variants retain the default V/S configuration with CE/WCE supervision.

Table R2. Adaptive routing ablation on VG150 (@100).

Fusion variantPredCls R/mR/FSGDet R/mR/F
Learnable per-class67.3 / 32.7 / 44.030.3 / 13.6 / 18.7
Instance-only68.9 / 31.5 / 43.231.0 / 13.2 / 18.5
Class-only68.1 / 32.5 / 44.030.4 / 13.7 / 18.9
Full Router68.5 / 33.3 / 44.830.5 / 14.2 / 19.4

Instance-only routing achieves the highest conventional recall in both tasks, while class-only routing improves class-balanced recall over instance-only routing. Combining both signals further improves mR and F, achieving the highest values under both PredCls and SGDet. Specifically, compared with class-only routing, the full router improves F@100 from 44.0 to 44.8 under PredCls and from 18.9 to 19.4 under SGDet.

These results demonstrate the complementary contributions of instance-level reliability and predicate-level class priors to balanced relation recognition.

Complete results are provided in Appendix B.8 (Table 10).

Q3. Reproducibility and Implementation Details

We have expanded Appendix B.3 with detailed training and SCA adaptation configurations, including WCE weights, temperature parameters, adaptive clustering and donor-mode selection rules, and dynamic synthesis-sampling settings. Representative predicate descriptions are provided in Table 6, and the complete descriptions are included in the supplementary material.

We appreciate the authors' responses and clarifications. The concerns about experimental details have been addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.