acceptodds
Withdrawn from ICLR 2027

Training-Free Test-Time Retrieval Augmentation for Visual Relationship Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual relationship detectors learn relational knowledge from annotated examples, yet these examples are typically discarded as explicit evidence after training. We revisit their value at inference time and introduce Training-Free Test-Time Retrieval Augmentation, i.e., TF-TRA, a recipe that complements frozen detectors with non-parametric relation memory constructed solely from existing training data. TF-TRA organizes training instances into a relation memory that associates semantic and visual representations with relation annotations while maintaining broad configuration coverage. For each query, it retrieves exemplars through joint semantic and visual matching, with diversity-aware reranking encouraging complementary relation evidence. The retrieved annotations are aggregated into query-specific support and calibrated with the memory prior to refine the frozen detector's predictions. Extensive experiments on three distinct VRD tasks, i.e., human-object interaction (HOI) detection, scene graph generation (SGG), and nonverbal interaction (NVI) detection validate the effectiveness of TF-TRA. It improves overall mAP under both fully supervised and four zero-shot settings on HICO-DET, increases mean Recall and F-score on Visual Genome for SGG, and yields substantial gains in NVI detection. These results show that training examples remain valuable beyond parameter learning, providing explicit relational evidence that strengthens existing detectors at test time. Our soure code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Relationship Understanding, Retrieval Augmented Classification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PJ81OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TF-TRA augments frozen visual relationship detectors with a memory built from their annotated training data. It caps exemplars by entity categories and complete relation label sets, retrieves using detector features and CLIP region features with label diversity reranking, and adds positive, memory prior calibrated log odds residuals to relation logits. Experiments cover supervised and zero shot HOI, scene graph, and nonverbal interaction detection.

Critical strengths
  • The approach improves overall HOI mAP, SGG mean Recall and F-score, and NVI recall across four frozen base detectors without parameter updates. This breadth supports usefulness beyond one model or relation vocabulary. The authors appropriately identify the zero shot gains as mainly improvements on seen interactions.

  • On fully supervised HICO-DET, Default Rare mAP rises from 37.45 to 40.71 while Full mAP rises from 38.56 to 39.75. On Visual Genome, mR@100 improves from 10.1 to 17.3 and F@100 from 15.6 to 22.3, with the acknowledged tradeoff that R@100 falls from 34.3 to 31.4. NVI test AR increases from 75.22 to 79.13.

  • Diagnostics support the internal refinements. Similarity weighting improves average Full mAP from 34.29 to 35.08, and removing prior correction reduces it to 34.00. The appendix also specifies removal of held out HOI annotations before memory construction.

Critical weaknesses
  • The contribution beyond existing cache methods needs sharper evidence. ADA-CM already offers a training free, concept guided memory mode for HOI detection (Lei et al., ICCV 2023), and Tip-Adapter already combines similarity weighted cached labels with base predictions. The paper cites Tip-Adapter and includes ADA-CM in its tables, but does not substantively distinguish ADA-CM's memory mechanism. Table 5 supports the proposed refinements within TF-TRA, yet leaves their advantage over a simple additive cache residual unclear under the same frozen detector, features, and memory budget. For SGG, a prior only correction would also help separate query specific retrieval from global rebalancing.

  • The practical tradeoff remains unmeasured. Each query scores and orders the memory, and visual matching uses additional CLIP ViT-L/14 region encoding. The paper gives per group budgets but no resulting memory cardinality, storage, construction time, or inference latency. Appendix D acknowledges overhead without quantifying it. These measurements are important for assessing the value of an augmentation that adds work at inference.

  • Hyperparameter selection is underspecified. Budgets and neighbor counts vary across tasks, while Table 5 explores budgets on the HICO-DET test set. The manuscript does not state which validation data or objective selected these values and the residual parameters. This is a reproducibility and evaluation protocol uncertainty, not evidence that test labels were used for selection.

Questions for authors
  • With identical frozen detectors, features, and memory budgets, how does TF-TRA compare with a simple similarity weighted additive cache residual and a prior only correction, particularly on SGG?

  • Which held out data and selection objective determined B, K, T, diversity strength, and residual strength? Were the Table 5 test sweeps performed after fixing the final configuration?

  • What are the actual memory sizes, storage requirements, offline construction times, and added inference latency, including CLIP crop encoding, for each task?

5 comments
AuthorsOpenReview ↗

We thank reviewer PJ81 for the valuable time and constructive feedback. We provide point-by-point responses below.

Q1: The contribution beyond existing cache methods needs sharper evidence. ADA-CM already offers a training free, concept guided memory mode for HOI detection (Lei et al., ICCV 2023), and Tip-Adapter already combines similarity weighted cached labels with base predictions. The paper cites Tip-Adapter and includes ADA-CM in its tables, but does not substantively distinguish ADA-CM's memory mechanism.

We thank the reviewer for this point. TF-TRA indeed belongs to the family of training-free cache methods, but it differs from Tip-Adapter and the training-free mode of ADA-CM in three key respects.

(1) Standalone classifier vs. plug-in. Tip-Adapter augments CLIP's zero-shot classifier with a cache built from the same CLIP features, and ADA-CM (TF) inherits this design by using a concept-guided instance memory as a standalone interaction classifier. TF-TRA, in contrast, is a plug-in: it keeps any trained detector frozen and only adds a memory-derived residual to its logits (Eq. 11), which is why it applies directly to EZ-HOI, LAIN, EGTR, and GroupHOI across HOI, SGG, and NVI, as well as to four additional HOI detectors (PViC, ViPLO, HOLa, and CMMP; Appendix C.3, Tables 12–13).

(2) Dense affinity vs. retrieval with calibration. Both Tip-Adapter and ADA-CM score a query by a dense affinity-weighted sum of labels over all cache entries; Tip-Adapter adds this sum to CLIP’s zero-shot logits, while ADA-CM uses it directly as the interaction score. This suffices in Tip-Adapter's few-shot setting, where the cache is balanced by construction (K shots per class), but not for VRD, where the memory is built from full, long-tailed, multi-label annotations. TF-TRA instead retrieves a top-K neighborhood with diversity-aware reranking (Eqs. 5–6), aggregates it with similarity weights (Eqs. 7–8), and calibrates the support against the memory prior as a clipped log-odds residual (Eqs. 9–10). To verify this, we replace TF-TRA's retrieval and calibration with this dense aggregation (affinity over all entries + linear label sum, added to the base logits) on the same frozen LAIN and the same memory, with all else fixed:

HOI (zero-shot)RFNFUOUVAvg.
LAIN34.4133.2334.2733.1233.76
+ Tip-Adapter/ADA-CM-style cache34.5233.3034.3733.2533.86
+ TF-TRA35.5534.4035.3934.9735.08

The Tip-Adapter/ADA-CM-style cache yields only +0.10 mAP on average over the baseline, whereas TF-TRA yields +1.32, confirming that the gain comes from the retrieval and calibration design rather than from caching per se. We attribute this to the dense, uncalibrated label sum over all entries, which largely reproduces the memory's category prior—already captured by the trained detector—rather than instance-level evidence from relevant exemplars.

(3) Memory construction. Tip-Adapter's cache is a balanced few-shot set (K shots per class), and ADA-CM selects memory entries per HOI category. TF-TRA instead groups training instances by entity metadata and their complete multi-label set, caps each group, and selects representatives by herding (Sec. 3.2). This keeps co-occurring relations intact and bounds head configurations, which neither cache design addresses.

As requested, the discussion of ADA-CM has been added to Related Work:

Sec. 2 (L120–124): This principle naturally connects to visual relationship detection (VRD) by reusing relational experiences to alleviate the data scarcity inherent in the combinatorial triplet space (Lu et al., 2016). ADA-CM (Lei et al., 2023) applies this principle to HOI detection, using a concept-guided instance memory as a standalone interaction classifier. In contrast, we propose TF-TRA, a plug-in to frozen detectors that organizes training annotations into a group-capped, multi-label relation memory and retrieves query-specific evidence calibrated against the memory prior to refine predicate prediction.

Q2: Table 5 supports the proposed refinements within TF-TRA, yet leaves their advantage over a simple additive cache residual unclear under the same frozen detector, features, and memory budget.

We thank the reviewer for this suggestion. We implement a simple additive cache residual under exactly the same setting as TF-TRA: the same frozen detector, features, memory bank, and retrieved neighbors. It keeps the diversity-aware neighborhood Ni\mathcal{N}_i (Eq. 6) and the similarity weights wijw_{ij} (Eq. 7) of TF-TRA; the only difference is that the aggregated support is added directly to the base logits, without prior calibration or clipping: zi,c′=zi,cbase+γ pi,cret,pi,cret=∑j∈Niwij yj,c.z'_{i,c} = z^{\mathrm{base}}_{i,c} + \gamma\, p^{\mathrm{ret}}_{i,c}, \qquad p^{\mathrm{ret}}_{i,c} = \sum_{j\in\mathcal{N}_i} w_{ij}\, y_{j,c}.

HOI (default)FullNon-RareRare
EZ-HOI38.5638.8937.45
+ Additive cache residual37.8438.2836.38
+ TF-TRA39.7539.4640.71
HOI (zero-shot)RFNFUOUVAvg
LAIN34.4133.2334.2733.1233.76
+ Additive cache residual34.8533.6534.7533.5534.20
+ TF-TRA35.5534.4035.3934.9735.08
SGGF@20F@50F@100mR@20mR@50mR@100R@20R@50R@100
EGTR8.912.515.65.57.910.123.530.234.3
+ Additive cache residual8.512.215.15.27.79.822.829.233.5
+ TF-TRA12.518.122.38.813.417.321.827.731.4
NVIVal mR@25Val mR@50Val mR@100Val ARTest mR@25Test mR@50Test mR@100Test AR
GroupHOI55.6776.7387.1673.1962.6776.5786.4275.22
+ Additive cache residual56.2471.0386.2771.1860.9176.1388.3275.12
+ TF-TRA60.5476.9587.5675.0266.6180.1990.6079.13

The simple additive residual not only fails to help but often hurts. On fully supervised HOI, it lowers Full mAP by 0.72 (38.56 → 37.84) and Rare by 1.07 (37.45 → 36.38). On SGG, it slightly degrades all metrics (e.g., mR@100 10.1 → 9.8, F@100 15.6 → 15.1). On NVI, it reduces val AR by 2.01 (73.19 → 71.18) and leaves test AR nearly unchanged (75.22 → 75.12). It only helps in zero-shot HOI (+0.44 Avg over LAIN), still well below TF-TRA (+1.32) on every split. These results indicate that simply adding retrieved memory to the base prediction is insufficient, and that the gains of TF-TRA come from the proposed retrieval and calibration designs rather than from the memory itself. These results have been added to Appendix C.4 (Tables 14–16).

Q3: For SGG, a prior only correction would also help separate query specific retrieval from global rebalancing.

We thank the reviewer for this suggestion. We implement a post-hoc logit adjustment baseline. It applies a global class-level rebalancing estimated from the same memory bank, with no query-dependent retrieval, i.e., zi,c′=zi,cbase−γlogit⁡(πcmem)z'_{i,c}=z^{\mathrm{base}}_{i,c}-\gamma\operatorname{logit}(\pi^{\mathrm{mem}}_c). We use γ\gamma = 0.5 on all tasks, the same residual strength as TF-TRA.

HOI (default)FullNon-RareRare
EZ-HOI38.5638.8937.45
+ Post-hoc logit adjustment38.2538.5537.24
+ TF-TRA39.7539.4640.71
HOI (zero-shot)RFNFUOUVAvg
LAIN34.4133.2334.2733.1233.76
+ Post-hoc logit adjustment33.4232.7433.6231.9332.93
+ TF-TRA35.5534.4035.3934.9735.08
SGGF@20F@50F@100mR@20mR@50mR@100R@20R@50R@100
EGTR8.912.515.65.57.910.123.530.234.3
+ Post-hoc logit adjustment11.215.819.47.611.114.221.727.430.8
+ TF-TRA12.518.122.38.813.417.321.827.731.4
NVIVal mR@25Val mR@50Val mR@100Val ARTest mR@25Test mR@50Test mR@100Test AR
GroupHOI55.6776.7387.1673.1962.6776.5786.4275.22
+ Post-hoc logit adjustment56.5277.4287.1173.6960.3779.1589.8476.45
+ TF-TRA60.5476.9587.5675.0266.6180.1990.6079.13

On SGG, post-hoc logit adjustment is effective (mR@100: 10.1 → 14.2; F@100: 15.6 → 19.4), indicating that global rebalancing accounts for part of the gain. However, it is not sufficient on its own: TF-TRA further reaches 17.3 mR@100 and 22.3 F@100. On HOI, post-hoc logit adjustment is harmful. Under the Default setting, it lowers Full and Rare mAP by 0.31 and 0.21. Under the zero-shot settings, it degrades all four splits, by 0.83 average Full mAP. On NVI, post-hoc logit adjustment also improves AR (+0.50 val, +1.23 test) and is slightly better than TF-TRA on val mR@50 (77.42 vs. 76.95), but it lowers test mR@25 below the base detector (62.67 → 60.37). TF-TRA remains ahead on AR (+1.33 val, +2.68 test), with the largest margins on mR@25 (+4.02 / +6.24).

Overall, a consistent gap remains between post-hoc logit adjustment and TF-TRA across all tasks, indicating that the gains come from query-specific retrieved evidence rather than global rebalancing alone. These results have been added to Appendix C.4 (Tables 14–16).

Reviewer_M4ofOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TF-TRA augments frozen visual relationship detectors with a non-parametric memory built from annotated training instances. It combines group-capped, herding-based memory selection, fused semantic and CLIP visual similarity, label-diversity reranking, and positive log-odds residuals relative to the memory prior. Experiments cover fully supervised and zero-shot HOI detection, scene graph generation, and nonverbal interaction detection. The gains appear in overall or category-averaged metrics, with important boundaries: zero-shot Full gains come mainly from seen classes, and SGG mean Recall rises while Recall falls.

Critical strengths
  1. The idea is simple and broadly applicable. A training-free memory plugs into frozen detectors and improves HOI detection, scene graph generation, and nonverbal interaction detection without parameter updates.
  2. The gains are substantial on tail-sensitive metrics. Fully supervised HOI Rare mAP rises by 3.26, SGG mR@100 by 7.2, and NVI test AR by 3.91.
  3. The method is clearly specified and ablated. Each component is formalized and tested, and the appendix makes the supervision boundary explicit, including the removal of held-out annotations in zero-shot settings.
Critical weaknesses
  1. A prior-only baseline is missing. Prior correction contributes 1.08 points, whereas diversity reranking adds 0.14 and herding 0.04, and the largest gains appear on tail-sensitive metrics. Without a post-hoc logit adjustment baseline, query-dependent retrieval cannot be separated from global score rebalancing.
  2. Resource costs are not reported. Group capping does not bound the total memory size, and inference adds CLIP ViT-L/14 encoding of every candidate region plus scoring against the full memory, with 1,024 neighbors per query for NVI. Memory size, construction time, and latency are all missing.
  3. Hyperparameter selection and generality are unclear. Memory budgets and neighbor counts vary widely across tasks, and the HOI values appear to be selected on test results. Each setting also relies on a single base detector, so the plug-and-play claim is tested on one model per setting.
Questions for authors
  1. How does prior-only score calibration, such as post-hoc logit adjustment, perform when its strength is tuned on the same data as TF-TRA's hyperparameters? Can the semantic-plus-visual baseline without diversity also be reported for fully supervised HOI, SGG, and NVI?
  2. What are the memory entry counts, storage, construction time, and base-versus-augmented latency on fixed hardware and candidate counts, including CLIP region encoding?
  3. On which data were B, K, T, γ, and λ_d selected, given that the budget study in Table 5a reports HICO-DET test mAP? Are the small reranking and herding gains stable across repeated memory sampling?
  4. Does TF-TRA also improve a second base detector on at least one task?
4 comments
AuthorsOpenReview ↗

We thank reviewer M4of for the valuable time and constructive feedback. We provide point-by-point responses below.

Q1: A prior-only baseline is missing. Prior correction contributes 1.08 points, whereas diversity reranking adds 0.14 and herding 0.04, and the largest gains appear on tail-sensitive metrics. Without a post-hoc logit adjustment baseline, query-dependent retrieval cannot be separated from global score rebalancing.

(1) Prior-only baseline. We thank the reviewer for the suggestion. We compare against a post-hoc logit adjustment baseline on frozen LAIN (Full mAP, zero-shot HICO-DET). It applies a global class-level rebalancing estimated from the same memory bank, with no query-dependent retrieval:

zi,c′=zi,cbase−γ logit⁡(πcmem)z'_{i,c} = z^{\mathrm{base}}_{i,c} - \gamma\,\operatorname{logit}(\pi^{\mathrm{mem}}_c)

MethodRFNFUOUVAvg.
LAIN34.4133.2334.2733.1233.76
Post-hoc logit adjustment (γ\gamma = 0.05)34.4333.2034.3533.1533.78
Post-hoc logit adjustment (γ\gamma = 0.25)34.4233.4034.3433.0933.81
Post-hoc logit adjustment (γ\gamma = 0.5)33.4232.7433.6231.9332.93
Post-hoc logit adjustment (γ\gamma = 1.0)28.0527.7628.7327.1127.91
TF-TRA35.5534.4035.3934.9735.08

Prior adjustment alone does not recover the improvement. Its best setting (γ\gamma = 0.25) raises the average Full mAP by only 0.05, and larger γ\gamma degrades performance. In contrast, TF-TRA improves all four splits and raises the average by 1.32. This indicates that the gains do not come from global class rebalancing alone, but from the complete TF-TRA pipeline, in which query-specific retrieved evidence is calibrated against the memory prior.

(2) Gains of herding. Herding is deterministic and requires a single run. We therefore repeat random sampling over three seeds and report mean ± std.

HOI. We first compare the two strategies on fully supervised HICO-DET (Default).

Memory selectionFullNon-RareRare
Random sampling39.659 ± 0.00339.407 ± 0.01240.503 ± 0.045
Herding39.74539.45640.712
Δ+0.086+0.049+0.209

We acknowledge that the gains of herding are modest on Non-Rare. On Rare, the gain (+0.209) is 4.6× the standard deviation of random sampling, which suggests the improvement is consistent rather than incidental. We attribute this to the limited memory support for rare categories. Rare queries have few matching entries in memory, so their neighborhoods are easily filled by samples from other categories. Random sampling may retain atypical samples that resemble rare cases, whereas herding retains typical ones that are less likely to be confused with them.

NVI:

Memory selectionVal mR@25Val mR@50Val mR@100Val ARTest mR@25Test mR@50Test mR@100Test AR
Random60.271 ± 0.05076.971 ± 0.07487.221 ± 0.33774.821 ± 0.12366.179 ± 0.22880.188 ± 0.04090.587 ± 0.04978.985 ± 0.091
Herding60.54276.95587.56375.02066.60680.18590.59779.129
Δ+0.271−0.016+0.342+0.199+0.427−0.003+0.010+0.144

On NVI, herding matches or slightly exceeds random sampling: AR improves on both splits (+0.199 on val, +0.144 on test). Random sampling is marginally better only on mR@50, and that difference lies within one standard deviation.

Overall, herding gives a consistent gain on rare HOI categories, where memory support is scarcest, and performs on par with random sampling elsewhere. Since it is deterministic, incurs no inference-time cost, and never underperforms random sampling beyond noise, we retain it as the default. We agree with the reviewer that the gain is modest and have toned down the corresponding claim in the revised manuscript (Appendix C.5).

(3) Gains of diversity reranking. To further verify that it is not specific to zero-shot HOI, we extend the ablation to fully supervised HOI, SGG, and NVI with all other settings unchanged, and report the complete set of metrics below.

HOI (default)FullNon-RareRare
w/o diversity reranking39.5339.4039.96
w/ diversity reranking39.75 (+0.22)39.46 (+0.06)40.71 (+0.75)
SGGF@20F@50F@100mR@20mR@50mR@100R@20R@50R@100
w/o diversity reranking12.1117.4521.728.3712.6916.4821.9127.9231.83
w/ diversity reranking12.55 (+0.44)18.06 (+0.61)22.32 (+0.60)8.80 (+0.43)13.41 (+0.72)17.33 (+0.85)21.83 (−0.08)27.66 (−0.26)31.37 (−0.46)
NVIVal mR@25Val mR@50Val mR@100Val ARTest mR@25Test mR@50Test mR@100Test AR
w/o diversity reranking60.3577.1387.2974.9266.6679.9188.4578.34
w/ diversity reranking60.54 (+0.19)76.95 (−0.18)87.56 (+0.27)75.02 (+0.10)66.61 (−0.05)80.19 (+0.28)90.60 (+2.15)79.13 (+0.79)

As shown above, diversity reranking improves the aggregate metric of every task (HOI Full +0.22, SGG F@K at all cutoffs, NVI AR on both splits), with the largest gains on tail-sensitive metrics: Rare mAP +0.75 on HOI, mR@K at every cutoff on SGG (up to +0.85), and test mR@100 +2.15 on NVI. The slight decrease in SGG Recall is expected, as the penalty mainly affects frequent predicates; the two decreases on NVI (val mR@50 −0.18, test mR@25 −0.05) are marginal. These results have been added to the revised manuscript (Appendix C.6, Tables 19–21).

Q2: Resource costs are not reported. Group capping does not bound the total memory size, and inference adds CLIP ViT-L/14 encoding of every candidate region plus scoring against the full memory, with 1,024 neighbors per query for NVI. Memory size, construction time, and latency are all missing.

We thank the reviewer and report all costs below. Memory bank construction is measured on four NVIDIA GeForce RTX 4090 GPUs, and inference latency on a single RTX 4090; both use batch size 1.

(1) Memory entry counts, storage. We clarify that only the group-capped memory M\mathcal{M} is stored and queried at inference; the source bank Msrc\mathcal{M}_\text{src} is traversed once during construction and then discarded. Although BB caps each group rather than the total, ∣M∣=∑gmin⁡(ng,B)≤B⋅G|\mathcal{M}|=\sum_g \min(n_g,B)\le B\cdot G is bounded by the number GG of observed (entity metadata, label set) groups, which is fixed by the training annotations. As shown below, M\mathcal{M} is 3.3–6.7× smaller than Msrc\mathcal{M}_\text{src} and occupies 27–181 MiB across tasks.

Task / Setting|M_src||M|Storage
HOI / Default (EZ-HOI)83,09515,75149.93 MiB
HOI / NF (LAIN)28,2948,24927.04 MiB
HOI / RF (LAIN)83,26115,48650.76 MiB
HOI / UV (LAIN)70,02312,09839.66 MiB
HOI / UO (LAIN)71,25812,84142.09 MiB
SGG (EGTR)260,78478,800181.11 MiB
NVI (GroupHOI)46,6446,93129.92 MiB

These results have been included in the revised manuscript (Appendix C.1, Tables 6–8).

(2) Construction time. Building the bank needs one forward pass over the training set and no gradient updates. Its total cost is close to that of a single native traversal of the training set by the base model.

Base model / SettingNative traversalFull bank construction
EZ-HOI48.71 min50.10 min
LAIN / NF-UC5.87 min7.71 min
LAIN / RF-UC13.02 min18.26 min
LAIN / UO10.64 min15.67 min
LAIN / UV12.15 min16.63 min
GroupHOI (NVI)6.38 min7.10 min
EGTR (SGG)16.65 min20.23 min

(3) Base-versus-augmented latency. All latency results are measured on a single NVIDIA GeForce RTX 4090 with batch size 1, and include the complete inference pipeline. We report the per-image time averaged over the full test set and over three repeated runs.

Taskbase (ms/img)base (FPS)w/ TF-TRA (ms/img)w/ TF-TRA (FPS)
HOI75.2013.3080.9212.36
SGG104.339.58129.647.71
NVI91.3410.95108.559.21

The overhead is 5.72 ms per image on HOI (+7.6%), 17.21 ms on NVI (+18.8%), and 25.31 ms on SGG (+24.3%), and all settings run at 7.7–12.4 FPS. Given that the method needs no training, we consider this a favorable accuracy–efficiency trade-off. All three measurements have been added to Appendix C.1 (Tables 6–8).

The overhead is bounded for three reasons.

  • First, the only additional network forward pass is the CLIP visual encoder on candidate regions, batched within each image; the relation representation sis_i is taken directly from the frozen detector's own features and incurs no extra computation.
  • Second, refinement is applied only to the top-200 candidate pairs per image (ranked by base score); the rest keep their base logits.
  • Third, CLIP encodes at most 24 regions per image: identical regions among the top-200 candidates are merged, only the 24 unique regions with the highest base scores are encoded, and each remaining candidate reuses the feature of its closest encoded region.

Both settings are used in all experiments of the original submission; we apologize for omitting them from the text, and they are now stated in Appendix A.2.

Regarding the large neighborhood on NVI (K = 1024): scoring and reranking are computed over all memory entries before the top-K selection, so their cost depends on |M| rather than on K. K only affects the final aggregation over the selected entries, which is a small weighted sum, so its effect on latency is negligible.

Q3: Hyperparameter selection and generality are unclear. Memory budgets and neighbor counts vary widely across tasks, and the HOI values appear to be selected on test results. Each setting also relies on a single base detector, so the plug-and-play claim is tested on one model per setting.

(1) Hyperparameter selection. For SGG and NVI, all hyperparameters are selected on the official validation splits, using the average F-score (F@20/50/100) for SGG and AR for NVI as the selection objectives. Since HICO-DET provides no official validation split, we follow common practice in HOI detection, where ablations and hyperparameter studies are reported on the test set [ref1,ref2], and select B, K, and T on the test set of a single configuration, zero-shot HICO-DET with LAIN, using the average Full mAP over the four splits (Table 5a for B; Table 9 for K and T). These values are not tailored to the selection configuration: the same values are applied without modification to the fully supervised setting with EZ-HOI and to the four additional detectors in Appendix C.3 (PViC and ViPLO under Default, Table 12; HOLa and CMMP under all four zero-shot settings, Table 13), none of which was used for selection, and TF-TRA improves all of them. Moreover, γ\gamma and λd\lambda_d are not tuned: both are fixed to a default value of 0.5 and shared across all three tasks and all base detectors. The complete sweeps over B, K, and T on all three tasks are reported in Appendix C.2 (Tables 9–11); performance is stable around the selected values, except for a very small temperature on HOI (T = 0.01). Regarding the different values across tasks: they follow from how the memory is grouped. HOI and SGG yield many small groups keyed by entity categories, whereas NVI involves only persons and yields far fewer but much larger groups, which calls for a larger B and K (Appendix C.2, "Differences across tasks"). This selection protocol is now stated explicitly in Appendix A.2.

[ref1] GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection. CVPR2022.

[ref2] Locality-Aware Zero-Shot Human-Object Interaction Detection. CVPR2025.

(2) More base models. Thank you for the suggestion. We apply TF-TRA to four additional HOI detectors without any parameter updates: PViC and ViPLO in the fully supervised (Default) setting, and HOLa and CMMP in the zero-shot setting. All other hyperparameters are kept the same as in the main paper.

MethodBackboneFullNon-RareRare
PViCR5034.6935.4532.14
+TF-TRAR5036.50 (+1.81)36.62 (+1.17)36.10 (+3.96)
ViPLOViT-B/1637.2237.7535.45
+TF-TRAViT-B/1638.03 (+0.81)37.82 (+0.07)38.75 (+3.30)
MethodBackboneRF-UCNF-UCUOUVAvg.
CMMPR50+ViT-B32.1830.1831.5931.8431.45
+TF-TRAR50+ViT-B33.15 (+0.97)31.21 (+1.03)33.06 (+1.47)32.62 (+0.78)32.51 (+1.06)
HOLaR50+ViT-B34.1932.3633.5934.0933.56
+TF-TRAR50+ViT-B35.27 (+1.08)33.16 (+0.80)34.60 (+1.01)34.65 (+0.56)34.42 (+0.86)

TF-TRA improves Full mAP for all four detectors, which supports its plug-and-play applicability across different base models. These results have been included in the revised manuscript (Appendix C.3, Tables 12–13).

Q4: How does prior-only score calibration, such as post-hoc logit adjustment, perform when its strength is tuned on the same data as TF-TRA's hyperparameters? Can the semantic-plus-visual baseline without diversity also be reported for fully supervised HOI, SGG, and NVI?

(1) Prior-only baseline. Please refer to Q1(1).

(2) Semantic-plus-visual baseline without diversity. Please refer to Q1(3).

Q5: What are the memory entry counts, storage, construction time, and base-versus-augmented latency on fixed hardware and candidate counts, including CLIP region encoding?

Please refer to Q2.

Q6: On which data were B, K, T, γ, and λ_d selected, given that the budget study in Table 5a reports HICO-DET test mAP? Are the small reranking and herding gains stable across repeated memory sampling?

(1) Hyperparameter selection. Please refer to Q3(1).

(2) Gains of herding. Please refer to Q1(2).

(3) Gains of diversity reranking. Please refer to Q1(3).

Q7: Does TF-TRA also improve a second base detector on at least one task?

Please refer to Q3(2).

We hope these responses address your concerns, and we are happy to clarify anything further during the discussion phase.

Reviewer_yvssOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper builds an explicit relation memory from annotated training instances and retrieves examples during inference to refine a frozen detector. Memory construction limits examples per configuration, retrieval combines relation and visual features, and score refinement adjusts retrieved support using memory priors. The evaluation covers human object interaction, scene graph generation, and nonverbal interaction detection.

Critical strengths

s1. The basic idea is practical and easy to understand. Training examples remain available as explicit evidence after model fitting, and the detector does not require further parameter updates. The three stages form a coherent pipeline.

s2. The gains on underrepresented relations are encouraging. Rare interaction precision improves substantially in the fully supervised setting, and scene graph generation gains in category averaged recall. These results suggest that specific stored examples can complement what the detector has learned.

s3. The method does not treat all retrieved annotations as equally trustworthy. Similarity weighting accounts for relevance, and prior calibration addresses label imbalance in the memory. The ablations provide useful evidence that both choices affect performance.

Critical weaknesses

w1. The contribution of retrieval is difficult to separate from that of score calibration. Removing prior correction costs 1.08 percentage points, while diversity reranking adds only 0.14 percentage points. Could a simpler class prior adjustment recover much of the improvement? A direct comparison with that baseline and a basic nearest neighbor fusion method seems essential.

w2. The zero shot results need a narrower interpretation. Most gains come from seen categories. Unseen precision falls in one composition split and stays unchanged for unseen verbs. For a verb with no positive memory examples, both retrieved support and its prior are zero, so the proposed correction is also zero. Preserving its performance is therefore an expected consequence of the formula.

w3. The practical cost is still unclear. Additional region encoding, memory scoring, and reranking all require computation, and one setting retrieves more than a thousand neighbors. How much slower is the complete detector? Actual memory size, storage requirements, and inference latency would make it possible to judge whether the gains are worth the overhead.

w4. Some component gains are very small. Representative selection improves test average recall over random selection by only 0.04 percentage points, and diversity reranking also offers a modest gain. Without repeated measurements, I cannot tell whether these differences are stable. The paper should also explain how memory budgets and other parameters were selected independently of test performance.

Questions for authors

See above comments. I see practical value in the approach, but I am not yet convinced that the proposed combination offers enough beyond simpler retrieval and calibration methods. The missing inference cost analysis is also important for a method whose appeal is practical deployment.

5 comments
AuthorsOpenReview ↗

We thank reviewer yvss for the valuable time and constructive feedback. We provide point-by-point responses below.

Q1: The contribution of retrieval is difficult to separate from that of score calibration. Removing prior correction costs 1.08 percentage points, while diversity reranking adds only 0.14 percentage points. A direct comparison with that baseline and a basic nearest neighbor fusion method seems essential.

(1) Prior-only baseline. We thank the reviewer for the suggestion. We compare against a post-hoc logit adjustment baseline on frozen LAIN (Full mAP, zero-shot HICO-DET). It applies a global class-level rebalancing estimated from the same memory bank, with no query-dependent retrieval:

zi,c′=zi,cbase−γ logit⁡(πcmem)z'_{i,c} = z^{\mathrm{base}}_{i,c} - \gamma\,\operatorname{logit}(\pi^{\mathrm{mem}}_c)

MethodRFNFUOUVAvg.
LAIN34.4133.2334.2733.1233.76
Post-hoc logit adjustment (γ\gamma = 0.05)34.4333.2034.3533.1533.78
Post-hoc logit adjustment (γ\gamma = 0.25)34.4233.4034.3433.0933.81
Post-hoc logit adjustment (γ\gamma = 0.5)33.4232.7433.6231.9332.93
Post-hoc logit adjustment (γ\gamma = 1.0)28.0527.7628.7327.1127.91
TF-TRA35.5534.4035.3934.9735.08

Prior adjustment alone does not recover the improvement. Its best setting (γ\gamma = 0.25) raises the average Full mAP by only 0.05, and larger γ\gamma degrades performance. In contrast, TF-TRA improves all four splits and raises the average by 1.32. This indicates that the gains do not come from global class rebalancing alone, but from the complete TF-TRA pipeline, in which query-specific retrieved evidence is calibrated against the memory prior.

(2) Nearest-neighbor fusion baseline. As requested, we also compare against a basic nearest-neighbor fusion baseline. It uses the same relation memory scoring (Eq. 4) and similarity-weighted aggregation (Eqs. 7–8) as TF-TRA, and removes its three designs: (i) it retrieves from the entire source bank Msrc\mathcal{M}_{\text{src}} without group-capped selection; (ii) it takes the top-K neighbors directly, without diversity-aware reranking; (iii) it adds the aggregated support directly to the base logits, without prior calibration or clipping, i.e.,

zi,c′=zi,cbase+γ pi,cret,pi,cret=∑j∈Niwij yj,c.z'_{i,c} = z^{\mathrm{base}}_{i,c} + \gamma\, p^{\mathrm{ret}}_{i,c}, \qquad p^{\mathrm{ret}}_{i,c} = \sum_{j\in\mathcal{N}_i} w_{ij}\, y_{j,c}.

To separate the effect of memory organization from that of calibration, we also report an intermediate variant, the additive cache residual, which restores (i) and (ii) but still omits (iii) (Appendix C.4).

MethodGroup-capped memoryDiversity rerankingPrior calibrationRFNFUOUVAvg.
LAIN–––34.4133.2334.2733.1233.76
Basic nearest-neighbor fusion✗✗✗34.1433.3134.0832.8433.59
Additive cache residual✓✓✗34.8533.6534.7533.5534.20
TF-TRA✓✓✓35.5534.4035.3934.9735.08

Basic NN fusion falls slightly below LAIN on average (−0.17), so naive retrieval from the raw training bank does not help. Organizing the memory with group-capped selection and diversity-aware reranking turns this into a gain (+0.44 over LAIN), and prior calibration adds a further +0.88, yielding the full +1.32 of TF-TRA. The gains therefore require both components: retrieval alone (−0.17) and prior adjustment alone (+0.05) are each insufficient.

Q2: The zero shot results need a narrower interpretation. Most gains come from seen categories. Unseen precision falls in one composition split and stays unchanged for unseen verbs. For a verb with no positive memory examples, both retrieved support and its prior are zero, so the proposed correction is also zero. Preserving its performance is therefore an expected consequence of the formula.

We thank the reviewer for the careful analysis. We agree that Unseen mAP decreases under NF-UC (−1.18) and that the unchanged UV result follows directly from the formula; the gains are thus mainly on seen categories. In both splits, held-out compositions have no exemplars of their own in memory; their verbs remain only with other objects, so unseen pairs receive only indirect cross-object support. We attribute the opposite trends to this: under RF-UC, the held-out compositions are rare and poorly handled by LAIN (Unseen 31.83), so even indirect evidence helps, mirroring the Rare mAP gains in the fully supervised setting. Under NF-UC, they are frequent and already well handled (Unseen 36.41), so the indirect support is less reliable than the detector's own prediction and hurts the ranking. This differs from the fully supervised setting, where Non-Rare categories have direct exemplars in memory and still improve.

Sec. 4.1 already states that the zero-shot gains are "primarily driven by improvements on seen categories" (L353–354). We have revised the corresponding statements as follows:

Sec. 4.1 (L357–362): Meanwhile, performance on unseen categories is mixed, with changes in Unseen mAP ranging from -1.18% to +0.78%. These results demonstrate that TF-TRA enhances recognition of seen interactions but does not improve unseen categories on average. Held-out compositions have no exemplars of their own in memory and receive only indirect support from exemplars of the same verb with other objects; this benefits rare compositions that the base detector handles poorly (RF-UC, +0.78) but hurts frequent ones it already handles well (NF-UC, −1.18). Under UV, held-out verbs receive no memory support and are thus unaffected by construction.

Introduction (L73–74): ...demonstrating the value of reusing annotated relational examples even under zero-shot evaluation, primarily through improved recognition of seen interactions.

Q3: The practical cost is still unclear. Additional region encoding, memory scoring, and reranking all require computation, and one setting retrieves more than a thousand neighbors. How much slower is the complete detector? Actual memory size, storage requirements, and inference latency would make it possible to judge whether the gains are worth the overhead.

Thank you for the suggestion. We report the inference latency, memory size, and storage cost of TF-TRA below.

(1) Inference latency. All latency results are measured on a single NVIDIA GeForce RTX 4090 with batch size 1, and include the complete inference pipeline. We report the average processing time per image over three runs.

Taskbase (ms/img)base (FPS)w/ TF-TRA (ms/img)w/ TF-TRA (FPS)
HOI75.2013.3080.9212.36
SGG104.339.58129.647.71
NVI91.3410.95108.559.21

The overhead is 5.72 ms per image on HOI (+7.6%), 17.21 ms on NVI (+18.8%), and 25.31 ms on SGG (+24.3%), and all settings run at 7.7–12.4 FPS. Given that the method needs no training, we consider this a favorable accuracy–efficiency trade-off. All three measurements have been added to Appendix C.1 (Tables 6–8).

The overhead is bounded for three reasons.

  • First, the only additional network forward pass is the CLIP visual encoder on candidate regions, batched within each image; the relation representation sis_i is taken directly from the frozen detector's own features and incurs no extra computation.
  • Second, refinement is applied only to the top-200 candidate pairs per image (ranked by base score); the rest keep their base logits.
  • Third, CLIP encodes at most 24 regions per image: identical regions among the top-200 candidates are merged, only the 24 unique regions with the highest base scores are encoded, and each remaining candidate reuses the feature of its closest encoded region.

Both settings are used in all experiments of the original submission; we apologize for omitting them from the text, and they are now stated in Appendix A.2.

Regarding the large neighborhood on NVI (K = 1024): scoring and reranking are computed over all memory entries before the top-K selection, so their cost depends on |M| rather than on K. K only affects the final aggregation over the selected entries, which is a small weighted sum, so its effect on latency is negligible.

(2) Memory size and storage. As shown in the table below, the retrieval memory banks require only 27.04–50.76 MiB for HOI across settings, 181.11 MiB for SGG, and 29.92 MiB for NVI.

Task / Setting|M_src||M|Storage
HOI / Default (EZ-HOI)83,09515,75149.93 MiB
HOI / NF (LAIN)28,2948,24927.04 MiB
HOI / RF (LAIN)83,26115,48650.76 MiB
HOI / UV (LAIN)70,02312,09839.66 MiB
HOI / UO (LAIN)71,25812,84142.09 MiB
SGG (EGTR)260,78478,800181.11 MiB
NVI (GroupHOI)46,6446,93129.92 MiB

These results have been included in the revised manuscript (Appendix C.1, Tables 6–8).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.