- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TF-TRA augments frozen visual relationship detectors with a memory built from their annotated training data. It caps exemplars by entity categories and complete relation label sets, retrieves using detector features and CLIP region features with label diversity reranking, and adds positive, memory prior calibrated log odds residuals to relation logits. Experiments cover supervised and zero shot HOI, scene graph, and nonverbal interaction detection.
Critical strengths
-
The approach improves overall HOI mAP, SGG mean Recall and F-score, and NVI recall across four frozen base detectors without parameter updates. This breadth supports usefulness beyond one model or relation vocabulary. The authors appropriately identify the zero shot gains as mainly improvements on seen interactions.
-
On fully supervised HICO-DET, Default Rare mAP rises from 37.45 to 40.71 while Full mAP rises from 38.56 to 39.75. On Visual Genome, mR@100 improves from 10.1 to 17.3 and F@100 from 15.6 to 22.3, with the acknowledged tradeoff that R@100 falls from 34.3 to 31.4. NVI test AR increases from 75.22 to 79.13.
-
Diagnostics support the internal refinements. Similarity weighting improves average Full mAP from 34.29 to 35.08, and removing prior correction reduces it to 34.00. The appendix also specifies removal of held out HOI annotations before memory construction.
Critical weaknesses
-
The contribution beyond existing cache methods needs sharper evidence. ADA-CM already offers a training free, concept guided memory mode for HOI detection (Lei et al., ICCV 2023), and Tip-Adapter already combines similarity weighted cached labels with base predictions. The paper cites Tip-Adapter and includes ADA-CM in its tables, but does not substantively distinguish ADA-CM's memory mechanism. Table 5 supports the proposed refinements within TF-TRA, yet leaves their advantage over a simple additive cache residual unclear under the same frozen detector, features, and memory budget. For SGG, a prior only correction would also help separate query specific retrieval from global rebalancing.
-
The practical tradeoff remains unmeasured. Each query scores and orders the memory, and visual matching uses additional CLIP ViT-L/14 region encoding. The paper gives per group budgets but no resulting memory cardinality, storage, construction time, or inference latency. Appendix D acknowledges overhead without quantifying it. These measurements are important for assessing the value of an augmentation that adds work at inference.
-
Hyperparameter selection is underspecified. Budgets and neighbor counts vary across tasks, while Table 5 explores budgets on the HICO-DET test set. The manuscript does not state which validation data or objective selected these values and the residual parameters. This is a reproducibility and evaluation protocol uncertainty, not evidence that test labels were used for selection.
Questions for authors
-
With identical frozen detectors, features, and memory budgets, how does TF-TRA compare with a simple similarity weighted additive cache residual and a prior only correction, particularly on SGG?
-
Which held out data and selection objective determined B, K, T, diversity strength, and residual strength? Were the Table 5 test sweeps performed after fixing the final configuration?
-
What are the actual memory sizes, storage requirements, offline construction times, and added inference latency, including CLIP crop encoding, for each task?
5 comments
Q2: Table 5 supports the proposed refinements within TF-TRA, yet leaves their advantage over a simple additive cache residual unclear under the same frozen detector, features, and memory budget.
We thank the reviewer for this suggestion. We implement a simple additive cache residual under exactly the same setting as TF-TRA: the same frozen detector, features, memory bank, and retrieved neighbors. It keeps the diversity-aware neighborhood (Eq. 6) and the similarity weights (Eq. 7) of TF-TRA; the only difference is that the aggregated support is added directly to the base logits, without prior calibration or clipping:
| HOI (default) | Full | Non-Rare | Rare |
|---|---|---|---|
| EZ-HOI | 38.56 | 38.89 | 37.45 |
| + Additive cache residual | 37.84 | 38.28 | 36.38 |
| + TF-TRA | 39.75 | 39.46 | 40.71 |
| HOI (zero-shot) | RF | NF | UO | UV | Avg |
|---|---|---|---|---|---|
| LAIN | 34.41 | 33.23 | 34.27 | 33.12 | 33.76 |
| + Additive cache residual | 34.85 | 33.65 | 34.75 | 33.55 | 34.20 |
| + TF-TRA | 35.55 | 34.40 | 35.39 | 34.97 | 35.08 |
| SGG | F@20 | F@50 | F@100 | mR@20 | mR@50 | mR@100 | R@20 | R@50 | R@100 |
|---|---|---|---|---|---|---|---|---|---|
| EGTR | 8.9 | 12.5 | 15.6 | 5.5 | 7.9 | 10.1 | 23.5 | 30.2 | 34.3 |
| + Additive cache residual | 8.5 | 12.2 | 15.1 | 5.2 | 7.7 | 9.8 | 22.8 | 29.2 | 33.5 |
| + TF-TRA | 12.5 | 18.1 | 22.3 | 8.8 | 13.4 | 17.3 | 21.8 | 27.7 | 31.4 |
| NVI | Val mR@25 | Val mR@50 | Val mR@100 | Val AR | Test mR@25 | Test mR@50 | Test mR@100 | Test AR |
|---|---|---|---|---|---|---|---|---|
| GroupHOI | 55.67 | 76.73 | 87.16 | 73.19 | 62.67 | 76.57 | 86.42 | 75.22 |
| + Additive cache residual | 56.24 | 71.03 | 86.27 | 71.18 | 60.91 | 76.13 | 88.32 | 75.12 |
| + TF-TRA | 60.54 | 76.95 | 87.56 | 75.02 | 66.61 | 80.19 | 90.60 | 79.13 |
The simple additive residual not only fails to help but often hurts. On fully supervised HOI, it lowers Full mAP by 0.72 (38.56 → 37.84) and Rare by 1.07 (37.45 → 36.38). On SGG, it slightly degrades all metrics (e.g., mR@100 10.1 → 9.8, F@100 15.6 → 15.1). On NVI, it reduces val AR by 2.01 (73.19 → 71.18) and leaves test AR nearly unchanged (75.22 → 75.12). It only helps in zero-shot HOI (+0.44 Avg over LAIN), still well below TF-TRA (+1.32) on every split. These results indicate that simply adding retrieved memory to the base prediction is insufficient, and that the gains of TF-TRA come from the proposed retrieval and calibration designs rather than from the memory itself. These results have been added to Appendix C.4 (Tables 14–16).
Q3: For SGG, a prior only correction would also help separate query specific retrieval from global rebalancing.
We thank the reviewer for this suggestion. We implement a post-hoc logit adjustment baseline. It applies a global class-level rebalancing estimated from the same memory bank, with no query-dependent retrieval, i.e., . We use = 0.5 on all tasks, the same residual strength as TF-TRA.
| HOI (default) | Full | Non-Rare | Rare |
|---|---|---|---|
| EZ-HOI | 38.56 | 38.89 | 37.45 |
| + Post-hoc logit adjustment | 38.25 | 38.55 | 37.24 |
| + TF-TRA | 39.75 | 39.46 | 40.71 |
| HOI (zero-shot) | RF | NF | UO | UV | Avg |
|---|---|---|---|---|---|
| LAIN | 34.41 | 33.23 | 34.27 | 33.12 | 33.76 |
| + Post-hoc logit adjustment | 33.42 | 32.74 | 33.62 | 31.93 | 32.93 |
| + TF-TRA | 35.55 | 34.40 | 35.39 | 34.97 | 35.08 |
| SGG | F@20 | F@50 | F@100 | mR@20 | mR@50 | mR@100 | R@20 | R@50 | R@100 |
|---|---|---|---|---|---|---|---|---|---|
| EGTR | 8.9 | 12.5 | 15.6 | 5.5 | 7.9 | 10.1 | 23.5 | 30.2 | 34.3 |
| + Post-hoc logit adjustment | 11.2 | 15.8 | 19.4 | 7.6 | 11.1 | 14.2 | 21.7 | 27.4 | 30.8 |
| + TF-TRA | 12.5 | 18.1 | 22.3 | 8.8 | 13.4 | 17.3 | 21.8 | 27.7 | 31.4 |
| NVI | Val mR@25 | Val mR@50 | Val mR@100 | Val AR | Test mR@25 | Test mR@50 | Test mR@100 | Test AR |
|---|---|---|---|---|---|---|---|---|
| GroupHOI | 55.67 | 76.73 | 87.16 | 73.19 | 62.67 | 76.57 | 86.42 | 75.22 |
| + Post-hoc logit adjustment | 56.52 | 77.42 | 87.11 | 73.69 | 60.37 | 79.15 | 89.84 | 76.45 |
| + TF-TRA | 60.54 | 76.95 | 87.56 | 75.02 | 66.61 | 80.19 | 90.60 | 79.13 |
On SGG, post-hoc logit adjustment is effective (mR@100: 10.1 → 14.2; F@100: 15.6 → 19.4), indicating that global rebalancing accounts for part of the gain. However, it is not sufficient on its own: TF-TRA further reaches 17.3 mR@100 and 22.3 F@100. On HOI, post-hoc logit adjustment is harmful. Under the Default setting, it lowers Full and Rare mAP by 0.31 and 0.21. Under the zero-shot settings, it degrades all four splits, by 0.83 average Full mAP. On NVI, post-hoc logit adjustment also improves AR (+0.50 val, +1.23 test) and is slightly better than TF-TRA on val mR@50 (77.42 vs. 76.95), but it lowers test mR@25 below the base detector (62.67 → 60.37). TF-TRA remains ahead on AR (+1.33 val, +2.68 test), with the largest margins on mR@25 (+4.02 / +6.24).
Overall, a consistent gap remains between post-hoc logit adjustment and TF-TRA across all tasks, indicating that the gains come from query-specific retrieved evidence rather than global rebalancing alone. These results have been added to Appendix C.4 (Tables 14–16).
We thank reviewer PJ81 for the valuable time and constructive feedback. We provide point-by-point responses below.
Q1: The contribution beyond existing cache methods needs sharper evidence. ADA-CM already offers a training free, concept guided memory mode for HOI detection (Lei et al., ICCV 2023), and Tip-Adapter already combines similarity weighted cached labels with base predictions. The paper cites Tip-Adapter and includes ADA-CM in its tables, but does not substantively distinguish ADA-CM's memory mechanism.
We thank the reviewer for this point. TF-TRA indeed belongs to the family of training-free cache methods, but it differs from Tip-Adapter and the training-free mode of ADA-CM in three key respects.
(1) Standalone classifier vs. plug-in. Tip-Adapter augments CLIP's zero-shot classifier with a cache built from the same CLIP features, and ADA-CM (TF) inherits this design by using a concept-guided instance memory as a standalone interaction classifier. TF-TRA, in contrast, is a plug-in: it keeps any trained detector frozen and only adds a memory-derived residual to its logits (Eq. 11), which is why it applies directly to EZ-HOI, LAIN, EGTR, and GroupHOI across HOI, SGG, and NVI, as well as to four additional HOI detectors (PViC, ViPLO, HOLa, and CMMP; Appendix C.3, Tables 12–13).
(2) Dense affinity vs. retrieval with calibration. Both Tip-Adapter and ADA-CM score a query by a dense affinity-weighted sum of labels over all cache entries; Tip-Adapter adds this sum to CLIP’s zero-shot logits, while ADA-CM uses it directly as the interaction score. This suffices in Tip-Adapter's few-shot setting, where the cache is balanced by construction (K shots per class), but not for VRD, where the memory is built from full, long-tailed, multi-label annotations. TF-TRA instead retrieves a top-K neighborhood with diversity-aware reranking (Eqs. 5–6), aggregates it with similarity weights (Eqs. 7–8), and calibrates the support against the memory prior as a clipped log-odds residual (Eqs. 9–10). To verify this, we replace TF-TRA's retrieval and calibration with this dense aggregation (affinity over all entries + linear label sum, added to the base logits) on the same frozen LAIN and the same memory, with all else fixed:
The Tip-Adapter/ADA-CM-style cache yields only +0.10 mAP on average over the baseline, whereas TF-TRA yields +1.32, confirming that the gain comes from the retrieval and calibration design rather than from caching per se. We attribute this to the dense, uncalibrated label sum over all entries, which largely reproduces the memory's category prior—already captured by the trained detector—rather than instance-level evidence from relevant exemplars.
(3) Memory construction. Tip-Adapter's cache is a balanced few-shot set (K shots per class), and ADA-CM selects memory entries per HOI category. TF-TRA instead groups training instances by entity metadata and their complete multi-label set, caps each group, and selects representatives by herding (Sec. 3.2). This keeps co-occurring relations intact and bounds head configurations, which neither cache design addresses.
As requested, the discussion of ADA-CM has been added to Related Work: