- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
GTR-HOI addresses ambiguous instance assignment in HOI detection trained with image-level labels. It first generates spatial pseudo-labels using three MLLM grounding perspectives, transfers uncertain supervision to detector pairs through unbalanced optimal transport, and trains visual and semantic branches with hierarchical targets. Cross-prompt PPO then refines interaction predictions on fixed regions. Pair Expert Routing combines appearance, geometry, relation, and context features. Evaluations cover HICO-DET, V-COCO, two detector architectures, and unseen interaction compositions.
Critical strengths
- The pipeline directly addresses weak-supervision ambiguity and has relevant internal controls. Full mAP progresses from 30.82 for the adapted baseline to 33.31, 35.58, and 36.44 after the successive stages. Comparisons with random prompts, hard IoU matching, balanced OT, and alternative supervision help attribute these gains to the proposed design. External comparisons use different backbones, making these internal controls particularly valuable. Results on both HICO-DET and V-COCO support usefulness beyond one benchmark (Tables 1–3).
- The appendix tests supervision quality rather than relying only on final accuracy. Confidence tiers correlate with grounding precision, and UOT increases positive assignment precision from 72.25% to 80.30% over hard IoU matching while providing broader supervision than balanced OT. Five repeated Stage 3 runs yield 36.34 ± 0.07 Full mAP. Together with threshold sensitivity and failure examples, these analyses provide concrete evidence for the supervision-transfer design and the stability of refinement (Tables 4, 6–10).
Critical weaknesses
- The independent benefit of PPO remains insufficiently established. Feedback comes from the same model under other prompts, the actions are binary, and the candidate regions remain fixed. This can be a valid one-step RL formulation, but the reported random-template and weighting ablations do not distinguish its benefit from ordinary consistency training or soft-label distillation. Since Stage 3 contributes 0.86 Full mAP, an equally budgeted BCE/KL baseline would materially clarify how much improvement requires PPO and how much follows from additional cross-prompt training. The current evidence supports refinement gains more strongly than a distinctive RL contribution (Sec. 3.5; Table 3).
- Instance coverage and semantic correction need more direct diagnosis. Each grounding prompt returns at most one pair, although several true pairs may share an image-level label. Table 4 measures representative precision or coverage of at least one correct hypothesis, rather than recall over all true pairs. The authors acknowledge this limitation, but its consequences for validity targets remain unquantified. Moreover, common errors across prompts can survive refinement: five stable runs demonstrate repeatability, not necessarily correction. These issues matter when uncertain pseudo-labels can influence subsequent positive and negative supervision (Appendices C.2 and E).
Questions for authors
- With identical prompts, samples, weighting, KL regularization, LoRA parameters, and update budgets, how does PPO compare with cross-prompt BCE/KL distillation?
- For image-label groups containing multiple true pairs, what is grounding recall, and how are missed true pairs distributed among positive, negative, and ignored validity targets?
- Between Stages 2 and 3, what fractions of predictions change from incorrect to correct and vice versa? A breakdown for errors shared across prompts would help test the correction interpretation.
4 comments
Q3: Between Stages 2 and 3, what fractions of predictions change from incorrect to correct and vice versa? A breakdown for errors shared across prompts would help test the correction interpretation.
A3: We compare Stage 2 and Stage 3 on five random samples of 150 test images each, using the standard single-prompt inference setting. Detector candidates, candidate interaction classes, and the decision threshold remain fixed. A pair-class entry is positive only when both boxes match the same annotated instance of that HOI class at IoU ≥ 0.5.
For each sample, the correction rate measures the fraction of initially incorrect predictions that become correct, while the degradation rate measures the fraction of initially correct predictions that become incorrect. The table below reports the results, with arrows indicating transitions from Stage 2 to Stage 3.
| Sample | TP | FP | FN | TN | Correction rate | Degradation rate |
|---|---|---|---|---|---|---|
| 1 | 243 → 258 | 3,282 → 3,248 | 136 → 121 | 33,039 → 33,073 | 5.24% | 0.39% |
| 2 | 300 → 303 | 6,189 → 6,091 | 178 → 175 | 57,585 → 57,683 | 4.60% | 0.33% |
| 3 | 250 → 269 | 5,366 → 5,444 | 158 → 139 | 48,597 → 48,519 | 5.47% | 0.74% |
| 4 | 261 → 286 | 6,904 → 6,812 | 180 → 155 | 65,601 → 65,693 | 5.15% | 0.38% |
| 5 | 217 → 226 | 4,369 → 4,456 | 162 → 153 | 48,364 → 48,277 | 6.18% | 0.74% |
Across the five samples, the mean correction rate is 5.33%, with a sample standard deviation of 0.57 percentage points. The mean degradation rate is 0.51%, with a sample standard deviation of 0.20 percentage points. Thus, Stage 3 corrects a measurable fraction of existing errors while preserving approximately 99.49% of initially correct predictions on average.
To examine errors shared across prompts, we conduct a separate analysis on two targeted sets of 100 test images each. The first focuses on pair-class predictions that are incorrect under all four Stage 2 prompt views. The second focuses on incorrect single-prompt predictions for which at least one other view predicts correctly.
Under the same fixed evaluation protocol, Stage 3 corrects 2.18% of initially incorrect predictions in the first group and 3.79% in the second, using standard single-prompt inference. Each rate uses the number of initially incorrect pair-class entries in its respective subset as the denominator. These results show that refinement also corrects some errors shared across all four views, extending its benefits beyond cases involving disagreement among prompts.
We have uploaded a revised paper. The changes are:
-
The PPO versus cross-prompt BCE/KL comparison and experimental settings are provided in Appendix D.4.
-
The dataset-wide instance recall and validity-target distribution for missed instances are reported in Appendix D.1.
-
Correction and degradation statistics across five test samples, together with the analysis of shared prompt errors, are presented in Appendix D.5.
Thank you again for your thoughtful comments. Our updated response includes controlled PPO versus BCE/KL comparisons, grounding recall and supervision analysis for missed instances, and correction/degradation statistics, including errors shared across prompts. We would appreciate your feedback on whether these analyses address your questions about instance coverage and semantic correction.
We appreciate your constructive comments and address your questions below.
Q1: With identical prompts, samples, weighting, KL regularization, LoRA parameters, and update budgets, how does PPO compare with cross-prompt BCE/KL distillation?
A1: We compare PPO with cross-prompt BCE and KL distillation using the same SFT initialization, four prompt views, training samples, spatial and certainty weights, trainable LoRA parameters, and number of optimizer updates. All methods obtain feedback from the other three views and retain KL regularization relative to the reference model with a coefficient of 0.1.
PPO achieves the best performance across all three subsets. Compared with forward KL, the strongest alternative in Full mAP, it improves Full, Rare, and Non-Rare performance by 0.50, 0.57, and 0.48 mAP, respectively. These controlled results support the advantage of the proposed PPO objective over ordinary cross-prompt distillation under the same training conditions.
Q2: For image-label groups containing multiple true pairs, what is grounding recall, and how are missed true pairs distributed among positive, negative, and ignored validity targets?
A2: Grounding recall measures the fraction of GT human-object instances that the retained grounding hypotheses recover. For a group defined by an image and an HOI label, recall is M/N, where N is the number of GT pairs and M is the number of successfully matched pairs. A match requires both human and object IoU ≥0.5 with the same annotated instance of the corresponding HOI label. Each GT instance contributes at most once to the matched count.
To characterize overall coverage, we additionally audit the entire HICO-DET training set. Hypotheses with high, medium, and low confidence recover 35,159, 8,516, and 3,429 GT instances, respectively. Together, they recover 47,104 of 117,871 instances, yielding 39.96% overall instance recall.
Among the remaining 70,767 missed instances, 43.02% match detector pairs with positive validity targets, 9.12% match pairs with negative validity targets, and 47.86% fall into the ignored category. For this audit, the ignored category includes both instances matched to ignored detector pairs and instances without a matching detector candidate. The latter receive no candidate-level validity target. Thus, 90.88% of grounding omissions do not translate into negative validity supervision. These results show that missing a Stage 1 grounding does not automatically cause a corresponding detector pair to receive a negative validity target.