acceptodds
Under review as a conference paper at ICLR 2027

GTR-HOI: Ground, Transport, and Reinforce for Weakly Supervised HOI Detection

Abstract

Weakly supervised human–object interaction (HOI) detection learns to localize and recognize interactions from image-level labels, without explicit correspondences between labels and human–object pairs. This ambiguity can reinforce incorrect interaction assignments during training. We propose GTR-HOI, a three-stage framework that grounds image-level labels, transports localization supervision, and reinforces interaction predictions. First, a pretrained multimodal large language model (MLLM) generates pseudo-ground truths from human, object, and relation perspectives, with spatial agreement determining their confidence tiers. To account for grounding uncertainty, unbalanced optimal transport transfers localization supervision to detector pairs through soft correspondences. Pair expert routing integrates appearance, spatial, and contextual cues for pair validity prediction and visual HOI scoring. Hierarchical supervision uses consensus assignments with high or medium confidence to train the visual scoring branch and fine-tune the MLLM for semantic HOI scoring. Finally, cross-prompt reinforcement learning uses semantic feedback across four prompt views to refine decisions to accept or reject candidate interactions through weighted proximal policy optimization. Spatial and fixed reference certainty weights regulate these updates to limit the influence of unreliable supervision. Experiments on HICO-DET and V-COCO demonstrate state-of-the-art performance among the compared weakly supervised methods, achieving 36.44 Full mAP and 60.6 Scenario 2 role AP, respectively. The code is provided in the supplementary materials.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Weakly Supervised HOI Detection, Multimodal Large Language Model, Unbalanced Optimal Transport, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7ZqeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GTR-HOI addresses ambiguous instance assignment in HOI detection trained with image-level labels. It first generates spatial pseudo-labels using three MLLM grounding perspectives, transfers uncertain supervision to detector pairs through unbalanced optimal transport, and trains visual and semantic branches with hierarchical targets. Cross-prompt PPO then refines interaction predictions on fixed regions. Pair Expert Routing combines appearance, geometry, relation, and context features. Evaluations cover HICO-DET, V-COCO, two detector architectures, and unseen interaction compositions.

Critical strengths
  1. The pipeline directly addresses weak-supervision ambiguity and has relevant internal controls. Full mAP progresses from 30.82 for the adapted baseline to 33.31, 35.58, and 36.44 after the successive stages. Comparisons with random prompts, hard IoU matching, balanced OT, and alternative supervision help attribute these gains to the proposed design. External comparisons use different backbones, making these internal controls particularly valuable. Results on both HICO-DET and V-COCO support usefulness beyond one benchmark (Tables 1–3).
  2. The appendix tests supervision quality rather than relying only on final accuracy. Confidence tiers correlate with grounding precision, and UOT increases positive assignment precision from 72.25% to 80.30% over hard IoU matching while providing broader supervision than balanced OT. Five repeated Stage 3 runs yield 36.34 ± 0.07 Full mAP. Together with threshold sensitivity and failure examples, these analyses provide concrete evidence for the supervision-transfer design and the stability of refinement (Tables 4, 6–10).
Critical weaknesses
  1. The independent benefit of PPO remains insufficiently established. Feedback comes from the same model under other prompts, the actions are binary, and the candidate regions remain fixed. This can be a valid one-step RL formulation, but the reported random-template and weighting ablations do not distinguish its benefit from ordinary consistency training or soft-label distillation. Since Stage 3 contributes 0.86 Full mAP, an equally budgeted BCE/KL baseline would materially clarify how much improvement requires PPO and how much follows from additional cross-prompt training. The current evidence supports refinement gains more strongly than a distinctive RL contribution (Sec. 3.5; Table 3).
  2. Instance coverage and semantic correction need more direct diagnosis. Each grounding prompt returns at most one pair, although several true pairs may share an image-level label. Table 4 measures representative precision or coverage of at least one correct hypothesis, rather than recall over all true pairs. The authors acknowledge this limitation, but its consequences for validity targets remain unquantified. Moreover, common errors across prompts can survive refinement: five stable runs demonstrate repeatability, not necessarily correction. These issues matter when uncertain pseudo-labels can influence subsequent positive and negative supervision (Appendices C.2 and E).
Questions for authors
  1. With identical prompts, samples, weighting, KL regularization, LoRA parameters, and update budgets, how does PPO compare with cross-prompt BCE/KL distillation?
  2. For image-label groups containing multiple true pairs, what is grounding recall, and how are missed true pairs distributed among positive, negative, and ignored validity targets?
  3. Between Stages 2 and 3, what fractions of predictions change from incorrect to correct and vice versa? A breakdown for errors shared across prompts would help test the correction interpretation.
4 comments
AuthorsOpenReview ↗

We appreciate your constructive comments and address your questions below.

Q1: With identical prompts, samples, weighting, KL regularization, LoRA parameters, and update budgets, how does PPO compare with cross-prompt BCE/KL distillation?

A1: We compare PPO with cross-prompt BCE and KL distillation using the same SFT initialization, four prompt views, training samples, spatial and certainty weights, trainable LoRA parameters, and number of optimizer updates. All methods obtain feedback from the other three views and retain KL regularization relative to the reference model with a coefficient of 0.1.

Optimization objectiveFullRareNon-Rare
Cross-prompt BCE35.9241.7934.17
Cross-prompt forward KL35.9441.9734.14
Cross-prompt symmetric KL35.9042.0734.06
GTR-HOI (PPO)36.4442.5434.62

PPO achieves the best performance across all three subsets. Compared with forward KL, the strongest alternative in Full mAP, it improves Full, Rare, and Non-Rare performance by 0.50, 0.57, and 0.48 mAP, respectively. These controlled results support the advantage of the proposed PPO objective over ordinary cross-prompt distillation under the same training conditions.

Q2: For image-label groups containing multiple true pairs, what is grounding recall, and how are missed true pairs distributed among positive, negative, and ignored validity targets?

A2: Grounding recall measures the fraction of GT human-object instances that the retained grounding hypotheses recover. For a group defined by an image and an HOI label, recall is M/NM/N, where NN is the number of GT pairs and MM is the number of successfully matched pairs. A match requires both human and object IoU ≥0.5\geq 0.5 with the same annotated instance of the corresponding HOI label. Each GT instance contributes at most once to the matched count.

To characterize overall coverage, we additionally audit the entire HICO-DET training set. Hypotheses with high, medium, and low confidence recover 35,159, 8,516, and 3,429 GT instances, respectively. Together, they recover 47,104 of 117,871 instances, yielding 39.96% overall instance recall.

Among the remaining 70,767 missed instances, 43.02% match detector pairs with positive validity targets, 9.12% match pairs with negative validity targets, and 47.86% fall into the ignored category. For this audit, the ignored category includes both instances matched to ignored detector pairs and instances without a matching detector candidate. The latter receive no candidate-level validity target. Thus, 90.88% of grounding omissions do not translate into negative validity supervision. These results show that missing a Stage 1 grounding does not automatically cause a corresponding detector pair to receive a negative validity target.

Q3: Between Stages 2 and 3, what fractions of predictions change from incorrect to correct and vice versa? A breakdown for errors shared across prompts would help test the correction interpretation.

A3: We compare Stage 2 and Stage 3 on five random samples of 150 test images each, using the standard single-prompt inference setting. Detector candidates, candidate interaction classes, and the decision threshold remain fixed. A pair-class entry is positive only when both boxes match the same annotated instance of that HOI class at IoU ≥ 0.5.

For each sample, the correction rate measures the fraction of initially incorrect predictions that become correct, while the degradation rate measures the fraction of initially correct predictions that become incorrect. The table below reports the results, with arrows indicating transitions from Stage 2 to Stage 3.

SampleTPFPFNTNCorrection rateDegradation rate
1243 → 2583,282 → 3,248136 → 12133,039 → 33,0735.24%0.39%
2300 → 3036,189 → 6,091178 → 17557,585 → 57,6834.60%0.33%
3250 → 2695,366 → 5,444158 → 13948,597 → 48,5195.47%0.74%
4261 → 2866,904 → 6,812180 → 15565,601 → 65,6935.15%0.38%
5217 → 2264,369 → 4,456162 → 15348,364 → 48,2776.18%0.74%

Across the five samples, the mean correction rate is 5.33%, with a sample standard deviation of 0.57 percentage points. The mean degradation rate is 0.51%, with a sample standard deviation of 0.20 percentage points. Thus, Stage 3 corrects a measurable fraction of existing errors while preserving approximately 99.49% of initially correct predictions on average.

To examine errors shared across prompts, we conduct a separate analysis on two targeted sets of 100 test images each. The first focuses on pair-class predictions that are incorrect under all four Stage 2 prompt views. The second focuses on incorrect single-prompt predictions for which at least one other view predicts correctly.

Under the same fixed evaluation protocol, Stage 3 corrects 2.18% of initially incorrect predictions in the first group and 3.79% in the second, using standard single-prompt inference. Each rate uses the number of initially incorrect pair-class entries in its respective subset as the denominator. These results show that refinement also corrects some errors shared across all four views, extending its benefits beyond cases involving disagreement among prompts.

We have uploaded a revised paper. The changes are:

  1. The PPO versus cross-prompt BCE/KL comparison and experimental settings are provided in Appendix D.4.

  2. The dataset-wide instance recall and validity-target distribution for missed instances are reported in Appendix D.1.

  3. Correction and degradation statistics across five test samples, together with the analysis of shared prompt errors, are presented in Appendix D.5.

Thank you again for your thoughtful comments. Our updated response includes controlled PPO versus BCE/KL comparisons, grounding recall and supervision analysis for missed instances, and correction/degradation statistics, including errors shared across prompts. We would appreciate your feedback on whether these analyses address your questions about instance coverage and semantic correction.

Reviewer_5xQsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies weakly supervised human–object interaction detection, where training data provide image-level HOI labels but do not specify which human–object pairs instantiate those interactions. The proposed GTR-HOI framework addresses this ambiguity through three stages. First, a pretrained Qwen2.5-VL-3B model generates label-conditioned pseudo-groundings from human-, object-, and relation-centric prompts, and spatial agreement across these views determines confidence tiers. Second, unbalanced optimal transport transfers supervision from these uncertain pseudo-groundings to detector-generated human–object pairs. Pair Expert Routing combines appearance, geometric, relational, and contextual features, while hierarchical supervision trains visual and semantic HOI scoring branches. Third, cross-prompt reinforcement learning uses agreement across four prompt views to construct accept/reject feedback and optimize the MLLM’s LoRA parameters using a weighted PPO objective.

The method is evaluated on HICO-DET and V-COCO with Faster R-CNN and DETR detectors, as well as under rare-first and non-rare-first unseen-composition settings. The reported best results are 36.44 Full mAP on HICO-DET and 60.6 Scenario-2 role AP on V-COCO, outperforming the weakly supervised methods included in the comparison. The paper also provides component ablations, pseudo-grounding and matching analyses, sensitivity studies, repeated Stage-3 runs, qualitative examples, and computational-cost measurements.

Critical strengths
  1. The method directly addresses the central instance-assignment ambiguity in weakly supervised HOI detection. Rather than only aggregating predictions over candidate pairs, GTR-HOI produces explicit label-conditioned human–object pseudo-groundings and models their uncertainty. The use of spatial agreement to form confidence tiers, followed by UOT to transfer supervision softly to detector proposals, is well motivated by the weak-supervision setting. This is consequential because the Stage-1 and Stage-2 ablations account for most of the improvement over the adapted baseline, suggesting that the central Ground-and-Transport design is empirically useful rather than merely conceptual.

  2. The empirical evaluation is broad and produces strong results. The paper evaluates two benchmarks, two detector architectures, standard and unseen-composition settings, and multiple aspects of the proposed pipeline. GTR-HOI achieves 36.44 Full mAP on HICO-DET with DETR, compared with 32.90 for the reported RegFormer DINO-B configuration, and obtains substantial improvements over RegFormer in both RF-UC and NF-UC unseen-composition evaluations. These results indicate that the overall system is competitive and that its benefits are not confined to one detector or only to seen HOI compositions.

Critical weaknesses
  1. The experiments do not establish that PPO is necessary for Stage 3 or that its gain should be attributed to reinforcement learning. Stage 3 obtains pseudo-target feedback from the predictions of other prompt views produced by the same underlying model. There is no external environment, independently grounded reward, or human feedback; functionally, this stage is closely related to cross-view self-training or consistency regularization. Although the complete Stage 3 improves Full mAP from 35.58 to 36.44, the paper does not compare its PPO objective against simpler, compute-matched alternatives such as stop-gradient soft-label binary cross-entropy, KL consistency, hard majority-vote pseudo-labeling, or an EMA teacher. This omission is consequential because the paper presents cross-prompt RL as a principal methodological contribution. The existing ablations show that an additional multi-prompt refinement stage is useful, but they do not isolate whether PPO, clipping, or exact action enumeration contributes beyond ordinary self-distillation. They also do not rule out confirmation bias when correlated prompt views agree on an incorrect prediction.
  2. The state-of-the-art comparison does not cleanly separate algorithmic improvements from foundation-model capacity and pretraining. GTR-HOI uses Qwen2.5-VL-3B for both visual and semantic representations, whereas major weakly supervised baselines use combinations such as DINO-B and CLIP-B. The same-backbone internal baseline and component ablations support the claim that the complete pipeline improves upon the authors’ adapted DA-HOI formulation. However, the comparison against RegFormer does not determine how much of the reported 2.74–3.54 Full-mAP advantage comes from GTR-HOI’s algorithms and how much comes from the stronger MLLM backbone and its pretraining. This issue is particularly relevant to the unseen-composition results. The appendix correctly explains that “unseen” means excluded from task-specific supervision rather than absent from foundation-model pretraining, and it acknowledges that overlap with the MLLM’s pretraining corpus cannot be audited. A capacity-matched baseline is therefore important for supporting the method-level SOTA claim.
Questions for authors

Clarification 1 — PPO attribution: Can the authors compare Stage 3 against compute- and data-matched alternatives using the same four prompt views, LoRA parameters, training examples, and number of updates—for example, soft-target binary cross-entropy, symmetric or stop-gradient KL consistency, hard majority-vote pseudo-labeling, and EMA-teacher self-training? This comparison would materially affect my assessment of cross-prompt PPO as a core contribution rather than a more complicated implementation of consistency-based self-training. Clarification 2 — Role of PPO components: What happens when PPO clipping is removed while retaining the same cross-prompt targets and certainty weights? Can the authors separately ablate exact action enumeration, the old-policy baseline, and KL-to-reference regularization? The current experiments demonstrate the benefit of the complete refinement stage but do not isolate the mechanisms specific to PPO. Clarification 3 — Capacity-matched comparison: Can the authors provide either (a) a strong weakly supervised baseline using the same Qwen2.5-VL backbone, or (b) a GTR-HOI configuration using a backbone closer in capacity and pretraining to RegFormer? Without such a comparison, it is difficult to attribute the reported SOTA gap to the proposed Ground–Transport–Reinforce design.

4 comments
AuthorsOpenReview ↗

We provide point-by-point answers to your constructive questions below.

Q1: PPO attribution: Can the authors compare Stage 3 against compute- and data-matched alternatives using the same four prompt views, LoRA parameters, training examples, and number of updates, for example, soft-target binary cross-entropy, symmetric or stop-gradient KL consistency, hard majority-vote pseudo-labeling, and EMA-teacher self-training? This comparison would materially affect my assessment of cross-prompt PPO as a core contribution rather than a more complicated implementation of consistency-based self-training.

A1: We initialize all refinement methods from the same Stage 2 checkpoint and use identical training examples, four prompt views, trainable LoRA parameters, numbers of optimizer updates, spatial and certainty weights, and KL regularization relative to the reference model with a coefficient of 0.1.

MethodFullRareNon-Rare
SFT initialization35.5841.7333.74
Soft-target BCE35.9241.7934.17
Forward KL, detached targets35.9441.9734.14
Symmetric KL, detached targets35.9042.0734.06
Hard majority vote27.2127.6827.06
EMA teacher35.8841.8634.09
GTR-HOI (PPO)36.4442.5434.62

Under matched training conditions, our PPO formulation achieves the best performance across all three subsets. It improves Full mAP from 35.58 to 36.44, outperforming forward KL, soft-target BCE, and self-training with an EMA teacher by 0.50, 0.52, and 0.56 mAP, respectively. It also achieves the highest Rare and Non-Rare mAP of 42.54 and 34.62, demonstrating benefits across both infrequent and frequent interaction categories. These consistent gains show that the proposed PPO objective uses the shared cross-prompt feedback more effectively than the evaluated methods for self-distillation and consistency training, providing stronger refinement with the same number of optimizer updates.

Q2: Role of PPO components: What happens when PPO clipping is removed while retaining the same cross-prompt targets and certainty weights? Can the authors separately ablate exact action enumeration, the old-policy baseline, and KL-to-reference regularization? The current experiments demonstrate the benefit of the complete refinement stage but do not isolate the mechanisms specific to PPO.

A2: We separately ablate each PPO component while keeping the remaining training configuration unchanged.

SettingFullRareNon-Rare
Without clipping35.6040.7834.05
Sampled actions35.8541.7534.09
Without reward baseline36.1941.9934.46
Without reference KL35.7344.6133.08
Complete PPO36.4442.5434.62

Removing clipping reduces Full mAP by 0.84, supporting the importance of controlling the magnitude of policy updates. Replacing exact action enumeration with sampled actions decreases Full mAP by 0.59. Since the action space contains only acceptance and rejection, exact enumeration evaluates the complete action expectation without sampling noise. Removing the reward baseline computed from the old policy further reduces Full mAP by 0.25, showing that advantage centering contributes to the effectiveness of the clipped objective.

KL regularization relative to the reference model also benefits overall detection performance. Including it improves Full and Non-Rare mAP by 0.71 and 1.54, respectively. Although removing this regularizer yields higher Rare mAP, its substantial reduction on Non-Rare classes lowers overall performance. The complete configuration achieves the highest Full and Non-Rare mAP. Together, these controlled ablations support the individual contributions of clipping, exact action enumeration, advantage centering, and reference regularization to the proposed PPO refinement.

Q3: Capacity-matched comparison: Can the authors provide either (a) a strong weakly supervised baseline using the same Qwen2.5-VL backbone, or (b) a GTR-HOI configuration using a backbone closer in capacity and pretraining to RegFormer? Without such a comparison, it is difficult to attribute the reported SOTA gap to the proposed Ground, Transport, and Reinforce design.

A3: We adapt RegFormer's modules for interaction learning to Qwen2.5-VL-3B, replacing its visual and language representations while retaining its learning formulation.

MethodFullRareNon-Rare
RegFormer-Qwen33.1436.6132.10
GTR-HOI36.4442.5434.62

With the same foundation model backbone, GTR-HOI improves Full, Rare, and Non-Rare mAP by 3.30, 5.93, and 2.52, respectively, supporting the contribution of our framework beyond backbone capacity alone. This advantage is consistent with how our framework connects Qwen’s semantic capabilities to spatial representations and supervision for individual human-object pairs.

At the representation level, RegFormer-Qwen aggregates image tokens from the language decoder and matches them against independently encoded class text. GTR-HOI uses PER to extract human, object, and union features directly from the visual encoder, incorporating geometry and role embeddings to model spatial relationships explicitly. It then jointly processes these interaction features with the image, pair coordinates, and candidate texts, allowing candidate representations to incorporate the context of the specific human-object pair.

To learn these pair-specific correspondences under image-level supervision, GTR-HOI further combines multi-perspective grounding with UOT. Grounding generates localization hypotheses, and UOT transfers their spatial support to detector pairs, providing explicit constraints for instance assignment. Together, the dedicated spatial representations and supervision from pseudo-GTs offer a plausible explanation for the performance gap: they help translate Qwen’s pretrained semantic knowledge into predictions for specific human-object pairs.

We have uploaded a revised paper. The changes are:

  1. Comparisons with BCE, KL distillation, majority voting, and EMA teacher self-training are presented in Appendix D.4.

  2. Separate ablations of clipping, exact action enumeration, the reward baseline, and reference KL regularization are reported in Appendix D.4.

  3. The RegFormer-Qwen comparison and analysis are presented in Appendix D.6.

Thank you again for your constructive feedback. Our updated response includes comparisons with alternative refinement objectives, individual PPO component ablations, and a RegFormer adaptation using the same Qwen backbone. When you have a chance, we would appreciate your thoughts on whether these experiments address your concerns about PPO’s contribution and backbone capacity. We would be happy to discuss any remaining questions.

Reviewer_FGv8OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method to deal with weakly supervised HOI detection problem. They leverage MLLM to provide peusdo labels and then

Critical strengths
  1. The paper is well-presented and well-constructed.
  2. The experiment is comprehensive and the method achieves competitive performances.
  3. The problem is valid as the annotation for HOI bounding box is demanding and weakly supervised learning in HOI is necessary.
Critical weaknesses
  1. The pseudo-label reliability depends on the confidence scores predicted from MLLM. However, the self-reported confidence from text is not calibrated. The paper does not evaluate the calibration of these scores or establish whether they reliably reflect grounding correctness. It would be useful to examine the correlation between MLLM-predicted score and localization accuracy and whether calibration improves pseudo-GT selection.
  2. Table 4 shows that high-confidence pseudo-GT achieves 71.35% localization precision. However, they will receive hard positive supervision, which may confuse the model by using confidently incorrecet pseudo-GTs.
  3. The proposed UOT over hard IoU matching and balanced OT are demonstrated, but it is unclear whether the designed UOT has advantage over traditional Hungarian matching, which is widely used in HOI detection training.
  4. The proposed method relies on the MLLM capability to provide accurate initial pseudo-labels. If MLLM has some systematic errors in som complex scenes such as multi-person scenarios, the subsequent stages may have limited ability to correct.
  5. Replacing SAP with PER improves Full mAP from 34.98 to 35.58 (+0.60) in Table 3. While this demonstrates some effectiveness, the improvement is relatively modest.
  6. Stage 3 improves mAP by only 0.86, which relies on agreement among different prompts for the same model and input. Such agreement can only make the model more self-confident, and cannot reflect the correctness.
Questions for authors
  1. Does the MLLM-predicted confidence really reflect the correctness?
  2. What is the influence of the confidently incorrect prediction?
  3. Provide the experiments to compare UOT and hungarian matching.
  4. Please compare the model size, inference latency and training computation with other methods.
8 comments
AuthorsOpenReview ↗

We sincerely appreciate your constructive comments and address them below.

Q1: Does the MLLM-predicted confidence really reflect correctness?

A1: The confidence predicted by the MLLM helps distinguish correct from incorrect localizations. We analyze the 49,277 groups with high spatial agreement on HICO-DET because their selected assignments receive hard positive supervision. We bin these groups by their mean self-reported confidence cˉi\bar c_i. A grounding is correct only when both human and object boxes achieve IoU ≥0.5\geq 0.5 with the same annotated instance of the corresponding HOI label.

Confidence binGroupsCorrect groupsMean confidence (%)Localization accuracy (%)
[0.6,0.7)[0.6,0.7)83636563.2743.66
[0.7,0.8)[0.7,0.8)3,1561,34676.6742.65
[0.8,0.9)[0.8,0.9)9,5485,28984.1355.39
[0.9,1.0][0.9,1.0]35,73728,15991.7578.80
Overall49,27735,15988.8271.35

Higher scores are associated with better localization in the upper bins, although the relationship is not strictly monotonic. Confidence exceeds accuracy in every bin, confirming overconfidence. Thus, these scores provide a reliability cue within this subset.

In our method, Stage 1 determines confidence tiers through spatial agreement across prompts. Calibrating scores after grouping therefore leaves the retained coordinates, group membership, and tiers unchanged. Its downstream influence arises primarily in Stage 2, where confidence contributes to source reliability ξi=cˉiκi\xi_i=\bar c_i\kappa_i and affects UOT costs and source marginals.

Q2: What is the influence of the confidently incorrect prediction?

A2: Confidently incorrect assignments can impair learning, but their measured effect is modest in our diagnostic experiment: masking their positive supervision improves Full mAP by 0.29.

To examine this issue, we first assess the assignments that actually enter supervised fine-tuning. The 71.35% localization precision concerns the original Stage 1 pseudo-GTs, whereas fine-tuning uses assignments to detector pairs retained after UOT matching and selection. Under the same matching criterion requiring the correct HOI label and both boxes to match the same GT instance, these assignments achieve GT matching rates of 94.39% for high confidence and 83.77% for medium confidence. They receive hard and soft positive supervision, respectively. Thus, the assignments receiving hard positive supervision are more reliable than the original pseudo-GT precision alone suggests, although residual mismatches remain.

We then conduct a GT-assisted diagnostic experiment to quantify the influence of these residual mismatches. Using annotations from the training set, we identify high-confidence assignments that fail the matching criterion. We rerun semantic fine-tuning and Stage 3 from the same validity checkpoint, mask only their positive loss terms, and retain all other supervision and training settings.

SettingFullRareNon-Rare
Standard GTR-HOI36.4442.5434.62
GT-assisted masking36.7342.6334.97
Improvement+0.29+0.09+0.35

The improvements show that residual incorrect assignments remain a source of performance loss, with the largest gain on Non-Rare classes in this experiment. However, standard GTR-HOI remains within 0.29 Full mAP of the GT-assisted variant. Together, these analyses indicate that the selected supervision is largely reliable.

Q3: Provide experiments comparing UOT and Hungarian matching.

A3: We compare UOT with Hungarian matching using both assignment quality and downstream HOI detection performance. The detection results below evaluate Stage 2 before Stage 3 refinement.

Matching methodFullRareNon-Rare
Hungarian34.9541.0133.14
UOT35.5841.7333.74
Improvement+0.63+0.72+0.60

UOT improves Full, Rare, and Non-Rare mAP by 0.63, 0.72, and 0.60, respectively. We further examine validity supervision over all evaluated detector pairs.

MethodPositive share (%)Positive precision (%)Negative share (%)Negative precision (%)Ignored share (%)
Hungarian10.6974.4689.3191.300.00
UOT12.7480.3039.3594.3747.91

UOT increases the positive share from 10.69% to 12.74% while improving positive precision by 5.84 percentage points and negative precision by 3.07 points. In the evaluated Hungarian configuration, unmatched pairs receive negative supervision. UOT instead withholds supervision for 47.91% of pairs when the assignment criteria provide insufficient evidence. These results show that the evaluated UOT strategy for transferring supervision produces more accurate positive and negative assignments and improves downstream detection performance.

Q4: Please compare the model size, inference latency and training computation with other methods.

A4: We compare our approach with RegFormer and the original fully supervised DA-HOI in terms of model size, inference latency, and computational cost during training.

Model size, recognition latency, and optimization cost. The optimization cost of GTR-HOI covers Stages 2 and 3 and excludes offline pseudo-GT generation in Stage 1.

MethodTotal parameters (M)Latency (ms/image)Optimization (GPU hours)
RegFormer164.17310.052.79
DA-HOI3778.24664.6177.85
GTR-HOI3780.86466.0652.61

All methods are benchmarked locally on NVIDIA GeForce RTX 5090 GPUs. Inference latency is measured on a single GPU with an image batch size of one, excluding external detector execution. For RegFormer and DA-HOI, we follow the training schedules and optimization settings reported in their respective papers and record the duration of complete training runs in our environment. We express optimization cost in GPU hours by multiplying the elapsed training time in hours by the number of GPUs used and summing across optimization stages. This comparison therefore reflects the computational requirements of each method’s training recipe on the same GPU hardware.

GTR-HOI adds approximately 2.62M parameters, or 0.07%, over DA-HOI, while maintaining comparable inference latency: 66.06 versus 64.61 ms/image. Its measured optimization cost is 52.61 GPU hours, compared with 77.85 GPU hours for DA-HOI. RegFormer remains substantially smaller and faster, reflecting the additional computational cost of incorporating an MLLM.

Q5: Replacing SAP with PER improves Full mAP from 34.98 to 35.58 (+0.60) in Table 3. While this demonstrates some effectiveness, the improvement is relatively modest.

A5: Replacing SAP with PER adds only 2.62M parameters (0.07% relative to DA-HOI) while improving Full mAP by 0.60. As a supporting component of GTR-HOI, PER delivers consistent gains across all three splits with minimal parameter overhead, demonstrating its effectiveness within the overall framework.

Q6: Stage 3 improves mAP by only 0.86, which relies on agreement among different prompts for the same model and input. Such agreement can only make the model more self-confident, and cannot reflect the correctness.

A6: We agree that cross-prompt agreement does not guarantee correctness. Stage 3 uses feedback from the other three views to support both accepting and rejecting candidate interactions, with uncertainty weighting and KL regularization constraining updates.

Ground-truth evaluation supports its effectiveness. Stage 3 improves Full mAP by 0.86 in the main run. Five repeated Stage 3 training runs yield 36.34±0.0736.34 \pm 0.07 Full mAP (mean and sample standard deviation), compared with 35.58 before refinement. Designed prompts outperform random templates by 0.40 Full mAP, while Gaussian weighting contributes 0.34 Full mAP. These results support the refinement's empirical benefit beyond prompt agreement itself, without additional prompt evaluations at inference.

We have uploaded a revised paper. The changes are:

  1. The confidence-bin analysis and calibration discussion are provided in Appendix D.1.

  2. The assignment-quality analysis and GT-assisted masking experiment are presented in Appendix D.3.

  3. The Hungarian comparison appears in Section 4.4 of the main text, with supervision-quality analysis in Appendix D.3.

  4. The comparison of model size, inference latency, and training cost is provided in Appendix D.7.

  5. The PER comparison appears in Section 4.4, with expert ablations in Appendix D.2 and parameter overhead in Appendix D.7.

  6. Stage 3 ablations and repeated runs are reported in Section 4.4 and Appendix D.4, with correction diagnostics in Appendix D.5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.