- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes HarnessHOI, a training-free human-object interaction (HOI) detection framework that integrates perception and interaction reasoning using a single frozen MLLM without auxiliary grounding models.
The framework consists of three main stages. First, interaction-guided perception performs initial participant localization, followed by hypothesis-directed discovery to identify potentially missed participants. When additional visual evidence is needed, targeted visual observation examines selected image regions to recover or refine participant groundings. Second, relation-agnostic geometric adjudication selects a consistent bounding box for each participant independently of the proposed interactions. Third, grounded interaction reasoning assesses human-object pairs and verifies interaction relations independently of the initial interaction proposals.
The method is evaluated on HICO-DET and V-COCO, including open-vocabulary settings. The authors further provide component ablations, controlled comparisons of design choices, and analyses of grounding coverage and inference cost to assess the proposed framework.
Critical strengths
S1. Unified Interaction-Guided Perception and HOI Reasoning with a Single Frozen MLLM
The paper proposes a unified framework that integrates participant localization, interaction-guided discovery, targeted visual observation, and HOI reasoning using a single frozen MLLM, without task-specific fine-tuning or auxiliary grounding models. This structured perception–reasoning pipeline achieves competitive training-free performance on HICO-DET (33.20 Full mAP) and V-COCO (44.65/49.70 S1/S2 role AP), demonstrating the effectiveness of the overall framework.
S2. Explicit Separation of Participant Grounding and Interaction Verification
The framework separates participant grounding from interaction reasoning through relation-agnostic geometric adjudication and proposal-independent relation verification. This design reduces reliance on initial interaction hypotheses when selecting participant boxes and assessing relations. Table 4 provides empirical support for these design choices, showing improvements of 0.41 and 0.70 mAP, respectively, over their controlled alternatives on HICO-DET.
Critical weaknesses
W1. Insufficient Direct Validation of the Effectiveness of Interaction-Guided Perception and Participant Recovery
A central contribution of this paper is interaction-guided perception, which employs hypothesis-directed discovery and targeted visual observation to recover participants missed during initial localization and improve participant grounding. However, the empirical evidence is insufficient to establish how accurately and reliably these mechanisms recover failures of initial localization.
According to Table 25 in Appendix D.3, hypothesis-directed discovery increases grounding coverage by 571 interaction annotations on HICO-DET and 217 role annotations on V-COCO, whereas subsequent targeted visual observation contributes only 12 and 23 additional annotations, respectively. Furthermore, Table 31 in Appendix E reports an average of only 0.19 and 0.40 local observations per image on the two benchmarks.
The authors provide qualitative examples of recovering initially missed objects in Appendix F and demonstrate in Table 4 that interaction-guided discovery and targeted visual observation outperform repeated participant localization and non-directed observation, respectively. These results provide evidence supporting the effectiveness of the proposed design choices. However, the paper does not directly quantify how accurately the MLLM-proposed observation regions contain their intended targets or how frequently these observations successfully recover participants missed during initial localization.
In particular, the grounding coverage reported in Table 25 is measured at the interaction-annotation level. Consequently, it does not clearly distinguish improvements arising from the recovery of previously missed unique human/object instances from those resulting from bounding-box refinement of already localized participants.
Moreover, Table 3 shows that Direct HOI generation achieves 15.62 mAP on HICO-DET, whereas the Initial-instance baseline, which performs visual pair assessment and relation verification over initially localized participants, achieves 31.45 mAP. The complete HarnessHOI framework, which additionally incorporates interaction-guided perception and relation-agnostic geometric adjudication, achieves 33.20 mAP, representing a further improvement of 1.75 mAP over the Initial-instance baseline.
Although the difference between Direct HOI generation and the Initial-instance baseline cannot be attributed exclusively to relation verification, since their candidate generation and assessment procedures also differ, these results highlight the importance of more directly examining the contribution of the additional perceptual stages.
Overall, the experiments demonstrate that interaction-guided perception improves final HOI detection performance. Nevertheless, the extent to which hypothesis-directed discovery and targeted visual observation effectively resolve specific perceptual failures, particularly the recovery of initially missed participants, remains insufficiently characterized. This limits the evidence supporting the practical effectiveness and significance of a central methodological contribution.
W2. Insufficient Evidence of Methodological Distinction from AgentHOI and the Independent Benefits of the Single-MLLM Design
This paper proposes a framework that uses a single frozen MLLM to perform interaction-guided perception, relation-agnostic geometric adjudication, and grounded interaction reasoning.
However, AgentHOI (Lei et al., 2026) already employs multi-round HOI re-mining to discover interactions missed during initial inference and uses instance-specific descriptions together with GroundingDINO to localize interaction participants. Although the two methods differ in their implementations, they share the underlying idea of leveraging previously inferred interactions to guide the discovery of additional interactions or participants.
HarnessHOI differs from AgentHOI by performing participant localization and relation verification using the same MLLM without an auxiliary grounding model, while introducing relation-agnostic geometric adjudication and grounded interaction reasoning. Nevertheless, the independent methodological advantages and significance of these components relative to existing approaches that combine multi-round interaction reasoning with specialized grounding models remain insufficiently established.
The authors provide controlled comparisons of individual design choices in Table 4 and re-evaluate the released AgentHOI predictions using a common evaluator in Appendix C.3. These experiments offer useful evidence for the effectiveness of HarnessHOI. However, the comparisons in Table 4 evaluate alternative configurations within the proposed framework, while neither Direct HOI generation nor repeated participant localization directly reproduces the multi-round HOI re-mining procedure employed by AgentHOI. Furthermore, the common-evaluator comparison with AgentHOI involves different foundation models and localization architectures, making it difficult to attribute the observed performance differences specifically to the newly proposed components of HarnessHOI.
An alternative design would preserve the MLLM-generated interaction hypotheses and target descriptions while delegating participant localization to a frozen specialized grounding model, such as GroundingDINO. Such a configuration would retain the general objective of interaction-guided participant discovery. However, the paper does not directly examine whether using the same MLLM for participant localization and targeted visual observation provides advantages in localization accuracy or computational efficiency over this alternative.
In this regard, Table 31 in Appendix E reports inference costs of 13.26 GPU-worker seconds per image on HICO-DET and 22.07 GPU-worker seconds per image on V-COCO. These measurements represent GPU-worker computation rather than per-image wall-clock latency, but they are relevant to assessing the computational overhead of the proposed framework. Although the authors provide performance–cost comparisons against repeated Direct HOI generation, the absence of a controlled comparison with a specialized grounding-based alternative leaves the accuracy–computation trade-off of the single-MLLM design insufficiently characterized.
The ability to perform the entire pipeline using a single frozen MLLM without auxiliary grounding models is a meaningful advantage. Nevertheless, given that AgentHOI already incorporates interaction-guided re-mining and instance grounding, the paper does not sufficiently establish what distinct methodological advances are provided by its localization, geometric adjudication, and relation verification procedures relative to the closest prior approach. This uncertainty is consequential to my assessment of the paper's novelty and significance.
Questions for authors
Q1. Effectiveness of Interaction-Guided Perception in Recovering Missed Participants
A central contribution of this paper is interaction-guided perception, which employs hypothesis-directed discovery and targeted visual observation to recover participants missed during initial localization and improve participant grounding.
The authors provide stage-wise grounding coverage analyses in Appendix D.3 (Tables 25–27) and evaluate targeted visual observation against non-directed observation in Table 4. However, Table 25 shows that targeted visual observation increases grounding coverage by only 12 interaction annotations on HICO-DET and 23 role annotations on V-COCO. Furthermore, Table 31 reports an average of only 0.19 and 0.40 local observations per image, respectively. These results raise questions about how effectively the proposed observation mechanism addresses failures in initial localization.
I would appreciate clarification on the following points:
(1) For executed targeted visual observations where the intended target can be identified from ground-truth annotations, what proportion of the proposed observation regions actually contain the intended target, and what is the subsequent localization success rate? In particular, how frequently does targeted visual observation successfully recover unique human or object instances that were missed during initial localization?
(2) Among the improvements in grounding coverage achieved through hypothesis-directed discovery and targeted visual observation, how much can be attributed to recovering previously missed participant instances, as opposed to refining the bounding boxes of already localized participants? Since the reported coverage is measured at the interaction-annotation level, it does not directly distinguish these two sources of improvement.
(3) Could the authors provide an oracle-grounding evaluation in which ground-truth human and object bounding boxes are supplied to the relation verification stage, to help distinguish limitations in participant localization from those in grounded interaction reasoning? Additionally, comparing the proposed observation regions against oracle observation regions derived from ground-truth target boxes would help isolate errors arising from region selection and subsequent localization. If these oracle evaluations are not feasible, an analysis based on the existing results that separates localization-related errors from relation reasoning errors would also be informative.
These questions aim to clarify which perceptual failures are actually addressed by interaction-guided perception and how effectively they are resolved. In particular, quantitative evidence on observation-region accuracy and the recovery of previously missed participant instances would materially strengthen the support for this central contribution.
Q2. Methodological Distinction from AgentHOI and the Value of Single-MLLM Localization
AgentHOI (Lei et al., 2026) already employs multi-round HOI re-mining to discover interactions missed during initial inference and uses GroundingDINO to localize interaction participants. HarnessHOI shares the general idea of using interaction hypotheses to guide further participant discovery, although the two methods differ in their implementations.
The authors provide controlled comparisons of individual design choices in Table 4 and re-evaluate the released AgentHOI predictions using a common evaluator in Appendix C.3. These experiments provide useful evidence for the effectiveness of HarnessHOI. However, the cross-method comparison involves different foundation models and localization architectures, making it difficult to isolate the contributions of the newly proposed components relative to the closest prior approach.
I would appreciate clarification on the following points:
(1) What specific methodological capabilities does the proposed hypothesis-directed discovery introduce beyond the multi-round HOI re-mining employed by AgentHOI? Furthermore, what distinct advantages do relation-agnostic geometric adjudication and grounded interaction reasoning provide over the grounding and interaction reasoning procedures used in AgentHOI? In particular, how do the controlled comparisons in Table 4 substantiate these distinctions relative to prior work, rather than only demonstrating improvements over alternative configurations within HarnessHOI?
(2) To assess the value of performing participant localization and targeted visual observation using the same frozen MLLM, would it be possible to conduct a controlled comparison in which the MLLM-generated interaction hypotheses and target descriptions are preserved, while participant localization is delegated to a frozen grounding model such as GroundingDINO? If feasible, comparing participant grounding accuracy, final HOI detection performance, and overall inference cost while keeping the downstream grounded interaction reasoning procedure unchanged would help clarify the benefits and accuracy–computation trade-offs of the single-MLLM design. If such a comparison is not feasible, could the authors identify the evidence in the current experiments that supports the advantages of MLLM-based localization over a specialized grounding model?
The purpose of these questions is not to require the incorporation of an external grounding model or a reduction in inference cost. Rather, it is to assess the distinct methodological contribution of HarnessHOI relative to AgentHOI. Clear empirical evidence supporting the proposed design choices and the value of single-MLLM localization would be important for evaluating the paper's novelty and significance.
11 comments
Response to Reviewer W6rY (2/6): Recovering Missed Instances versus Refining Boxes
Q1(2). Among the improvements in grounding coverage achieved through hypothesis-directed discovery and targeted visual observation, how much can be attributed to recovering previously missed participant instances, as opposed to refining the bounding boxes of already localized participants? Since the reported coverage is measured at the interaction-annotation level, it does not directly distinguish these two sources of improvement.
Recovery and refinement can now be separated at the instance level. Appendix D.3 of the submitted paper reports annotation-level coverage, and Equation (17) states what that quantity can and cannot establish: with M_c the number of annotations that have an eligible matching prediction and G_c the ground-truth count, one eligible prediction per covered annotation yields a class-wise AP bound of U_c(HICO) = (1/11) Σ_t 1[t ≤ M_c/G_c] and U_c(VCOCO) = M_c/G_c. Coverage bounds attainable AP under fixed boxes and labels, but it says nothing about whether a box came from a recovered participant or from a refined one. We have added the instance-level decomposition the question asks for, after deduplicating by image and physical instance and assigning each target to the stage that first brought new coverage.
| Stage | Confirmed initially unlisted | Class/box correction or initially ambiguous identity | Deduplicated total |
|---|---|---|---|
| HICO-DET, hypothesis-directed discovery | 274 | 42 | 316 |
| HICO-DET, targeted observation (submitted trigger) | 0 | 12 | 12 |
| V-COCO, hypothesis-directed discovery | 63 | 0 | 63 |
| V-COCO, targeted observation (submitted trigger) | 5 | 14 | 19 |
| Total, submitted policy | 342 | 68 | 410 |
The 12 newly covered HICO-DET annotations map to 12 unique targets, all involving class/box corrections or identity resolution. The 23 newly covered V-COCO annotations map to 19 unique targets: 5 initially unlisted participants and 14 class/box corrections or identity resolutions. Appendices D.3 and D.4 separately record the 479 / 424 instances whose final box came from observation. Under the submitted policy, discovery recovers 274/63 initially unlisted participants and observation recovers 0/5; observation also refines groundings and supplies local visual evidence.
With two additional triggers for already-boxed targets, at the same two-observation cap, observation recovers 52 / 46 such participants in total (52 / 41 additional over the submitted policy) and adds 140 / 179 covered annotations. Deduplicated observation targets rise to 75 / 103, of which 23 / 57 are class or box corrections. The two datasets together gain 147 new targets beyond the submitted policy (93 initially unlisted, 3 class corrections, 41 box corrections, 10 with initially ambiguous identity). Deduplicated targets from discovery and observation rise from 410 to 557. Table 6(b) in the revised PDF reports this variant as a trigger-sensitivity control, with the submitted policy retained for the main tables. Appendix D.3 provides the deduplication rule and the per-stage breakdown. We can now separate recovery of missed participants from refinement of boxes.
Response to Reviewer W6rY (3/6): Separating Localization and Relation Errors
Q1(3). Could the authors provide an oracle-grounding evaluation in which ground-truth human and object bounding boxes are supplied to the relation verification stage, to help distinguish limitations in participant localization from those in grounded interaction reasoning? Additionally, comparing the proposed observation regions against oracle observation regions derived from ground-truth target boxes would help isolate errors arising from region selection and subsequent localization. If these oracle evaluations are not feasible, an analysis based on the existing results that separates localization-related errors from relation reasoning errors would also be informative.
We added two GT-input conditions and a pair-reasoning ablation on both complete test sets. Table 4 in the submitted paper reports component controls; Appendix D.8 adds these diagnostics under the same matching and AP rules. Equation (6) scores each triplet as s(h,a,o) = q_theta(true | x_rel), where x_rel contains the selected boxes and visual evidence. O1 supplies GT observation regions while fixing every final output box and discarding observation-induced box updates. O2 initializes GT participants, supplies GT observation regions and re-runs downstream prediction; its final boxes are generated by that prediction process. The pair-reasoning ablation retains detection and single-target action classification while removing inference about specific human-target combinations.
| Condition | HICO Default Full | HICO KO Full | V-COCO S1 | V-COCO S2 |
|---|---|---|---|---|
| HarnessHOI | 33.20 | 36.85 | 44.65 | 49.70 |
| O1: GT observation regions, original output boxes held fixed | 36.68 | 40.42 | 48.09 | 53.24 |
| O2: GT participant initialization + GT observation regions, downstream re-run | 40.72 | 45.31 | 58.46 | 62.18 |
| Detection + single-target action classification, pair reasoning removed | 26.18 | 29.34 | 35.72 | 40.18 |
In O1 we replace predicted observation regions with GT-derived ones, hold every final output box fixed, and discard observation-induced box updates. HICO Default Full rises by 3.48, KO Full by 3.57, and V-COCO S1/S2 by 3.44 / 3.54. With the boxes fixed, this gain can only come from visual evidence reaching relation verification. O2 changes the participant initialization and the resulting downstream candidates, raising HICO Default Full and KO Full by another 4.04 / 4.89 points over O1. Removing pair-specific reasoning reduces HICO Default/KO Full by 7.02 / 7.51 and V-COCO S1/S2 by 8.93 / 9.52, demonstrating the contribution of judgments about the specific human-target pair.
Scoring O1 under the original HICO evaluation implementation gives the same +3.48 / +3.57, with a constant 0.23 to 0.24 point offset between evaluators (full method 33.20 vs 32.97, O1 36.68 vs 36.45), and the O2−O1 gap becomes 4.12 / 4.95. O1's gains are identical across evaluation implementations; both preserve the additional improvement from GT participant initialization.
Appendix D.8 of the revised PDF reports the input replacement, box handling and full metrics for all three conditions. Together, these diagnostics quantify the effects of visual evidence, participant initialization and pair-specific reasoning on final detection.
Response to Reviewer W6rY (1/6): Observation Regions and Localization
We sincerely thank you for recognizing the value of single-MLLM perception and reasoning, and for highlighting the explicit separation of participant grounding and interaction verification. We appreciate your detailed questions about observation-region accuracy, recovery of unique participants and the methodological distinction from AgentHOI. We have added request-level diagnostics, instance-recovery analyses, GT-input experiments and controlled localizer comparisons to provide direct evidence on these aspects of the framework.
We measured both region accuracy and localization success on the two complete test sets. Appendix D.3 in the original paper reports coverage from discovery and observation, and we added the request-level diagnostics your question asks for in Table 6(a) of the revised PDF. Of the executed observations, 1,387/1,835 (HICO-DET) and 1,591/1,978 (V-COCO) can be matched uniquely to a ground-truth target, and we use them as the denominator. A localization counts as successful only when the instance is correct and the box reaches IoU ≥ 0.5. Equation (9) maps every box proposed inside a local crop back to full-image coordinates, so a region-level hit and an instance-level recovery are measured in the same frame and the rates below are directly comparable.
The four direct outcomes are mutually exclusive and sum to the denominator.
Among requests where the instance was identified correctly but the box fell below IoU 0.5, observation corrections succeeded in 186/248 (75.0%) and 214/302 (70.9%) cases. Region proposals contain the target centre in 96.7% / 96.0% of uniquely matchable requests. The accepted corrections localize the residual error to the region-to-box step, which observation resolves in 75.0% / 70.9% of deviated cases. Wrong same-class selection occurs in 7.8% (40/512) / 9.4% (64/684) of requests with same-class distractors and in 0% (0/875, 0/907) without them. Deduplicating by image and physical instance gives the instance-level count: targeted observation recovers 0 (HICO-DET) and 5 (V-COCO) unique participants absent from the initial set under the submitted trigger, and 52 / 46 when the trigger is extended. Appendix D.3 gives the full per-stage decomposition. Appendix D.3 in the revised manuscript carries the request-level results and the four-way classification. These results quantify observation-region accuracy and participant recovery on both complete test sets, and show that targeted observation supplies usable visual evidence.