- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces HEIR (Human–Entity Interaction with Roles), a benchmark for evaluating complete human–entity interaction events rather than individual human–object relations. Each event is represented by an actor, an action, and a variable-sized set of participants assigned functional roles. HEIR contains 18,730 images from 29 existing sources, covering 105 actions, 437 noun classes, and six functional roles. The benchmark introduces Role mAP and Set mAP to evaluate relation prediction and exact recovery of complete participant–role sets, respectively.
The paper also proposes CoRISP, a model that combines role-conditioned context aggregation with a normalized participant-set distribution. It uses cardinality and role-composition potentials, dynamic programming for set inference, and a localization-marginalized set likelihood for training.
Experiments compare CoRISP with existing HOI methods on HEIR and V-COCO. The results demonstrate differences between relation-level and complete-event evaluation, with CoRISP achieving the strongest results on HEIR's Repeat and Shared Set mAP and competitive performance on V-COCO. Component ablations and diagnostic analyses further examine the contributions of structured modeling and the effects of candidate coverage on exact-set recovery.
Critical strengths
S1. A meaningful benchmark for evaluating complete event composition beyond individual relations.
HEIR addresses an important limitation of existing HOI evaluation by assessing complete grounded participant–role sets rather than only individual relations. The paper provides convincing evidence that relation-level performance does not fully capture event recovery: even after aligning action weights, 41 of 120 method pairs exhibit ranking reversals between Role AP and Set AP. Furthermore, only 53.58–61.81% of edge-covered multi-participant events are recovered exactly among five evaluated baselines, highlighting failures in selecting the complete set of required participants. The structural evaluation across Multi, Repeat, and Shared events further helps characterize these challenges. This is a significant contribution because it establishes a distinct and measurable evaluation target that existing relation-level metrics do not adequately capture.
S2. A coherent set prediction formulation supported by controlled component ablations.
CoRISP provides a structured approach to complete-event prediction by combining role-conditioned context aggregation, cardinality and role-multiplicity potentials, and a normalized participant-set distribution. The controlled component ablations provide meaningful evidence for these design choices. For example, removing the role-multiplicity potential reduces Repeat Set mAP from 8.99 to 5.39 and Shared Set mAP from 23.80 to 19.96 under the same upstream configuration. Notably, the full model achieves higher Set mAP despite slightly lower aggregate Role mAP, supporting the value of modeling event composition beyond improving individual relation scores. CoRISP also achieves the highest Repeat and Shared Set mAP among the evaluated methods on HEIR. These results provide empirical support for the proposed structured modeling components and their relevance to complete-event recovery.
Critical weaknesses
W1. Limited Control for Differences in Pretrained Components and Total Model Size in Cross-Method Comparisons
The paper provides evidence that CoRISP’s components contribute to set prediction. In Table 2, removing the role-multiplicity potential while retaining the same upstream components reduces Repeat Set mAP from 8.99 to 5.39 and Shared Set mAP from 23.80 to 19.96. The full model also achieves higher Set mAP despite slightly lower aggregate Role mAP than this ablation. These results support benefits for set prediction that cannot be explained simply by an increase in aggregate relation AP. My concern is therefore not that CoRISP’s improvements arise entirely from its backbone or that its individual components lack empirical support.
However, these component ablations should be distinguished from comparisons against existing HOI methods. CoRISP uses frozen H-DETR/Swin-L proposals together with DINOv3 ViT-L/16 and a pretrained DINO.txt vision head, whereas the compared methods retain their respective detectors, visual encoders, and pretraining configurations. For example, InCoM-Net uses ResNet-50 with CLIP-L/14. The cross-method differences therefore reflect not only interaction modeling but also participant proposals and pretrained visual–semantic representations. SL-HOI also uses DINOv3-L/16, so it would be inaccurate to assume that all compared methods use weaker encoders; nevertheless, using the same encoder family does not establish equivalent detection and feature-processing configurations.
Furthermore, the reported approximately 9.9M parameters count only trainable components and exclude the frozen upstream modules. This number does not establish that CoRISP achieves its results with a smaller total model. Conversely, the available evidence does not establish that CoRISP has more total parameters than every compared method or that model size causes its performance advantages. The unresolved issue is the extent to which cross-method gains can be attributed to CoRISP’s proposed formulation when pretrained components and total system size differ.
This distinction is particularly important for complete-set prediction, where a localization or noun error for a single required participant can make the entire event incorrect. Even similar aggregate detection AP would not guarantee comparable participant identities, localization quality, or distractor candidates for the same events.
A representative comparison using shared detector and visual-encoder configurations would therefore complement, rather than replace, the existing component ablations. Without such a comparison or equivalent evidence, it remains unclear whether the reported advantages on Repeat and Shared persist under comparable participant proposals and visual representations. This limits the interpretation of CoRISP’s cross-method advantages and the extent of its methodological contribution.
W2. Limited Characterization of the Sources of Complete-Event Improvements and Their Distribution Across Functional Roles
Section 5.4 distinguishes candidate coverage from exact-set recovery through the action–cardinality oracle. CoRISP’s higher Multi recovery despite lower coverage than SOV-STG Swin-L supports the conclusion that coverage alone does not determine recovery. However, the events covered by the two methods need not be identical, and the oracle supplies the true action and participant count. This analysis should therefore be distinguished from a comparison in which both methods receive comparable participant evidence and must predict complete sets without access to the true action or cardinality.
The classification-with-correct-grounding and edge-covered-event analyses in Appendix C are also useful. The former compares action and role classification on commonly grounded person–participant pairs; the latter shows that extra members can prevent exact recovery even when the required relations are available. Thus, complete-set failures cannot be attributed solely to missed detections. Nevertheless, these analyses do not directly compare CoRISP and the other methods on the same events for which all participants are available. It remains insufficiently characterized whether CoRISP primarily reduces missing participants, excludes additional participants, improves cardinality or role assignment, or ranks correct set hypotheses more effectively.
The distribution of these improvements across functional roles is also unclear. Approximately 91.8% of the relation annotations across the full release are target or support, as calculated from the counts in Appendix A. This is not the test-split distribution, and Role mAP averages over test-supported action–noun–role classes; annotation imbalance alone therefore does not establish that these two roles dominate the metric. However, the current Full, Multi, Repeat, and Shared results do not show whether improvements are concentrated in participant selection for frequent roles or also occur in events involving instrument, source, destination, and constraint.
The role-ambiguity analysis in Figure 3(a) demonstrates the additional difficulty of retaining role labels, but it does not characterize the role-specific contributions of CoRISP’s components. In particular, improvements on Repeat may reflect better selection of multiple participants sharing the same role. This is a valid contribution aligned with the paper’s objective, but it does not by itself establish improved discrimination among different functional roles.
This concern does not challenge HEIR’s evaluation objective or require uniform improvements across all six roles. Rather, it concerns which composition errors CoRISP reduces when the required participant evidence is available, and on which roles and event structures those improvements occur. Such characterization would more directly connect the reported gains to role-conditioned modeling and set potentials and clarify their effectiveness and limitations.
Questions for authors
Q1. Fair Comparison of Detection and Visual Representations
While Table 2 supports the contributions of CoRISP's individual components under the same upstream configuration, the compared methods use different detectors, visual encoders, and pretrained components.
(1) Could the authors provide a controlled comparison with a representative HOI method using shared detector proposals and visual features (e.g., ResNet-50 + CLIP-L/14 or CoRISP's current upstream configuration)? Reporting relation performance, Full/Multi/Repeat/Shared Set mAP, and complete-participant coverage would help clarify whether CoRISP's advantages persist under comparable conditions.
(2) Could the authors clarify the total model size, including frozen components such as H-DETR/Swin-L, DINOv3, and the DINO.txt vision head, rather than reporting only trainable parameters? This would help assess the influence of model capacity and pretrained components on the comparisons.
Q2. Sources of Complete-Event Improvements and Role-Specific Contributions
Although the action–cardinality oracle and common-grounding analyses provide useful evidence, the sources of CoRISP's complete-set improvements remain insufficiently characterized.
(1) Could the authors compare CoRISP and representative methods on the same events with complete-participant coverage, without providing the true action or cardinality? An analysis of missing/extra participants, cardinality errors, and role assignment errors would clarify which aspects of event composition CoRISP improves. Separating failures caused by unavailable proposals from those arising during relation selection would also be informative.
(2) Appendix A shows that approximately 91.8% of relation annotations belong to target and support. Could the authors report role-specific performance and component-ablation gains across all six functional roles, including the corresponding sample sizes? This would clarify whether the improvements extend beyond the dominant roles.
These questions are intended to clarify the sources and scope of CoRISP's improvements rather than prescribe specific experiments. I would also welcome alternative analyses, additional explanations, or any other evidence that addresses the concerns raised above, and would reconsider my assessment accordingly.
4 comments
Reply to Reviewer MnAS 2/3
W2 / Q2(1). Error analysis on commonly covered events
Could the authors compare CoRISP and representative methods on the same events with complete-participant coverage, without providing the true action or cardinality?
We took CoRISP and PViC Swin-L, which share the detector, and kept the 7,715 test events for which both systems' candidate boxes, before relation scoring, contain the actor and every participant at IoU 0.5 with the right noun. Neither system receives the true action or cardinality; the predictions are the ordinary 100-set outputs. For each event, we classify the best-scoring predicted set of the right actor and action, and report recall in the budget, i.e., whether an exact set appears anywhere among the 100. Percent of events; CoRISP: mean of three seeds.
| Events | n | System | Exact | Cardinality | Member error | No set | Recall in budget |
|---|---|---|---|---|---|---|---|
| all | 7,715 | CoRISP | 73.7 | 7.0 | 17.9 | 1.4 | 77.9 |
| PViC Swin-L | 68.5 | 9.6 | 18.0 | 3.9 | 73.0 | ||
| Multi | 768 | CoRISP | 28.4 | 56.6 | 13.6 | 1.4 | 55.8 |
| PViC Swin-L | 0.0 | 96.0 | 0.0 | 4.0 | 45.3 | ||
| Repeat | 591 | CoRISP | 19.2 | 68.6 | 10.3 | 1.9 | 50.1 |
| PViC Swin-L | 0.0 | 94.8 | 0.0 | 5.3 | 42.6 | ||
| Shared | 2,644 | CoRISP | 65.1 | 10.7 | 21.9 | 2.4 | 70.6 |
| PViC Swin-L | 57.3 | 13.2 | 22.8 | 6.8 | 63.8 |
The comparison has to separate the models from the set construction.
- No set for the event. PViC outputs no set for the annotated actor and action 2.8 times as often (3.9 vs 1.4%).
- Recovery within the budget. Under the paper's prefix construction, the highest-scoring set of a pair-output baseline for a Multi or Repeat event is essentially always its single-member prefix (min-member scoring), so the Exact and Cardinality columns of these rows mainly reflect the construction. Recall in the budget does not depend on the ordering of sets within the retained 100-set budget: 77.9 vs 73.0 overall, 55.8 vs 45.3 on Multi, and 50.1 vs 42.6 on Repeat. With the MAP readout of our reply to vYa7 Q5, PViC Swin-L reaches only 5.61 Multi and 6.73 Repeat Set mAP, against CoRISP's 8.50 and 7.04.
- Member errors (right size, wrong members) are similar (17.9 vs 18.0%), so the gain does not come from relabeling roles.
Restricted to these events, Set mAP (Full / Multi / Repeat / Shared) is 27.57 / 17.76 / 9.15 / 29.45 for CoRISP and 21.34 / 10.38 / 7.79 / 20.91 for PViC Swin-L; events that neither covers contribute almost nothing (Set mAP below 0.04 for both). The full-test margins therefore come from events with complete candidate evidence, and they widen when coverage is equalized.
Reply to Reviewer MnAS 3/3
Q2(2). Per-role results and component effects
Could the authors report role-specific performance and component-ablation gains across all six functional roles, including the corresponding sample sizes?
Per-role Role mAP is the mean AP over the test-supported classes of a role (the paper's Role mAP averages all 1,104 classes). Test counts and per-role Role mAP (CoRISP: mean of three seeds):
| Role | Instances | Classes | CoRISP | RLIPv2 Swin-L | SOV-STG Swin-L | SL-HOI | PViC Swin-L | InCoM-Net |
|---|---|---|---|---|---|---|---|---|
| target | 9,609 | 811 | 19.51 | 24.51 | 21.84 | 23.96 | 17.99 | 14.70 |
| support | 1,277 | 96 | 27.98 | 29.38 | 25.11 | 24.07 | 25.61 | 21.56 |
| instrument | 493 | 108 | 22.61 | 29.53 | 28.46 | 24.08 | 22.98 | 12.09 |
| destination | 186 | 43 | 13.58 | 33.55 | 28.36 | 30.52 | 15.77 | 11.51 |
| source | 119 | 31 | 16.93 | 32.06 | 33.54 | 45.55 | 22.24 | 16.08 |
| constraint | 99 | 15 | 1.45 | 18.85 | 21.69 | 29.76 | 4.76 | 2.56 |
| all | 11,783 | 1,104 | 20.00 | 25.91 | 23.35 | 24.92 | 18.99 | 14.79 |
Rare roles. The HEIR-adapted detector shared by CoRISP and PViC Swin-L proposes same-noun candidates for the actor and every participant in only 25% of destination events and 2% of constraint events (table below); 14 of the 15 constraint classes have the action "fasten", with a seat belt, shoelace, strap, or screw as the constraint. RLIPv2 Swin-L and GroupHOI-S, which train their own detection components, recover 56% and 53% of the constraint instances within their relation budget. At relation level, CoRISP's scores are role marginals of a normalized set distribution, so within the 100-relation budget the rare-role relations of an event rank below its target relations (with 1,000 relations, run 1 reaches 22.8 instead of 17.3 on source); Set mAP is not subject to this relation cap, since each set is scored as a whole within the 100-set budget.
Recovery when candidates are available. On the events that both systems fully cover, an exact set appears within the budget more often for CoRISP than for the same-detector baseline for target, support, instrument, source, and destination, in every seed; only two constraint events are covered by both systems, so we draw no conclusion for that role. This measure does not depend on the ordering of sets within the retained 100-set budget (CoRISP: mean of three seeds):
| Participant role | Events with the role | Covered by both | Exact set in budget, CoRISP | Exact set in budget, PViC Swin-L |
|---|---|---|---|---|
| target | 8,471 | 6,601 (78%) | 77.3% | 72.7% |
| support | 1,268 | 983 (78%) | 82.2% | 77.7% |
| instrument | 488 | 214 (44%) | 78.5% | 55.1% |
| source | 118 | 57 (48%) | 73.1% | 68.4% |
| destination | 186 | 47 (25%) | 48.9% | 25.5% |
| constraint | 99 | 2 (2%) | – | – |
Corrected ablation. The numbers quoted in the review (without G, Repeat 8.99 → 5.39 and Shared 23.80 → 19.96) came from the submitted Table 2, whose ablated rows had been read out with the prefix construction instead of CoRISP's set prediction (details in our reply to vYa7 Q5). Read out natively and averaged over three seeds, removing G lowers Repeat by 1.17, Multi by 0.67, and Shared by 0.70; removing context messages lowers Repeat by 1.84; and removing G or role feedback lowers Multi in every seed.
Per-role effect of the components. Change in per-role Role mAP when one component is removed (mean of three seeds; * = same sign in every seed):
| Removed | target | support | instrument | destination | source | constraint |
|---|---|---|---|---|---|---|
| G | +0.39* | +0.09 | −0.13 | −0.90* | −1.35* | +0.10 |
| context messages | +0.31* | +0.61 | +0.30 | +1.04* | −0.98* | +0.59 |
| role feedback | +0.15 | +0.09 | +0.46 | +0.13 | −0.01 | +0.06 |
At relation level, the consistent effects are on the rare roles: G helps source and destination in every seed, and context messages help source in every seed. The components' main effect is on set composition, which Role mAP does not measure.
Summary. With identical candidates and identical features, CoRISP's structural margins persist, and on events both systems fully cover, an exact set more often appears within its budget. We would be glad to provide further analysis.
Kind Reminder to Review Our Rebuttal
Thank you again for your detailed review. As the discussion period closes on November 18, we would be grateful to know whether our replies address your concerns. In brief, we added controls with the same detector and with the same detector and visual features (Q1(1)), the total parameter count of every system (Q1(2)), an error analysis on the events that CoRISP and the same-detector baseline both cover (Q2(1)), and per-role results with event counts and the corrected component ablations (Q2(2)). If any point remains unclear, we would be glad to clarify it before the discussion closes.
Reply to Reviewer MnAS 1/3
We thank the reviewer for the precise statement of what is and is not established, and answer both questions with the controlled evidence requested.
W1 / Q1(1). Comparison under shared detection and visual features
We agree that Table 1 compares complete systems; the revision says so explicitly. Two controls hold the upstream fixed, and a third changes CoRISP's visual encoder (CoRISP: mean of three seeds unless noted).
(a) Same detector. PViC Swin-L uses the boxes of the HEIR-adapted H-DETR Swin-L that CoRISP uses (reply to vYa7 Q2): the same detector boxes and noun labels, which each system filters with its own candidate selection, so the covered events nearly coincide (CoRISP 7,746, PViC Swin-L 7,751, 7,715 in common). CoRISP leads by 3.7 points on Full, 3.4 on Multi, 0.9 on Repeat and 6.3 on Shared.
(b) Same detector and same features. We trained a per-relation head on exactly CoRISP's inputs and schedule: the same frozen boxes, DINOv3 ViT-L/16 features, and DINO.txt head. It scores each relation independently (factorized interaction and conditional role, focal loss) and has no recurrent updates, context messages, set potentials or set likelihood; its sets are read out with CoRISP's exact state-wise MAP decoder, which for pair-output models also gives a higher Repeat than the prefix construction (reply to vYa7 Q5). Relation-level scores are within 0.4 points, yet CoRISP leads by 2.1 on Full, 3.1 on Multi, 4.7 on Repeat and 3.7 on Shared. With identical candidates and features, the structural margins are not explained by CoRISP's inputs.
(c) A different visual encoder. We retrained CoRISP with the same detector, data, schedule and seed as run 1 and replaced only the frozen DINOv3 encoder and DINO.txt head with CLIP ViT-L/14 at 336 px, the encoder family of InCoM-Net, with class prompts re-encoded by the CLIP text tower (Appendix Table 10).
With CLIP features (one run, same seed as run 1), CoRISP stays above every baseline of Table 1 on Repeat and Shared, so the margins do not depend on DINOv3.
The revision words the cross-method claim as "among the evaluated systems", states controls (a) and (c) in Section 5.2 with the details of (c) in Appendix C, and we will add (b) to the supplementary material.
Q1(2). Total model size
For every system, we counted the unique parameters of all networks evaluated at inference, trainable and frozen (detectors, backbones, frozen encoders); offline encoders used only to precompute fixed classifiers are excluded for all systems alike. Millions:
CoRISP's total comprises H-DETR Swin-L (218M), DINOv3 ViT-L/16 (303M), the DINO.txt head and the interaction modules. We agree that 9.9M describes only the trainable part; the revision adds the totals to Table 1 and no longer uses the trainable count as an argument. Capacity alone does not produce the structural margins: the independent head of (b) has the same frozen components.