acceptodds
Under review as a conference paper at ICLR 2027

HEIR: Learning Human-Entity Interactions with Functional Roles

10 views

Abstract

Understanding human–entity interactions requires recovering each person–action event’s participants, roles, and shared identities. This structure can support embodied agents by clarifying who acts on which entities and how, informing anticipation and coordination in shared environments. Standard HOI metrics score individual links, leaving complete event composition undermeasured. We introduce HEIR (Human–Entity Interactions with Functional Roles), an image benchmark for complete grounded participant–role sets across object, interpersonal, and self-directed interactions. It contains 18,730 images, six roles, 105 actions, and 437 nouns, with shared entities, role changes, and repeated fillers; 51.6% of images contain multiple actors and 62.1% contain multiple actions. HEIR pairs relation AP with complete-set AP and structural evaluation. We also introduce CoRISP (Compositional Role-aware Interaction Set Prediction), which uses shared entity identities to combine role-conditioned evidence and predict normalized participant–role sets. Cardinality and role-multiplicity potentials couple assignments through event size and role composition, with exact per-event normalization. Across 16 baselines, relation and complete-event rankings diverge even after aligning action weights. CoRISP leads the evaluated systems on repeated-role events and shared-participant images in HEIR by 2.87 and 3.82 Set mAP points, respectively. On V-COCO, CoRISP achieves 73.72/76.23 role AP and 61.06/68.59 complete-set AP on two-slot actions under Scenarios 1/2. These results show the value of learning and evaluating event composition alongside individual relations. We will release the code and benchmark data.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human-Entity Interaction, Human-Object Interaction Detection, Functional Role Learning, Structured Prediction, Visual Relationship Understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MnASOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces HEIR (Human–Entity Interaction with Roles), a benchmark for evaluating complete human–entity interaction events rather than individual human–object relations. Each event is represented by an actor, an action, and a variable-sized set of participants assigned functional roles. HEIR contains 18,730 images from 29 existing sources, covering 105 actions, 437 noun classes, and six functional roles. The benchmark introduces Role mAP and Set mAP to evaluate relation prediction and exact recovery of complete participant–role sets, respectively.

The paper also proposes CoRISP, a model that combines role-conditioned context aggregation with a normalized participant-set distribution. It uses cardinality and role-composition potentials, dynamic programming for set inference, and a localization-marginalized set likelihood for training.

Experiments compare CoRISP with existing HOI methods on HEIR and V-COCO. The results demonstrate differences between relation-level and complete-event evaluation, with CoRISP achieving the strongest results on HEIR's Repeat and Shared Set mAP and competitive performance on V-COCO. Component ablations and diagnostic analyses further examine the contributions of structured modeling and the effects of candidate coverage on exact-set recovery.

Critical strengths

S1. A meaningful benchmark for evaluating complete event composition beyond individual relations.

HEIR addresses an important limitation of existing HOI evaluation by assessing complete grounded participant–role sets rather than only individual relations. The paper provides convincing evidence that relation-level performance does not fully capture event recovery: even after aligning action weights, 41 of 120 method pairs exhibit ranking reversals between Role AP and Set AP. Furthermore, only 53.58–61.81% of edge-covered multi-participant events are recovered exactly among five evaluated baselines, highlighting failures in selecting the complete set of required participants. The structural evaluation across Multi, Repeat, and Shared events further helps characterize these challenges. This is a significant contribution because it establishes a distinct and measurable evaluation target that existing relation-level metrics do not adequately capture.

S2. A coherent set prediction formulation supported by controlled component ablations.

CoRISP provides a structured approach to complete-event prediction by combining role-conditioned context aggregation, cardinality and role-multiplicity potentials, and a normalized participant-set distribution. The controlled component ablations provide meaningful evidence for these design choices. For example, removing the role-multiplicity potential reduces Repeat Set mAP from 8.99 to 5.39 and Shared Set mAP from 23.80 to 19.96 under the same upstream configuration. Notably, the full model achieves higher Set mAP despite slightly lower aggregate Role mAP, supporting the value of modeling event composition beyond improving individual relation scores. CoRISP also achieves the highest Repeat and Shared Set mAP among the evaluated methods on HEIR. These results provide empirical support for the proposed structured modeling components and their relevance to complete-event recovery.

Critical weaknesses

W1. Limited Control for Differences in Pretrained Components and Total Model Size in Cross-Method Comparisons

The paper provides evidence that CoRISP’s components contribute to set prediction. In Table 2, removing the role-multiplicity potential GG while retaining the same upstream components reduces Repeat Set mAP from 8.99 to 5.39 and Shared Set mAP from 23.80 to 19.96. The full model also achieves higher Set mAP despite slightly lower aggregate Role mAP than this ablation. These results support benefits for set prediction that cannot be explained simply by an increase in aggregate relation AP. My concern is therefore not that CoRISP’s improvements arise entirely from its backbone or that its individual components lack empirical support.

However, these component ablations should be distinguished from comparisons against existing HOI methods. CoRISP uses frozen H-DETR/Swin-L proposals together with DINOv3 ViT-L/16 and a pretrained DINO.txt vision head, whereas the compared methods retain their respective detectors, visual encoders, and pretraining configurations. For example, InCoM-Net uses ResNet-50 with CLIP-L/14. The cross-method differences therefore reflect not only interaction modeling but also participant proposals and pretrained visual–semantic representations. SL-HOI also uses DINOv3-L/16, so it would be inaccurate to assume that all compared methods use weaker encoders; nevertheless, using the same encoder family does not establish equivalent detection and feature-processing configurations.

Furthermore, the reported approximately 9.9M parameters count only trainable components and exclude the frozen upstream modules. This number does not establish that CoRISP achieves its results with a smaller total model. Conversely, the available evidence does not establish that CoRISP has more total parameters than every compared method or that model size causes its performance advantages. The unresolved issue is the extent to which cross-method gains can be attributed to CoRISP’s proposed formulation when pretrained components and total system size differ.

This distinction is particularly important for complete-set prediction, where a localization or noun error for a single required participant can make the entire event incorrect. Even similar aggregate detection AP would not guarantee comparable participant identities, localization quality, or distractor candidates for the same events.

A representative comparison using shared detector and visual-encoder configurations would therefore complement, rather than replace, the existing component ablations. Without such a comparison or equivalent evidence, it remains unclear whether the reported advantages on Repeat and Shared persist under comparable participant proposals and visual representations. This limits the interpretation of CoRISP’s cross-method advantages and the extent of its methodological contribution.

W2. Limited Characterization of the Sources of Complete-Event Improvements and Their Distribution Across Functional Roles

Section 5.4 distinguishes candidate coverage from exact-set recovery through the action–cardinality oracle. CoRISP’s higher Multi recovery despite lower coverage than SOV-STG Swin-L supports the conclusion that coverage alone does not determine recovery. However, the events covered by the two methods need not be identical, and the oracle supplies the true action and participant count. This analysis should therefore be distinguished from a comparison in which both methods receive comparable participant evidence and must predict complete sets without access to the true action or cardinality.

The classification-with-correct-grounding and edge-covered-event analyses in Appendix C are also useful. The former compares action and role classification on commonly grounded person–participant pairs; the latter shows that extra members can prevent exact recovery even when the required relations are available. Thus, complete-set failures cannot be attributed solely to missed detections. Nevertheless, these analyses do not directly compare CoRISP and the other methods on the same events for which all participants are available. It remains insufficiently characterized whether CoRISP primarily reduces missing participants, excludes additional participants, improves cardinality or role assignment, or ranks correct set hypotheses more effectively.

The distribution of these improvements across functional roles is also unclear. Approximately 91.8% of the relation annotations across the full release are target or support, as calculated from the counts in Appendix A. This is not the test-split distribution, and Role mAP averages over test-supported action–noun–role classes; annotation imbalance alone therefore does not establish that these two roles dominate the metric. However, the current Full, Multi, Repeat, and Shared results do not show whether improvements are concentrated in participant selection for frequent roles or also occur in events involving instrument, source, destination, and constraint.

The role-ambiguity analysis in Figure 3(a) demonstrates the additional difficulty of retaining role labels, but it does not characterize the role-specific contributions of CoRISP’s components. In particular, improvements on Repeat may reflect better selection of multiple participants sharing the same role. This is a valid contribution aligned with the paper’s objective, but it does not by itself establish improved discrimination among different functional roles.

This concern does not challenge HEIR’s evaluation objective or require uniform improvements across all six roles. Rather, it concerns which composition errors CoRISP reduces when the required participant evidence is available, and on which roles and event structures those improvements occur. Such characterization would more directly connect the reported gains to role-conditioned modeling and set potentials and clarify their effectiveness and limitations.

Questions for authors

Q1. Fair Comparison of Detection and Visual Representations

While Table 2 supports the contributions of CoRISP's individual components under the same upstream configuration, the compared methods use different detectors, visual encoders, and pretrained components.

(1) Could the authors provide a controlled comparison with a representative HOI method using shared detector proposals and visual features (e.g., ResNet-50 + CLIP-L/14 or CoRISP's current upstream configuration)? Reporting relation performance, Full/Multi/Repeat/Shared Set mAP, and complete-participant coverage would help clarify whether CoRISP's advantages persist under comparable conditions.

(2) Could the authors clarify the total model size, including frozen components such as H-DETR/Swin-L, DINOv3, and the DINO.txt vision head, rather than reporting only trainable parameters? This would help assess the influence of model capacity and pretrained components on the comparisons.

Q2. Sources of Complete-Event Improvements and Role-Specific Contributions

Although the action–cardinality oracle and common-grounding analyses provide useful evidence, the sources of CoRISP's complete-set improvements remain insufficiently characterized.

(1) Could the authors compare CoRISP and representative methods on the same events with complete-participant coverage, without providing the true action or cardinality? An analysis of missing/extra participants, cardinality errors, and role assignment errors would clarify which aspects of event composition CoRISP improves. Separating failures caused by unavailable proposals from those arising during relation selection would also be informative.

(2) Appendix A shows that approximately 91.8% of relation annotations belong to target and support. Could the authors report role-specific performance and component-ablation gains across all six functional roles, including the corresponding sample sizes? This would clarify whether the improvements extend beyond the dominant roles.

These questions are intended to clarify the sources and scope of CoRISP's improvements rather than prescribe specific experiments. I would also welcome alternative analyses, additional explanations, or any other evidence that addresses the concerns raised above, and would reconsider my assessment accordingly.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer MnAS 1/3

We thank the reviewer for the precise statement of what is and is not established, and answer both questions with the controlled evidence requested.

W1 / Q1(1). Comparison under shared detection and visual features

Could the authors provide a controlled comparison with a representative HOI method using shared detector proposals and visual features (e.g., ResNet-50 + CLIP-L/14 or CoRISP's current upstream configuration)?

We agree that Table 1 compares complete systems; the revision says so explicitly. Two controls hold the upstream fixed, and a third changes CoRISP's visual encoder (CoRISP: mean of three seeds unless noted).

ControlSystemRoleFullMultiRepeatShared
(a) same detectorPViC Swin-L18.9914.575.116.1216.62
CoRISP20.0018.298.507.0422.95
(b) same detector and featuresindependent head19.8616.435.852.2619.23
CoRISP (run 1)20.1918.568.976.9422.91

(a) Same detector. PViC Swin-L uses the boxes of the HEIR-adapted H-DETR Swin-L that CoRISP uses (reply to vYa7 Q2): the same detector boxes and noun labels, which each system filters with its own candidate selection, so the covered events nearly coincide (CoRISP 7,746, PViC Swin-L 7,751, 7,715 in common). CoRISP leads by 3.7 points on Full, 3.4 on Multi, 0.9 on Repeat and 6.3 on Shared.

(b) Same detector and same features. We trained a per-relation head on exactly CoRISP's inputs and schedule: the same frozen boxes, DINOv3 ViT-L/16 features, and DINO.txt head. It scores each relation independently (factorized interaction and conditional role, focal loss) and has no recurrent updates, context messages, set potentials or set likelihood; its sets are read out with CoRISP's exact state-wise MAP decoder, which for pair-output models also gives a higher Repeat than the prefix construction (reply to vYa7 Q5). Relation-level scores are within 0.4 points, yet CoRISP leads by 2.1 on Full, 3.1 on Multi, 4.7 on Repeat and 3.7 on Shared. With identical candidates and features, the structural margins are not explained by CoRISP's inputs.

(c) A different visual encoder. We retrained CoRISP with the same detector, data, schedule and seed as run 1 and replaced only the frozen DINOv3 encoder and DINO.txt head with CLIP ViT-L/14 at 336 px, the encoder family of InCoM-Net, with class prompts re-encoded by the CLIP text tower (Appendix Table 10).

CoRISP encoderRoleFullMultiRepeatShared
DINOv3 ViT-L/16 + DINO.txt (run 1)20.1918.568.976.9422.91
CLIP ViT-L/14-33621.0619.159.187.2723.90
strongest baseline of Table 1, per column25.9122.219.876.1219.98

With CLIP features (one run, same seed as run 1), CoRISP stays above every baseline of Table 1 on Repeat and Shared, so the margins do not depend on DINOv3.

The revision words the cross-method claim as "among the evaluated systems", states controls (a) and (c) in Section 5.2 with the details of (c) in Appendix C, and we will add (b) to the supplementary material.

Q1(2). Total model size

Could the authors clarify the total model size, including frozen components such as H-DETR/Swin-L, DINOv3, and the DINO.txt vision head, rather than reporting only trainable parameters?

For every system, we counted the unique parameters of all networks evaluated at inference, trainable and frozen (detectors, backbones, frozen encoders); offline encoders used only to precompute fixed classifiers are excluded for all systems alike. Millions:

MethodTrainableTotal
QPIC R50 / R10141.6 / 60.541.8 / 60.7
MUREN75.175.3
GEN-VLKT-S / L46.9 / 75.447.2 / 75.6
SOV-STG-S / R101 / Swin-L54.1 / 87.4 / 240.454.4 / 87.6 / 240.8
SOV-STG-VLA-S88.21268.5
PViC R50 / Swin-L12.2 / 12.160.4 / 230.6
RLIPv2 Swin-T / Swin-L213.4 / 382.8214.2 / 383.8
GroupHOI-S82.0231.8
InCoM-Net42.2388.1
SL-HOI31.1459.8
CoRISP9.9556.9

CoRISP's total comprises H-DETR Swin-L (218M), DINOv3 ViT-L/16 (303M), the DINO.txt head and the interaction modules. We agree that 9.9M describes only the trainable part; the revision adds the totals to Table 1 and no longer uses the trainable count as an argument. Capacity alone does not produce the structural margins: the independent head of (b) has the same frozen components.

AuthorsOpenReview ↗

Reply to Reviewer MnAS 2/3

W2 / Q2(1). Error analysis on commonly covered events

Could the authors compare CoRISP and representative methods on the same events with complete-participant coverage, without providing the true action or cardinality?

We took CoRISP and PViC Swin-L, which share the detector, and kept the 7,715 test events for which both systems' candidate boxes, before relation scoring, contain the actor and every participant at IoU 0.5 with the right noun. Neither system receives the true action or cardinality; the predictions are the ordinary 100-set outputs. For each event, we classify the best-scoring predicted set of the right actor and action, and report recall in the budget, i.e., whether an exact set appears anywhere among the 100. Percent of events; CoRISP: mean of three seeds.

EventsnSystemExactCardinalityMember errorNo setRecall in budget
all7,715CoRISP73.77.017.91.477.9
PViC Swin-L68.59.618.03.973.0
Multi768CoRISP28.456.613.61.455.8
PViC Swin-L0.096.00.04.045.3
Repeat591CoRISP19.268.610.31.950.1
PViC Swin-L0.094.80.05.342.6
Shared2,644CoRISP65.110.721.92.470.6
PViC Swin-L57.313.222.86.863.8

The comparison has to separate the models from the set construction.

  • No set for the event. PViC outputs no set for the annotated actor and action 2.8 times as often (3.9 vs 1.4%).
  • Recovery within the budget. Under the paper's prefix construction, the highest-scoring set of a pair-output baseline for a Multi or Repeat event is essentially always its single-member prefix (min-member scoring), so the Exact and Cardinality columns of these rows mainly reflect the construction. Recall in the budget does not depend on the ordering of sets within the retained 100-set budget: 77.9 vs 73.0 overall, 55.8 vs 45.3 on Multi, and 50.1 vs 42.6 on Repeat. With the MAP readout of our reply to vYa7 Q5, PViC Swin-L reaches only 5.61 Multi and 6.73 Repeat Set mAP, against CoRISP's 8.50 and 7.04.
  • Member errors (right size, wrong members) are similar (17.9 vs 18.0%), so the gain does not come from relabeling roles.

Restricted to these events, Set mAP (Full / Multi / Repeat / Shared) is 27.57 / 17.76 / 9.15 / 29.45 for CoRISP and 21.34 / 10.38 / 7.79 / 20.91 for PViC Swin-L; events that neither covers contribute almost nothing (Set mAP below 0.04 for both). The full-test margins therefore come from events with complete candidate evidence, and they widen when coverage is equalized.

AuthorsOpenReview ↗

Reply to Reviewer MnAS 3/3

Q2(2). Per-role results and component effects

Could the authors report role-specific performance and component-ablation gains across all six functional roles, including the corresponding sample sizes?

Per-role Role mAP is the mean AP over the test-supported classes of a role (the paper's Role mAP averages all 1,104 classes). Test counts and per-role Role mAP (CoRISP: mean of three seeds):

RoleInstancesClassesCoRISPRLIPv2 Swin-LSOV-STG Swin-LSL-HOIPViC Swin-LInCoM-Net
target9,60981119.5124.5121.8423.9617.9914.70
support1,2779627.9829.3825.1124.0725.6121.56
instrument49310822.6129.5328.4624.0822.9812.09
destination1864313.5833.5528.3630.5215.7711.51
source1193116.9332.0633.5445.5522.2416.08
constraint99151.4518.8521.6929.764.762.56
all11,7831,10420.0025.9123.3524.9218.9914.79

Rare roles. The HEIR-adapted detector shared by CoRISP and PViC Swin-L proposes same-noun candidates for the actor and every participant in only 25% of destination events and 2% of constraint events (table below); 14 of the 15 constraint classes have the action "fasten", with a seat belt, shoelace, strap, or screw as the constraint. RLIPv2 Swin-L and GroupHOI-S, which train their own detection components, recover 56% and 53% of the constraint instances within their relation budget. At relation level, CoRISP's scores are role marginals of a normalized set distribution, so within the 100-relation budget the rare-role relations of an event rank below its target relations (with 1,000 relations, run 1 reaches 22.8 instead of 17.3 on source); Set mAP is not subject to this relation cap, since each set is scored as a whole within the 100-set budget.

Recovery when candidates are available. On the events that both systems fully cover, an exact set appears within the budget more often for CoRISP than for the same-detector baseline for target, support, instrument, source, and destination, in every seed; only two constraint events are covered by both systems, so we draw no conclusion for that role. This measure does not depend on the ordering of sets within the retained 100-set budget (CoRISP: mean of three seeds):

Participant roleEvents with the roleCovered by bothExact set in budget, CoRISPExact set in budget, PViC Swin-L
target8,4716,601 (78%)77.3%72.7%
support1,268983 (78%)82.2%77.7%
instrument488214 (44%)78.5%55.1%
source11857 (48%)73.1%68.4%
destination18647 (25%)48.9%25.5%
constraint992 (2%)––

Corrected ablation. The numbers quoted in the review (without G, Repeat 8.99 → 5.39 and Shared 23.80 → 19.96) came from the submitted Table 2, whose ablated rows had been read out with the prefix construction instead of CoRISP's set prediction (details in our reply to vYa7 Q5). Read out natively and averaged over three seeds, removing G lowers Repeat by 1.17, Multi by 0.67, and Shared by 0.70; removing context messages lowers Repeat by 1.84; and removing G or role feedback lowers Multi in every seed.

Per-role effect of the components. Change in per-role Role mAP when one component is removed (mean of three seeds; * = same sign in every seed):

Removedtargetsupportinstrumentdestinationsourceconstraint
G+0.39*+0.09−0.13−0.90*−1.35*+0.10
context messages+0.31*+0.61+0.30+1.04*−0.98*+0.59
role feedback+0.15+0.09+0.46+0.13−0.01+0.06

At relation level, the consistent effects are on the rare roles: G helps source and destination in every seed, and context messages help source in every seed. The components' main effect is on set composition, which Role mAP does not measure.

Summary. With identical candidates and identical features, CoRISP's structural margins persist, and on events both systems fully cover, an exact set more often appears within its budget. We would be glad to provide further analysis.

AuthorsOpenReview ↗

Kind Reminder to Review Our Rebuttal

Thank you again for your detailed review. As the discussion period closes on November 18, we would be grateful to know whether our replies address your concerns. In brief, we added controls with the same detector and with the same detector and visual features (Q1(1)), the total parameter count of every system (Q1(2)), an error analysis on the events that CoRISP and the same-detector baseline both cover (Q2(1)), and per-role results with event counts and the corrected component ablations (Q2(2)). If any point remains unclear, we would be glad to clarify it before the discussion closes.

Reviewer_vYa7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the recovery of complete participant–role sets for person–action events in images. It introduces HEIR, a benchmark containing 18,730 images, 105 actions, 437 nouns, and six functional roles, with image-local entity identities, repeated role fillers, and object, interpersonal, and self-directed interactions. Its evaluation combines role-qualified relation AP with complete-event Set AP. The proposed CoRISP model aggregates role-conditioned context and defines a normalized distribution over candidate participant assignments using cardinality and role-multiplicity potentials, with exact per-event inference over retained candidates and fixed support. Experiments compare 16 baselines, investigate discrepancies between relation and event metrics, and analyze candidate coverage and set recovery. CoRISP achieves the strongest reported results on HEIR repeated-role events and shared-participant images, with additional evidence from V-COCO two-slot actions.

Critical strengths
  1. The benchmark defines a meaningful evaluation target that is insufficiently captured by individual relation accuracy. Section 3 and Equation (1) require recovery of the complete participant–role set, including the exclusion of incorrect extra members, while retaining shared entity identities and permitting repeated fillers for a role. These choices expose concrete failures that a collection of individually correct relations does not rule out. The combination of these structures in a common benchmark is a substantive contribution, independently of whether the proposed model leads the aggregate leaderboard.
  2. The benchmark produces useful diagnostic findings beyond reporting new scores. Section 5.3 and Figure 3 show that role-label errors concentrate in action–noun contexts admitting multiple observed roles. They also show that relation/event ranking reversals persist after matching action weights: 41 of 120 model pairs reverse order, including 10 of 55 pairs within the shared-inventory subset. These controls make the case for separate event evaluation more convincing than a comparison of differently averaged headline metrics. This is an informative empirical contribution even where CoRISP is not the strongest system.
  3. The model is technically aligned with the event-composition problem. Section 4 and Appendix B use cardinality and role-multiplicity potentials to represent structural properties that the benchmark explicitly evaluates. The dynamic program computes the event partition function over the stated candidate and support space, and the supervision marginalizes over compatible complete assignments rather than arbitrarily selecting one localization match. This provides a coherent connection between the prediction target, model, and learning objective. The reviewed formulation does not reveal a confirmed mathematical error, although its exactness is conditional on the retained candidates and fixed support.
  4. The reported improvements on the targeted structures are supported by controlled component evidence. In Table 1, CoRISP improves Repeat and Shared Set mAP over the strongest corresponding baselines by 2.87 and 3.82 points. Table 2 uses the same set-decoding rule and training protocol across component removals: removing the role-composition potential reduces Repeat from 8.99 to 5.39, while removing context reduces Shared from 23.80 to 19.75. These results support the usefulness of the proposed components within the system. The paper also appropriately acknowledges that other methods lead overall HEIR Set mAP and Multi, which keeps the performance claim focused.
  5. The coverage and conditional-classification analyses help explain where event recovery succeeds or fails. Section 5.4 and Tables 3/6 distinguish whether complete noun-compatible evidence is available from whether the correct members are selected. For example, CoRISP has lower Multi coverage than SOV-STG Swin-L, but higher recovery. The common-grounding analysis additionally retains 10,383 jointly recoverable person–participant pairs, including negative pairs, and reports conditional Role mAP of 61.50 versus 50.26 for SL-HOI. These analyses do not fully control visual features or detection confidence, but they meaningfully reduce the ambiguity of interpreting final Set AP alone.
Critical weaknesses
  1. The actual scope of test-time false-positive accounting is insufficiently clear. Section 3 and Appendix A describe complete human-corrected interaction annotations. However, the supplied evaluation/heir_sets.py, in prediction(), supports an agent_only scope that discards predicted sets unless their action and actor localization match an annotated actor–action scope. This behavior can change false-positive accounting and therefore the interpretation of complete-event AP. The code behavior is identifiable, but its use in the reported test results is not: the actual annotation files are absent. This is a protocol ambiguity, not evidence of an invalid result or improper filtering. The concern would be resolved by documenting the scope definitions, their counts in each split, and their consistent application across methods; if no test records use this branch, that should be stated explicitly.
  2. The HEIR-specific adaptation of the upstream detector is not sufficiently specified to reconstruct the complete system. Section 5.1 and Appendix B describe frozen H-DETR/Swin-L proposals, whereas heir_rift/model.py, in build(), requires a completed heir_detector_adaptation_v2 checkpoint with HEIR-specific classification heads and noun ordering. The supplied material does not provide a complete adaptation recipe. Freezing a previously adapted detector is entirely consistent, but the initialization, adaptation data, schedule, and checkpoint-selection rule remain important experimental information. Section 5.4 itself establishes that candidate coverage substantially affects event recovery, so this is consequential for reproducibility and interpretation of cross-system differences. Providing the adaptation configuration and checkpoint provenance would address the concern. It does not invalidate the component ablations that share the same detector.
Questions for authors
  1. Which annotation scopes were actually used in the reported HEIR evaluation? Please distinguish completeness of participants within an annotated actor–action event from completeness of the actor–action events annotated in an image. How many train, validation, and test images use agent_only, and are equivalent scope rules applied to every evaluated method and relevant metric? A zero count in the test split would remove the main concern in Critical Weakness 1. If the scope is used, its source distribution and clearly scoped results would determine whether the headline claims need qualification. Simply counting all test images does not establish exhaustive action coverage.
  2. How was the required HEIR-adapted H-DETR checkpoint produced? Please provide the initial checkpoint, adaptation supervision, training configuration, validation/checkpoint-selection procedure, and provenance of the resulting asset. Please also explain how this adaptation relates to the detection components used by the closest HEIR baselines. The question concerns traceable task-specific training, rather than an exhaustive audit of generic large-scale pretraining. This information would resolve the reconstruction issue and help assess whether candidate-coverage differences reflect comparable training conditions.
  3. What information was used to construct and freeze the 10,072-entry semantic compatibility inventory? The README describes AI-assisted semantic assessment followed by two human reviewers, and the model uses this inventory to define admissible positive states. Was construction based only on the vocabulary and role definitions, or did it also inspect instance annotations from particular splits? Please provide the table or its construction rules and report its coverage of annotated positives, especially the 133 test-supported classes with zero training instances. A vocabulary-based, consistently shared prior would substantially alleviate the concern. Undisclosed dependence on instance-level evaluation annotations, or systematic exclusion of valid positives, would change the interpretation of the results. The current evidence does not establish either problem.
  4. Do the actual V-COCO baseline checkpoints recover their expected strength under their original evaluation conventions? Table 1 reports 70.11/73.80 for the reimplemented InCoM-Net, whereas Table 7 lists 73.6/75.4 from the literature; the corresponding PViC Swin-L values are 60.85/68.72 and 64.1/70.2. The paper correctly notes that evaluation conventions differ, so these gaps do not themselves establish implementation problems. Could the exact checkpoints used in Table 1 be evaluated under their respective original conventions, with any remaining discrepancy explained? This matters particularly for InCoM-Net, the closest competitor on several set-level metrics. Rescoring existing predictions would help distinguish protocol effects from reduced baseline strength.
  5. How much of the set-level advantage remains under a common readout of the learned relation marginals? Appendix C uses prefix enumeration with minimum-member scores for the baselines, whereas CoRISP uses state-wise MAP assignments and normalized set probabilities. Could the fixed CoRISP checkpoint also be evaluated using the common-prefix decoder on Full, Multi, Repeat, and Shared? This would help separate the contribution of improved relation evidence from that of the structured set readout. Table 2 already supports the usefulness of the composition potential under a fixed native decoder, so this question does not presume that the gains are merely post-processing. A strong result under both readouts would strengthen the interpretation of the learned evidence; a large gap would clarify the central role of structured decoding.
  6. What improvement is observed specifically on events that participate in cross-event entity sharing? Appendix C defines Shared using all 3,217 events in 534 images containing shared participants, while Table 2 removes the event, pair, and shared-entity context neighborhoods together. Could existing predictions be analyzed separately for events directly connected through shared participants and other events in those images, preferably controlling the compared action set? Evidence of fewer identity splits or incorrect bindings on the directly shared events would strengthen the proposed explanation. Otherwise, the demonstrated conclusion should remain the narrower one already used in parts of the paper: better event recovery in shared-participant images. A new whole-scene exact-match benchmark is not required to answer this question.
  7. Is there an existing audit of residual event-completeness and identity errors after human review? Appendix A describes substantial quality control, including full human correction and a second review by a different person. Given Equation (1), however, a missing ground-truth participant can make an otherwise correct prediction fail as an entire event, and an identity error can affect multiple events through shared correspondence. Could the authors provide existing review-change statistics or a representative audit stratified by Multi, Repeat, and Shared, distinguishing participant omissions, role disagreements, and identity errors? Low residual error would strengthen confidence in the benchmark; systematic structure-dependent errors could affect the interpretation of its central comparisons. This is a request for targeted validation, not an assumption that the annotations are unreliable or a request to relabel the full dataset.
5 comments
AuthorsOpenReview ↗

Reply to Reviewer vYa7 1/4

We thank the reviewer for the careful reading of both the paper and the supplementary code, and answer every question in order.

Q1 / W1. Annotation scope in the HEIR evaluation

Which annotation scopes were actually used in the reported HEIR evaluation?

Two levels of completeness. By protocol, every annotated person–action event is participant-complete: all its participants, roles, and entity identities are annotated and were manually corrected (Q7 estimates the residual omissions). The agent_only scope concerns the second level, whether every person–action event of an image is annotated. Under it, a predicted set whose actor and action match no annotated actor–action of the image is neither a true nor a false positive; sets whose actor–action is annotated are scored exactly as under the full scope, including the penalties for extra members and wrong roles.

Affected images. agent_only is an image property stored with the annotation and applies exactly to the 1,828 HEIR images drawn from SWiG, which were selected around one salient agent: train 1,358 of 15,158 images (9.0%), validation 63 of 615 (10.2%), test 407 of 2,957 (13.8%). The 407 test images hold 810 of the 11,783 test relations and 407 of the 9,955 test events. The rule is identical for every system and metric; like COCO's crowd regions, it neither credits nor penalizes predictions outside the annotated agent's event. Appendix A states the rule, and Section 3 now describes annotation completeness at the event level.

Effect on the reported results. We rescored the seven locally scorable systems with the rule disabled (A: every prediction outside the annotated agent's event counts as a false positive) and with the 407 images removed (B). Shared is unaffected by construction. Paper / A / B:

SystemRole mAPFullMultiRepeat
CoRISP (3 seeds)20.00 / 18.13 / 22.3518.29 / 16.77 / 20.308.50 / 6.30 / 9.507.04 / 6.57 / 7.45
RLIPv2 Swin-L25.91 / 24.56 / 25.2322.21 / 21.30 / 22.727.26 / 6.97 / 6.264.66 / 4.54 / 4.84
PViC Swin-L18.99 / 16.80 / 20.4714.57 / 13.10 / 16.335.11 / 5.07 / 6.126.12 / 6.11 / 6.39
InCoM-Net14.79 / 14.02 / 16.6314.94 / 14.16 / 17.214.58 / 4.48 / 5.545.18 / 5.05 / 5.35
PViC R5016.02 / 14.86 / 18.1213.83 / 13.52 / 15.293.93 / 3.53 / 5.574.78 / 4.16 / 4.92
GroupHOI-S13.32 / 11.96 / 14.2814.52 / 13.73 / 15.954.22 / 3.98 / 4.653.43 / 3.40 / 3.53
SL-HOI24.92 / – / –10.93 / 8.36 / 11.321.78 / 0.78 / 1.560.27 / 0.23 / 0.28

SL-HOI was rescored at the set level only.

Disabling the rule lowers every score of every system, since predictions outside the annotated agent's event become false positives for all methods alike; removing the images raises almost every score. Under both alternatives, the orderings on Role mAP, HOI mAP, and Shared are unchanged, and CoRISP remains first on Repeat and Shared, the strata behind the paper's claims. On Multi, it falls below RLIPv2 Swin-L (6.30 vs 6.97) only with the rule disabled.

Q2 / W2. Provenance of the HEIR-adapted H-DETR checkpoint

How was the required HEIR-adapted H-DETR checkpoint produced?

We agree that the adaptation is part of the system. The revision adds this recipe to Appendix B, and we will release its configuration, training script, and checkpoint.

  • Initialization: the official COCO-trained H-DETR Swin-L; no data other than COCO and the HEIR training boxes enters.
  • Supervision: boxes of the 437 HEIR nouns. COCO classifier rows go to same-name nouns, and the rest are newly initialized. Data: the 13,800 fully box-annotated training images (the 1,358 SWiG images, with incomplete boxes, are excluded); no validation or test image is used for training.
  • Training: the whole detector, backbone included; AdamW, learning rate 2e-4 (heads) / 2e-5 (backbone), weight decay 0.05, gradient clipping 0.1, batch 16, 36 epochs with a ×0.1 step after epoch 30.
  • Selection: best COCO-style box AP on the 552 fully annotated validation images (epoch 8, AP 27.2); then frozen.
  • Baselines: PViC Swin-L in Table 1 uses exactly these boxes and PViC R50 its R50 counterpart (same recipe); InCoM-Net uses a DETR-R50 that we adapted from its COCO checkpoint on the same 13,800 training images with DETR's learning rates for 30 epochs; all other baselines train their own detection components on HEIR with their official recipes.
AuthorsOpenReview ↗

Reply to Reviewer vYa7 2/4

Q3. Construction of the compatibility inventory

What information was used to construct and freeze the 10,072-entry semantic compatibility inventory?

Definition. 10,072 semantically admissible (action, noun, role) triples that define the support of CoRISP's set distribution. It carries no instance information (images, boxes, counts, or splits), and the scorer does not use it.

Construction. (1) An AI-assisted pass proposed rules for all 105 × 437 × 6 combinations from the action and role definitions. (2) Human reviewers adjudicated the low-confidence rules, checking the draft against the annotated triples of all splits. (3) The remaining annotated triples were added, so that no annotated class lies outside the support. The draft of steps 1–2 already admitted 2,529 of the 2,972 annotated triples (85.1%), covering 97.6 / 98.5 / 98.3% of the train / validation / test relations. Step 3 added 443 triples, of which 23 occur only in test (29 of the 11,783 test relations). All 133 test-supported classes without training instances are in the support: 117 occur only in test, of which the draft already admitted 94 and step 3 added 23. The paper omitted step 3; the revision describes it in Appendix B, and we will release the inventory with its rules.

Use across systems. Twelve of the 17 systems in Table 1 (QPIC, MUREN, GEN-VLKT, SOV-STG, SOV-STG-VLA, RLIPv2, and CoRISP) use it as their label space; all systems are scored on the same 1,104 test-supported classes.

Status and effect. Because step 2 checked the draft against the class labels of all splits, the inventory is not independent of test class labels; Appendix B documents this construction step. It is a dataset-level task definition shared by all inventory-based systems, like the interaction category lists of HICO-DET and V-COCO, and it holds no instance information; test-only triples have no training instances, so they are never positive targets in training. When predictions are restricted post hoc to the 2,855 triples seen in training or validation, Role mAP changes by at most 1.54 points for any system (RLIPv2 Swin-L), and the ranking barely changes (Kendall's τ = 0.96). Re-decoding CoRISP (run 1) with the restricted support changes its Set mAP by less than 0.5 points (Full / Multi / Repeat / Shared 18.56 / 8.97 / 6.94 / 22.91 → 18.70 / 8.56 / 6.93 / 23.23).

Q4. V-COCO baselines under their original evaluation conventions

Do the actual V-COCO baseline checkpoints recover their expected strength under their original evaluation conventions?

We ran PViC Swin-L with its official code and detector and reimplemented InCoM-Net (DETR-R50, CLIP-L/14; no official code or V-COCO weights are available): 30 epochs, seed 42, last epoch, original evaluator (vsrl_eval.py, 4,946 test images, IoU 0.5). Role AP, S1 / S2:

System25 classes incl. point24 classes (Table 1)Published
PViC Swin-L58.42 / 66.1060.85 / 68.7264.1 / 70.2
InCoM-Net67.63 / 71.1970.11 / 73.8073.6 / 75.4
CoRISP70.77 / 73.1873.72 / 76.23–

Papers differ in convention. With their released weights, QPIC, MUREN, and RLIPv2 reproduce their published values with point excluded (Table 1), whereas identical GEN-VLKT-S predictions reproduce its published values only with point; PViC and InCoM-Net do not state theirs. Our runs therefore fall 3.3 / 1.5 (PViC Swin-L) and 3.5 / 1.6 (InCoM-Net) points short if the published values exclude point, and 5.7 / 4.1 and 6.0 / 4.2 if they include it. We report these runs without further tuning; for InCoM-Net, a reimplementation without official code, implementation details may also differ. With point included, CoRISP (70.77 / 73.18) still leads both reproduced baselines. Table 1 evaluates every system under one convention, without point (31 positives, most without an object box), and does not rank literature-only rows.

AuthorsOpenReview ↗

Reply to Reviewer vYa7 3/4

Q5. Set-level results under a common readout

How much of the set-level advantage remains under a common readout of the learned relation marginals?

Common readout for CoRISP. We fixed the CoRISP checkpoint of the submitted Table 1 and changed only how its relation scores are read out, using the prefix construction that the paper applies to the pair-output baselines.

Set mAPFullSingleMultiRepeatShared
CoRISP, native set decoding18.0420.578.948.9923.80
CoRISP, prefix construction16.9819.786.896.9821.14
best baseline, prefix construction*22.2125.149.876.1219.98

* Full, Single, and Shared: RLIPv2 Swin-L; Multi: SOV-STG Swin-L; Repeat: PViC Swin-L.

From the same relation scores, the native set decoding adds 2.1 points on Multi, 2.0 on Repeat, and 2.7 on Shared; the gains are similar for run 1 of the three-seed models (1.5, 2.6, and 2.8). Most of CoRISP's event-level gain therefore comes from the normalized set model, the paper's central proposal, rather than from the relation scores alone.

CoRISP's decoder for the baselines. Conversely, we gave the pair-output baselines CoRISP's set readout as a post-hoc decoder: exact MAP over independent relation scores with K=8 winners, with thresholds, length caps and a cardinality prior tuned on HEIR validation (RLIPv2 Swin-L, without validation predictions, gets the parameter-free version). Prefix → MAP:

SystemFullMultiRepeatShared
PViC Swin-L14.57 → 15.265.11 → 5.616.12 → 6.7316.62 → 16.74
InCoM-Net14.94 → 15.434.58 → 4.935.18 → 5.6217.75 → 18.61
RLIPv2 Swin-L22.21 → 22.757.26 → 9.284.66 → 5.3519.98 → 20.42
CoRISP (3 seeds, native)18.298.507.0422.95

Post-hoc decoding adds at most 0.9 points on Full, Repeat and Shared and lifts RLIPv2's Multi to 9.28. CoRISP's Shared lead holds in every seed under every construction (22.58–23.35 vs at most 20.42), and its mean Repeat (7.04) stays above every baseline under either construction (at most 6.73): the advantage of a model trained with the set likelihood is not recovered by decoding independent scores post hoc.

Correction to Table 2. While preparing this answer, we found that the three ablation rows of the submitted Table 2 were read out with the prefix construction, while its CoRISP row used native decoding; the submitted Section 5.1, Table 2 caption, and Appendix C wrongly stated that the ablations use CoRISP's set prediction rule. We re-decoded every ablation natively against its paired full model (same seed and recipe) and added two seeds. The revised Table 2 reports the three-seed means, and Table 1 now reports the same three full models. The run-1 ablation checkpoints are those of the submitted rows; Role mAP also changes because the submitted rows took it from the training pipeline's evaluation over all candidate relations, whereas Table 1 scores the top 100 per image.

Mean of 3 seedsRoleFullSingleMultiRepeatShared
CoRISP20.0018.2920.848.507.0422.95
w/o G+0.22−0.31−0.32−0.67−1.17−0.70
w/o context messages+0.33+0.04+0.29−0.26−1.84−0.40
w/o role feedback+0.17+0.11+0.08−0.22−1.33+0.32

Removing context messages lowers Repeat in every seed, and removing G or role feedback lowers Multi in every seed. Full and Single, dominated by the 8,514 one-participant test events, change by at most 0.32 points. Removing more of the set structure lowers Repeat further: removing G and context messages together lowers Repeat from 6.94 to 4.92 (run 1), and the independent head of our reply to MnAS Q1(1), which has none of the set components, reaches 2.26.

Reviewer_7UgpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

A benchmark dataset (HEIR) is presented in the submission, where it focuses on event composition of human-entity interactions. This is different from standard human-object detection (HOI) that examines individual human-object pairs. The proposed benchmark's unit is a person-action event, with a set of localized participants, each assigned a functional role (target, instrument, destination, etc). The proposed benchmark aims to measure compositional complexity, and relation and complete-event AP are used.

Apart from the proposed benchmark, a normalized model of interaction sets (CoRISP) is proposed as well in the paper, where it predicts a normalized distribution over an event's participant-role assignments. The experimental results show that the proposed CoRISP achieves competitive performance given the small number of model parameters, especially on Repeat and Shared Set mAP.

The authors provide the detailed information regarding the source datasets used in the appendix, where 29 sources are used to construct the proposed benchmark dataset. The authors also provide additional information about the proposed CoRISP in the appendix.

Critical strengths

The strength of this submission is that the proposed benchmark can measure event composition in human-entity interactions. This provides complementary information compared to standard human-object interaction (HOI) detection, where HOI detection does not directly measure a model's prediction in terms of relation or complete event.

Critical weaknesses

The readability of the paper needs to be improved, it is not easy for a reader to go through and understand the paper based on the current version. The motivation of this paper is clear. However, it is challenging to follow the paper itself. It is mentioned in the AI Use Statement that generative AI was used to aid and polish the paper writing. Using AI tools is not an issue, but it is important to make sure the readability is good so others can easily get the main ideas of the paper.

A related issue to the above one is the presentation of figures. For example, Figure 2 is the overview of the proposed approach (CoRISP). However, the figure is not an overview but a detailed technical workflow, squeezing too much information into a figure does not always help, particularly for an overview figure.

The motivation of the proposed benchmark is to measure relation and event. However, based on the statistics of the proposed benchmark, it appears that nearly half of the images (all and test) have only one acting person per image. In this case, half of the dataset involves a single acting person per image, which may not be ideal to measure the proposed evaluation metrics such as relation and complete-event AP.

Questions for authors

Questions:

  • Discuss the reason of having nearly half of the datasets (all and test) have only one acting person per image and how it might affect the measurement of relation and complete-event AP.

  • Clarify the key findings in Sec. 5.3 and Figure 3. For example, what does Figure 3 (a) measure? After reading the caption and the discussion of Figure 3, it is still not clear.

Minor issues:

  • When referring to tables in the supplementary material, consider making it clear the tables are in the supplementary material not the main paper. Otherwise, it is potentially confusing when mentioning Table 6 or Table 5 in line 404 and 429.

Future work:

  • Consider adding a figure to show the concept introduced, otherwise, it is challenging for a reader to connect the concepts discussed in the papers to particular human-entity interactions. If the authors can show a few representative cases, and what the proposed metrics measure, that'd be helpful for the audience to understand the proposed benchmark.
5 comments
AuthorsOpenReview ↗

Reply to Reviewer 7Ugp 1/2

We thank the reviewer for the clear account of what was hard to follow. The uploaded revision addresses each point; we answer the two questions first.

Q1 / W3. Proportion of single-actor images

Discuss the reason of having nearly half of the datasets (all and test) have only one acting person per image and how it might affect the measurement of relation and complete-event AP.

Unit of evaluation. The unit of evaluation is the person–action event, not the image. An image with one actor can still contain several actions of that actor, events with several participants, two entities in the same role, and one entity in different roles across actions.

On the test set, 1,420 of 2,957 images (48.0%) have one annotated actor, but they hold only 2,352 of the 9,955 test events (23.6%). Of these events, 1,518 belong to an actor with two or more actions, 541 are Multi and 115 are Repeat events. The 1,420 images include the 407 test images drawn from SWiG, which were selected around one salient agent; without them, 39.7% of the test images have one actor. Appendix Table 5 cross-tabulates test events by actor count and stratum.

Set recovery in single-actor images. With one actor, a predicted set can contain correct relations and still be wrong: it may add an extra member, miss a participant, or give one participant a wrong role. Relation and set rankings can therefore diverge without a second actor.

Results by actor count (CoRISP: mean of three seeds; Set mAP unless noted).

ImagesSystemRole mAPFullMultiRepeatShared
single-actor (1,420)CoRISP26.3321.1311.9014.43–
RLIPv2 Swin-L34.0227.7110.1712.56–
PViC Swin-L26.3920.066.0410.43–
InCoM-Net18.8215.457.0711.49–
multi-actor (1,537)CoRISP25.0321.909.854.5722.95
RLIPv2 Swin-L25.9320.958.874.2019.98
PViC Swin-L23.0416.457.133.5216.62
InCoM-Net19.0418.586.525.2317.75

On multi-actor images, CoRISP has the highest Full, Multi, and Shared Set mAP of these four systems in every seed, so its advantage over them does not depend on single-actor images.

Source of the distribution. HEIR draws on 29 sources. Everyday object-interaction sources are mostly single-actor, and interpersonal sources are multi-actor. We did not rebalance the sources by actor count, so both regimes are represented; overall, 51.6% of the images contain more than one actor and 62.1% more than one action.

AuthorsOpenReview ↗

Reply to Reviewer 7Ugp 2/2

Q2. Measurements in Section 5.3 and submitted Figure 3(a)

Clarify the key findings in Sec. 5.3 and Figure 3. For example, what does Figure 3 (a) measure?

We agree that the caption and the paragraph gave neither the unit nor the computation; the revision rewrites both (Section 5.3 and Figure 4, the submitted Figure 3).

  • Measured quantity. For a test (action, noun) pair, HOI AP ignores the role label, whereas Role AP also requires the correct role. We average the Role APs of each pair's roles, so that every pair contributes one HOI and one role-qualified value ("pair-balanced"), and then average over pairs. Each row of panel (a) is one baseline; the x-axis is HOI AP minus pair-balanced Role AP, the AP lost when the role must also be correct.
  • Groups of pairs. Of the 1,057 test-supported pairs, 1,010 are annotated with one role (open circles) and 47 with several (filled circles), e.g., pour, cup (source or destination) and drive, car (instrument or target). The revised panel also adds SOV-STG R101, which the submitted panel omitted.
  • Finding. The gap is at most 0.34 points on single-role pairs but 4.78 to 12.66 points on multi-role pairs. Role difficulty is concentrated in action–noun contexts that admit several functions, which aggregate HOI–Role differences hide.

Panel (b) asks whether event recovery is merely a reweighting of relation accuracy. Role mAP averages over classes and Set mAP over actions, so we average Role AP within each action and then over the same 101 actions ("action-balanced") and plot it against Set mAP for the 16 baselines. Of the 120 baseline pairs, 41 reverse order between the two axes, as do 10 of the 55 pairs that share the class inventory; for example, SL-HOI exceeds PViC Swin-L by 7.49 points in action-balanced Role AP but trails it by 3.64 in Set mAP.

W1 / W2 / Future work. Readability and figures

  1. Concept figure (new Figure 1). It uses the example of the submitted Figure 1(a), a girl feeding a baby with a bottle on a sofa, and draws its six annotated events as cards grouped by actor, with a colored ring for each participant's role: the bottle takes three roles in three events, the sofa supports both people, and the girl is the target of the baby's look. Panel (c) scores feed against three predicted sets (correct; bottle missing; bottle as source instead of instrument): Role AP still credits the correct relations of the two wrong sets, whereas Set AP credits neither.
  2. Overview figure (Figure 3, submitted Figure 2). We redrew it at print size from our original figure and example, without equations. It shows two stages: (1) each candidate relation gathers role-pooled messages from its event, pair, and entity neighborhoods, and its role estimates are fed back; (2) for each event, a dynamic program over candidates and role counts, including the set-size and role-count potentials B and G, exactly normalizes the distribution over complete participant sets. Relation scores are role marginals and event scores are set probabilities.
  3. Text and notation. Section 4's opening paragraph now ends with a roadmap of its subsections, Appendix B adds a notation table, and terms such as the detector-weighted probabilities, role-wise pooling, the prefix set construction and edge coverage are defined where they are first used.
  4. Figures are drawn at print size with one color code for the six roles throughout; the statistics figure now shows test images and test events by actor count (Q1).

Minor. References to supplementary tables

When referring to tables in the supplementary material, consider making it clear the tables are in the supplementary material not the main paper.

Appendix tables are now cited as "Appendix Table N"; the references at lines 404 and 429 become Appendix Tables 8 and 7.

AuthorsOpenReview ↗

Kind Reminder to Review Our Rebuttal

Thank you again for your helpful review. As the discussion period closes on November 18, we would be grateful to know whether our replies and the revised PDF address your concerns. In brief, we show that the 48.0% of test images with one actor hold only 23.6% of the test events (Q1), rewrote Section 5.3 and the caption of the former Figure 3 (Q2), and added a concept figure and a redrawn overview figure for readability. If any point remains unclear, we would be glad to clarify it before the discussion closes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the three reviewers for their detailed and constructive reviews. All three agree that complete event composition is a meaningful evaluation target that single-relation metrics do not capture (7Ugp, vYa7, MnAS S1). The main concerns are protocol transparency, reproducibility of the detector adaptation, attribution of the gains across methods, and readability. In short, CoRISP's lead on shared-participant images holds with the same detector, with the same detector and visual features, with CLIP instead of DINOv3 features, and under every set construction we gave the baselines; on the events that it and the same-detector baseline both fully cover, an exact set more often appears within its budget.

Reported numbers. The revision reports CoRISP on HEIR as the mean of three seeds trained with the same recipe as its component ablations, so every ablation is compared with identically trained full models; single-model analyses use one of these seeds. In every seed, CoRISP leads all baselines of Table 1 on Shared (22.91 / 22.58 / 23.35 vs 19.98) and on Repeat (6.94 / 6.71 / 7.46 vs 6.12), by 2.97 and 0.92 points on average. The submission reported a single earlier run with the same hyperparameters and effective batch size, trained on 4 instead of 8 GPUs (Shared 23.80, Repeat 8.99); the one reply that analyzes this submitted checkpoint (vYa7 Q5) says so. The submission's single-model analyses (oracle, conditional classification, qualitative examples) already used run 1 (the first value of each triple above), so their numbers are unchanged; the hypothesis-count readout (Appendix Table 7) now also uses run 1. We also correct the component ablations: the ablated rows of the submitted Table 2 had been read out with the baselines' prefix construction instead of CoRISP's set prediction. Read out natively, the components act mainly on Repeat and Multi rather than Shared (vYa7 Q5, MnAS Q2(2)).

Summary of the replies.

  1. Evaluation scope: definition, per-split counts, and the seven locally scorable systems rescored with the scope rule disabled and with the affected images removed (vYa7 Q1).
  2. Controlled comparisons: same detector; same detector and features; another visual encoder (CLIP); a readout-only comparison (MnAS Q1(1), vYa7 Q5).
  3. Model size: trainable and total parameters of every system (MnAS Q1(2)).
  4. Sources of the event-level gains: an error decomposition on commonly covered events, per-role results with counts, and a split of the Shared events (MnAS Q2, vYa7 Q6).
  5. Reproducibility: the detector adaptation recipe, the construction of the compatibility inventory, and the V-COCO baselines under their original conventions (vYa7 Q2–Q4).
  6. Annotation quality: a stratified audit of 200 test events (vYa7 Q7).
  7. Statistics and readability: results by actor count and a rewritten Section 5.3 (7Ugp).

Revised PDF (additions and rewrites in blue; OpenReview's pdfdiff also shows deletions): a concept figure (Figure 1); a redrawn overview figure (Figure 3); the 16-baseline diagnostics with a rewritten Section 5.3 (Figure 4); a same-detector comparison on commonly covered events (Section 5.4); the evaluation scope (Appendix A); trainable and total parameters and three-seed results (Table 1); three-seed component ablations (Table 2); a CLIP-encoder control (Appendix C); the detector adaptation and inventory construction (Appendix B); the actor statistics (Appendix A). Submitted Figures 1–4 are Figures 2–5 in the revision; figure and table numbers in the replies refer to the revision unless marked as submitted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.