acceptodds
Withdrawn from ICLR 2027

Compositional Evaluation of Human-Object Interaction Understanding with Real and Synthetic Multi-Tag Scenes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Human–object interaction (HOI) understanding requires models to distinguish multiple people, objects, and interactions that may coexist within the same scene. However, existing HOI evaluation commonly reports aggregate performance or analyzes isolated interaction configurations, providing limited insight into model behavior under compositional scenarios involving multiple sources of ambiguity. We first analyze multi-tag images from HICO-DET and evaluate HOI detection models on combinations of interaction configurations. Our analysis reveals that real multi-tag images are limited and highly imbalanced across combinations, making systematic compositional evaluation difficult. To address this limitation, we construct an image-conditioned synthetic evaluation set by editing real HICO-DET images to introduce controlled combinations of human–object interactions while preserving the original scene context, lighting, and visual complexity. Because the intended interaction configurations and HOI classes are specified during generation, the corresponding labels are obtained automatically and verified through human quality control. Using the real and synthetic subsets, we evaluate both specialized HOI detection methods and vision-language foundation models under a unified compositional framework. Our study provides a systematic analysis of how multiple interaction factors jointly affect model behavior and offers a scalable approach for evaluating compositional HOI understanding beyond the limited coverage of existing benchmarks.

Primary area:
datasets and benchmarks
Keywords:
Human-object interaction detection, compositional evaluation, vision-language models, synthetic benchmarks, image synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UHDEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses an overlooked problem in Human-Object Interaction evaluation that real-world multi-person scenes often contain multiple interaction patterns simultaneously, but existing benchmarks provide uneven coverage and little systematic diagnosis of model behavior in such settings. The authors introduce HICO-DET-SYN, a controlled synthetic benchmark built from real HICO-DET images through a multi-agent image-editing pipeline, which provides systematic coverage of multi-tag combinations while preserving scene context, camera viewpoint, and existing interactions. They evaluate representative specialized HOI detectors and general-purpose VLMs. The key finding is that compositional difficulty is not monotonic with tag cardinality: performance drops from single-tag to two-tag compositions but partially recovers for three- and four-tag combinations, with substantial variation across specific tag combinations.

Critical strengths

The paper identifies a genuine gap in HOI evaluation, the lack of systematic analysis of multi-tag scenes where multiple interaction patterns coexist. This is a practically relevant and under-explored problem.

HICO-DET-SYN is constructed through a carefully designed multi-agent pipeline that preserves original scene context, camera viewpoint, and existing interactions while adding specified multi-tag configurations. The use of controlled image editing on real HICO-DET images helps maintain visual realism and annotation reliability.

The V-shaped trend in compositional difficulty, that performance dropping from single-tag to two-tag but recovering for three- and four-tag, is a novel and interesting empirical finding that challenges the naive assumption that more tags always mean harder. The observation that specific two-tag combinations can be more challenging than higher-order compositions is particularly noteworthy.

The paper evaluates a diverse set of models, including specialized HOI detectors and general-purpose LVLMs, across multiple metrics (Instance F1, Macro F1, Exact Match), providing a rich picture of model behavior.

Critical weaknesses

While the paper takes care to preserve scene context and validate visual realism, synthetic images may still differ from real images in subtle ways. The mAP parity check is helpful but limited to shared HOIs.

Class frequencies and image characteristics may still differ across tag combinations, so the observed performance differences reflect associations rather than causal effects. This weakens the strength of the conclusions about compositional difficulty.

The paper offers a possible explanation for the V-shaped trend, but this is not empirically validated. A more systematic analysis of why certain combinations are easier or harder (e.g., through error analysis, attention visualization, or controlled manipulations) would strengthen the findings.

The current dataset has relatively few samples for some higher-order combinations, which may limit statistical reliability.

The paper focuses on zero-shot evaluation of VLMs and specialized HOI detectors. It does not explore whether fine-tuning on HICO-DET-SYN or incorporating synthetic data into training could improve model performance.

Images with the same tags can differ in human/object scale, spatial arrangement, occlusion, and background clutter, but does not systematically control for or analyze these factors. Detailed visibility annotations and matched-scene comparisons would help distinguish interaction structure from visual complexity.

Questions for authors

The mAP parity check is limited to shared HOIs and does not fully characterize subtle differences between synthetic and real images. Could you provide finer-grained comparisons (per-tag/per-combination gaps, human plausibility ratings, error-type distributions)?

Shared-class matching does not ensure matched frequencies or visual characteristics, so observed differences reflect associations rather than isolated causal effects of adding a tag. What additional controls (e.g., matched-scene comparisons, frequency-matched subsets) would be needed for causal claims?

Could you provide error-type analysis, attention visualization, or controlled manipulations to systematically explain why certain combinations are easier or harder?

Images with the same tags can differ in scale, spatial arrangement, occlusion, and background clutter, but these factors are not systematically controlled. Could the observed differences be driven by visual complexity rather than interaction structure?

How well do the findings generalize to other HOI benchmarks (e.g., V-COCO, SWiG-HOI) or video-based HOI?

Reviewer_4dZVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies HOI understanding when multiple interaction patterns coexist in a multi-person scene. Building on a taxonomy defined by action agreement and shared versus distinct object instances, it introduces HICO-DET-SYN through targeted editing of real HICO-DET images, automated validation, and human review. Specialized detectors and VLMs are evaluated using shared HOI vocabularies and several metrics. The main finding is that performance depends on the particular composition and does not decrease monotonically with tag count.

Critical strengths
  1. The diagnostic question adds information beyond aggregate accuracy. As Fig. 1c illustrates, models with similar overall mAP can differ in assigning interactions to the correct people and object instances. The composition-level results expose such differences and identify configurations, notably B+D and C+D, that challenge most evaluated models. The diagnostic angle is valuable in its own right, even without a new detector.
  2. Targeted synthesis provides coverage that natural data lacks. The pipeline targets sparse configurations and checks scene preservation, interaction plausibility, and tag satisfaction before independent human review. The disclosed prompt templates, failure cases, and multi-metric results make the construction reasonably inspectable.
Critical weaknesses
  1. The difficulty analysis does not separate composition from distribution or domain. The shared 19- and 17-class vocabularies fix class identities but not class frequencies, instance counts, object scale, occlusion, or background. The drop from single-tag to two-tag scenes also compares real images with synthetic ones, although the paper's own controlled comparison in Fig. 3b shows real–synthetic gaps of about ten mAP points, in opposite directions, on Tags B and C. The synthesized single-tag images exist but are not evaluated under the main protocol. The results therefore show non-monotonic performance on the selected subsets, but only weakly support the central claim that compositional structure is a key factor. The authors acknowledge this limitation; evaluating the synthesized single-tag images under the main protocol and reweighting class frequencies, with uncertainty estimates, would substantially strengthen the interpretation.
  2. Key benchmark validation details are inconsistent or undefined. The text reports 3,308 retained synthetic images, whereas Table A.1 reports 3,019. Within that table, the synthetic rows sum to 3,020, the real rows to 2,067 rather than 2,075, and the totals to 5,087 rather than 5,094; Tags E and F are never defined. The perceptual study reports 47.96% accuracy for one annotator and 54.59% on average, implying about 61% for the other, and calls this near chance. With four response options, uniform guessing yields 25% when the four cases are balanced; 50% applies only to per-image binary scoring. Without the scoring unit and response distribution, the study does not support the realism claim.
Questions for authors
  1. Which final image list was evaluated, and how do the three counts reconcile? Please specify the source-image split and duplicate handling.
  2. Is perceptual accuracy measured per four-way response or per image? Please provide its definition, response distribution, and corresponding chance baseline.
  3. Within the existing corpus, do the main trends persist after matching or reweighting class frequencies and available scene characteristics? An analysis separating the real-to-synthetic comparison from comparisons among synthetic compositions would be especially helpful.
  4. In Table A.11, GPT-5.6-Sol and Gemini-3.8-Flash reach 41.08 and 35.74 Macro F1, above every specialized model, and in Table 2 HOICLIP trails all three VLMs in overall Instance F1. How does this square with the claim that specialized models generally outperform VLMs, and with the statement in Appendix G that the three metrics show broadly consistent trends?
Reviewer_JEb9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies fine-grained evaluation of human-object interaction understanding in multi-person scenes. It introduces a synthetic benchmark to provide better coverage of different interaction compositions, and evaluates several HOI models and vision-language models under different settings.

Critical strengths
  1. The paper is generally well-written, and the motivation is clear.

  2. The reviewer appreciates the effort in constructing and validating the benchmark.

Critical weaknesses
  1. The major concern is the format inconsistency. Specifically, the reviewer finds critical mismatch between the openreview title and the pdf one. While the reviewer tries to review the paper based on its pdf information, this leaves the reviewer with confusion.

  2. Another key concern is that it is still not very clear to the reviewer how much of the observed performance difference really comes from the interaction composition itself. The paper tries to control the evaluated HOIs, but the compared subsets may still differ in other aspects. This makes the main trend somewhat difficult to interpret.

  3. Another concern is the use of synthetic data for the main multi-tag evaluation. The authors include several validation steps and human verification, which is appreciated, but it is still unclear how well the observations would transfer to naturally occurring multi-person scenes.

  4. The evaluation is also based on a relatively restricted set of shared HOIs so that different tag combinations can be compared. This is reasonable, but it is not fully clear whether the same observations would hold over a broader range of interactions.

Questions for authors

The reviewer appreciates the authors to clarify and answer towards the four critical weaknesses as elaborate above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.