acceptodds
Withdrawn from ICLR 2027

ProgAlign: Progressive Semantic Alignment with Visual Chain-of-Thought Reasoning for Open-Vocabulary HOI Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Open-vocabulary human-object interaction (OV-HOI) detection is a fundamental task for human-centric scene understanding. Existing end-to-end HOI methods often suffer from insufficient fine-grained visual semantics mining and limited reasoning-process interpretability. To address these challenges, we propose ProgAlign, an end-to-end framework that integrates visual chain-of-thought reasoning with a progressive semantic alignment mechanism. Specifically, a visual chain-of-thought module is designed to construct a multi-step coarse-to-fine HOI evidence chain across spatial, contact, and interaction stages, enabling the model to progressively capture fine-grained interaction cues, which in turn helps improve the discrimination of visually similar interactions. Furthermore, a progressive semantic alignment mechanism is introduced to align Large Language Model-derived stage-specific semantic priors with intermediate visual representations, thereby helping alleviate black-box reasoning issues and enhancing the semantic consistency of the intermediate reasoning process. Extensive experiments on the SWIG-HOI and HICO-DET datasets demonstrate that ProgAlign achieves state-of-the-art performance among comparable open-vocabulary methods, validating its effectiveness in advancing OV-HOI understanding and distinguishing complex and fine-grained interactive relations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
open-vocabulary, human-object interaction detection, chain-of-thought, semantic alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WEJDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ProgAlign, an open-vocabulary human–object interaction (HOI) detection framework that combines progressive visual reasoning with semantic alignment. Its Image Chain-of-Thought (I-CoT) module sequentially models spatial configurations, physical contact, and interaction semantics, using the resulting representations to enhance HOI queries. Progressive Semantic Alignment (PSA) aligns intermediate visual features with stage-specific textual descriptions generated by an LLM, while an LLM-enhanced text branch provides discriminative category descriptions for classification. Experiments on HICO-DET and SWIG-HOI demonstrate improvements over comparable open-vocabulary HOI detection methods.

Critical strengths

S1. Consistent Empirical Improvements Across Benchmarks

ProgAlign achieves consistent improvements over the closely related SGC-Net on both HICO-DET and SWIG-HOI, including unseen interaction categories. Although the gains are relatively modest, the results provide evidence that the proposed framework is effective in open-vocabulary HOI detection. The component ablations further support the practical utility of combining multi-stage feature processing with semantic alignment, even though the contributions of the specific reasoning stages remain insufficiently isolated.

S2. Practical Integration of LLM-Based Semantic Guidance

The framework integrates stage-specific textual descriptions and discriminative category-level descriptions into a unified HOI detection pipeline. Since the LLM-generated descriptions are precomputed and cached, the method does not require LLM inference during detector training or deployment. This offers a practical way to incorporate richer semantic knowledge into open-vocabulary detection, regardless of whether the proposed stage decomposition constitutes a substantial methodological advance.

Critical weaknesses

W1. Insufficient Differentiation from Prior Work

Several elements of ProgAlign have direct precedents. CLIP4HOI [1] incorporates pairwise spatial information into HOI queries; BCOM [2] combines spatial and semantic features; and InCoM-Net [3] progressively aggregates intra-instance, inter-instance, and global contexts. SICHOI [4] integrates GPT-generated spatial, posture, and situational descriptions with multi-view visual features, while SGC-Net [5] combines intermediate visual feature alignment with LLM-generated comparative descriptions.

Query augmentation also has precedents: InterProDa [6] adds prompt-distribution guidance to learnable queries, whereas ProgAlign adds image-conditioned Q_init. Their information sources differ, but both augment learnable queries with additional guidance.

These similarities do not establish that the methods are identical. ProgAlign's specific distinction appears to be its sequential spatial-contact-interaction decomposition and stage-specific alignment. However, the additional methodological value of this combination beyond existing contextual aggregation and semantic guidance remains insufficiently established. The most closely related approaches should be discussed more explicitly.

W2. Limited Evidence for the Claimed Progressive Reasoning Mechanism

The Contact StageBlock processes the original visual tokens conditioned on pooled H_spa, and the Interaction StageBlock uses image-level summaries [H_spa, H_con, G] as memory. This is a reasonable sequential conditioning architecture, but its structure alone does not establish spatial reasoning, physical contact verification, and semantic interaction inference as distinct learned functions.

Table 3(b) reports successive Full mAP improvements from 11.45 to 15.91, 16.76, and 17.15. These results establish the usefulness of adding stages, not necessarily that their designated semantic functions explain the gains. Additional feature transformations and contextual conditioning remain alternative explanations.

Image-level pooling also leaves unclear how evidence for different human-object pairs is preserved in multi-interaction images. Since the final decoder retrieves evidence from the original image features, its dependence on preceding stage representations requires further characterization. Figure 4 provides qualitative support, but does not establish the claimed semantic specialization or the role of intermediate evidence in final predictions.

W3. Insufficient Isolation of Semantic Decomposition and Stage-Specific Alignment

The ablations appropriately retain the same category descriptions and frozen CLIP text encoder. However, adding stages also increases network depth and processing capacity. The experiments therefore do not isolate the specific spatial-contact-interaction decomposition from generic multi-stage processing.

Similarly, PSA is not compared with shared semantic targets or alternative stage-prior assignments. Its targets average textual embeddings of the image's annotated HOI categories, providing image-level semantic supervision rather than direct supervision of pair-specific spatial or contact evidence. Whether the designated stage-prior correspondence matters beyond generic auxiliary alignment remains unclear.

The Table 3 baseline also removes SGC-Net's GSA and HGC modules. Its large ablation gains should therefore be distinguished from the smaller improvements over complete SGC-Net: +0.41/+0.91 Full mAP and +0.31/+0.12 Unseen mAP on HICO-DET/SWIG-HOI, respectively. These positive gains would be easier to assess with run-to-run variability, which is not reported despite the stated averaging over four runs.

W4. Unclear Contact-Importance Computation and Ablation Settings

Appendix A.3 specifies visual tokens as queries and pooled H_spa as a single key-value token. Under standard scaled dot-product attention, omitting learned projections:

A_con = softmax(QK^T / sqrt(d))

With N queries and one key, the attention logits have shape N x 1. Softmax over the key dimension therefore gives A_con[i, 1] = 1 for every query, before attention dropout. Consequently, these normalized weights cannot directly produce spatially varying patch importance.

Yet Section 3.1 derives w_con from contact attention responses, and Figure 4 shows nonuniform Contact-stage heatmaps. Residual features, pre-softmax scores, or another operation could produce such variation; the StageBlock's token outputs need not be identical. However, the exact computation connecting the stated attention mechanism to w_con and the heatmaps is missing.

Two ablation details also require clarification: which representations receive PSA supervision in the PSA-only configuration, and why I-CoT-only achieves 17.15 Full mAP in Table 3(a), whereas lambda_icot = 0 achieves 17.62 in Table 8. These ambiguities limit reproducibility and interpretation of PSA's contribution.

References

[1] Mao et al. CLIP4HOI: Towards Adapting CLIP for Practical Zero-Shot HOI Detection. NeurIPS, 2023.

[2] Wang et al. Bilateral Adaptation for Human-Object Interaction Detection with Occlusion-Robustness. CVPR, 2024. (BCOM)

[3] Seo et al. Mining Instance-Centric Vision-Language Contexts for Human-Object Interaction Detection. CVPR, 2026. (InCoM-Net)

[4] Luo et al. Discovering Syntactic Interaction Clues for Human-Object Interaction Detection. CVPR, 2024. (SICHOI)

[5] Lin et al. SGC-Net: Stratified Granular Comparison Network for Open-Vocabulary HOI Detection. CVPR, 2025.

[6] Jia et al. Orchestrating the Symphony of Prompt Distribution Learning for Human-Object Interaction Detection. AAAI, 2025. (InterProDa)

Questions for authors

Q1. Additional Contribution over Prior Work

What substantive advantages distinguish ProgAlign's stage decomposition, stage-specific alignment, and image-conditioned query generation from the spatial/contextual processing, textual guidance, and query augmentation in [1–6]?

Please connect these differences to the supporting evidence and appropriately cite and discuss the relevant studies in Related Work, adding missing references and clarifying distinctions from already cited methods. Reimplementing all referenced methods is not required.

Q2. Stage Dependencies and Semantic Specialization

How does H_spa specifically improve contact evidence, and what distinct information do H_spa, H_con, and G contribute to interaction prediction? Selective removal or replacement of memory components could help establish these dependencies.

Beyond Figure 4, is there evidence connecting each stage's designated semantics to final predictions, particularly when multiple human-object pairs occur in one image? The goal is to distinguish semantic refinement from general contextual feature processing, while acknowledging Table 3(b)'s positive results.

Q3. Controlled Evaluation of I-CoT and PSA

(1) Could I-CoT be compared with a generic multi-stage architecture of comparable depth and parameter count, or a configuration retaining the StageBlocks but modifying their conditioning? Please keep the encoder, query count, category descriptions, and training protocol comparable.

(2) Does PSA outperform alignment using a shared semantic target or altered stage-prior assignments? Such comparisons would clarify whether the intended correspondence matters beyond auxiliary semantic supervision.

Please also report variability across the experimental runs to clarify the reliability of the gains.

Q4. Attention Computation and Experimental Consistency

(1) How are w_spa and w_con calculated? Given the single-key contact attention, please identify whether w_con uses post-softmax weights, pre-softmax scores, residual features, or another operation, including aggregation and normalization dimensions. Are Figure 4's Contact-stage heatmaps generated from the same w_con used for token reweighting?

(2) Which modules and intermediate representations are retained in the PSA-only configuration, and where are the three semantic priors applied?

(3) What explains 17.15 Full mAP for I-CoT-only in Table 3(a) versus 17.62 Full mAP for lambda_icot = 0 in Table 8? Please specify any architectural, training, evaluation, or aggregation differences.

These requests are possible ways to address the concerns, not a mandatory experimental checklist. I welcome alternative analyses, explanations of existing results, or other evidence clarifying the methodological contribution, stage-specific reasoning, and technical consistency, and would reassess my recommendation accordingly.

Reviewer_jaMAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The hypothesis of this submission is that existing open-vocabulary human-object interaction (OV-HOI) detection methods often focus on final or global semantics instead of local visual cues and intermediate reasoning. Thus, the proposed OV-HOI approach combines visual chain-of-thought (CoT) reasoning and progressive semantic alignment (PSA) to mitigate the issues.

The paper is easy to follow, the authors make it clear to understand their main hypothesis, technical contribution, and the results and discussions. I also appreciate the authors for the detailed information for the execution environment and the training/model configuration provided in the supplementary material.

The quantitative comparisons show that the proposed approach achieves comparable performance to existing methods. The results seem to not support the main hypothesis as if the hypothesis is true, incorporating the local visual cues and intermediate reasoning would result in significant performance improvement.

Critical strengths

I appreciate the author's efforts to make the paper easy to follow. As a reader, it is easy to understand the motivation, technical contributions, the comparisons and ablation studies. I'd like to emphasize this is not an easy task, a paper may propose a sophisticated approach, but at the same time, it might be difficult to go through and understand. This submission does a good job in paper writing.

Critical weaknesses

The main weakness of this submission is that it seems the main hypothesis is not supported by the experimental results. As discussed in the paper, the main hypothesis of this paper is that existing methods focusing on final/global semantic matching not local visual cues or intermediate reasoning, leading to their failures or lower performance. This clearly indicates that if one can address these issues, significant improvement is expected, where the proposed approach addresses these issues through visual chain-of-thought (CoT) reasoning and progressive semantic alignment (PSA). The quantitative comparisons (Tables 1 and 2) show that the proposed approach is comparable to the existing methods like SGC-Net, especially the unseen performance. The ablation studies provide additional information to not support the hypothesis. As explained at the beginning of Sec. 4.4 (Line 413-414) SGC-Net is used as the baseline by removing its modules, in this case, if local visual cues and intermediate reasoning are important, it is expected to observe significant performance improvement compared with using SGC-Net's own modules (GSA + HGC). However, it is not the case as shown in Tables 1 and 2.

Questions for authors

Questions:

  • Consider providing additional evidence to support the main hypothesis. Quantitative comparison is important but not the single factor. The relatively small improvement might be interpreted in different ways. 1) Local visual cues or intermediate reasoning may not be that important for single-image HOI detection. 2) The proposed approach does not effectively incorporate local visual cues or intermediate reasoning. 3) The performance (mAP) over a whole dataset may not be a good indicator to show the effectiveness of the proposed approach (then what metric to use?).

Suggestions (minor issues or future work):

  • L49-50, references are missing for the sentence of "recent methods further introduce vision–language alignment for open-vocabulary generalization."

  • This OV-HOI baseline is not included in the comparison [R1].

  • The submitted supplementary material (zip file) is the LaTeX file of the appendix, in this case, there is no need to submit it.

  • The authors may consider analyzing in which scenarios the proposed approach helps to address the issues in existing methods. For example, single vs. multi-person scenarios, or simple vs. complex interactions. An overall performance may not showcase the strength of the proposed approach (just a possibility), where HICO-DET and SWiG-HOI include a relatively high percentage of simple scenes [R2].

[R1] Learning to Diversify and Focus: A Reinforcement Framework for Open-Vocabulary HOI Detection, CVPR 2026

[R2] CrossHOI-Bench: A Unified Benchmark for HOI Evaluation across Vision-Language Models and HOI-Specific Methods, CVPR 2026

Reviewer_kcQwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ProgAlign, a framework for open-vocabulary human-object interaction (HOI) detection. The method combines visual chain-of-thought reasoning (I-CoT) with progressive semantic alignment (PSA). I-CoT organizes visual reasoning into spatial, contact, and interaction stages, while PSA aligns stage-specific semantic priors generated by a large language model with intermediate visual representations. Experiments on HICO-DET and SWIG-HOI show improvements over the compared open-vocabulary HOI detection methods.

Critical strengths

Open-vocabulary HOI detection requires models to recognize interactions beyond the categories observed during training. The paper focuses on aligning visual representations with semantic information at multiple reasoning stages, which is relevant to this challenge. I-CoT introduces staged visual reasoning, while PSA aligns the intermediate representations with corresponding semantic priors. This provides a coherent design rather than relying solely on final-layer semantic alignment.

Critical weaknesses
  1. The comparison with recent methods is not sufficiently comprehensive. The open-vocabulary comparisons in Table 1 and Table 2 include methods such as THID (2022), CMD-SE (2024), INP-CC (2025), and SGC-Net (2025). Although SGC-Net is a recent and strong baseline, the paper would benefit from a more comprehensive comparison with recent, directly comparable OV-HOI methods. This is particularly important for a submission to ICLR 2027.

  2. The improvements over the strongest reported baseline are modest on key metrics. On HICO-DET, ProgAlign improves Full mAP from 27.22 to 27.63 over SGC-Net, a gain of 0.41 percentage points. Unseen mAP increases from 23.27 to 23.58, a gain of 0.31 points. On SWIG-HOI, Full mAP improves from 17.20 to 18.11, but Unseen mAP increases only from 12.46 to 12.58, a gain of 0.12 points. These results support an improvement in overall performance, but provide more limited evidence for a substantial improvement in unseen-category generalization.

  3. The ablation study does not fully explain the interaction between I-CoT and PSA. Table 3 shows that the combination outperforms either component alone. However, the reported results do not establish why aligning stage-specific semantic priors with intermediate representations is more effective than applying semantic alignment at the final prediction stage. Additional comparisons with alternative alignment locations or alignment strategies would help isolate the contribution of progressive alignment.

  4. The role of LLM-generated semantic priors needs clearer validation. PSA relies on stage-specific semantic priors generated by an LLM. The paper should clarify how these descriptions are generated, whether they require manual intervention, and how sensitive the results are to the choice of LLM or the wording of the descriptions. Without such analysis, it is difficult to determine whether the gains arise from progressive alignment itself or from the additional semantic supervision.

  5. The improvements over SGC-Net are small on several metrics, especially unseen-category mAP on SWIG-HOI. The paper does not provide sufficient evidence in the reported comparisons to determine whether these gains are stable across repeated runs or sensitive to experimental settings. Repeated-run results or uncertainty estimates would strengthen the empirical claims.

Questions for authors
  1. The comparison with recent methods is not sufficiently comprehensive. The open-vocabulary comparisons in Table 1 and Table 2 include methods such as THID (2022), CMD-SE (2024), INP-CC (2025), and SGC-Net (2025). Although SGC-Net is a recent and strong baseline, the paper would benefit from a more comprehensive comparison with recent, directly comparable OV-HOI methods. This is particularly important for a submission to ICLR 2027.

  2. The improvements over the strongest reported baseline are modest on key metrics. On HICO-DET, ProgAlign improves Full mAP from 27.22 to 27.63 over SGC-Net, a gain of 0.41 percentage points. Unseen mAP increases from 23.27 to 23.58, a gain of 0.31 points. On SWIG-HOI, Full mAP improves from 17.20 to 18.11, but Unseen mAP increases only from 12.46 to 12.58, a gain of 0.12 points. These results support an improvement in overall performance, but provide more limited evidence for a substantial improvement in unseen-category generalization.

  3. The ablation study does not fully explain the interaction between I-CoT and PSA. Table 3 shows that the combination outperforms either component alone. However, the reported results do not establish why aligning stage-specific semantic priors with intermediate representations is more effective than applying semantic alignment at the final prediction stage. Additional comparisons with alternative alignment locations or alignment strategies would help isolate the contribution of progressive alignment.

  4. The role of LLM-generated semantic priors needs clearer validation. PSA relies on stage-specific semantic priors generated by an LLM. The paper should clarify how these descriptions are generated, whether they require manual intervention, and how sensitive the results are to the choice of LLM or the wording of the descriptions. Without such analysis, it is difficult to determine whether the gains arise from progressive alignment itself or from the additional semantic supervision.

  5. The improvements over SGC-Net are small on several metrics, especially unseen-category mAP on SWIG-HOI. The paper does not provide sufficient evidence in the reported comparisons to determine whether these gains are stable across repeated runs or sensitive to experimental settings. Repeated-run results or uncertainty estimates would strengthen the empirical claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.