- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ProgAlign, an open-vocabulary human–object interaction (HOI) detection framework that combines progressive visual reasoning with semantic alignment. Its Image Chain-of-Thought (I-CoT) module sequentially models spatial configurations, physical contact, and interaction semantics, using the resulting representations to enhance HOI queries. Progressive Semantic Alignment (PSA) aligns intermediate visual features with stage-specific textual descriptions generated by an LLM, while an LLM-enhanced text branch provides discriminative category descriptions for classification. Experiments on HICO-DET and SWIG-HOI demonstrate improvements over comparable open-vocabulary HOI detection methods.
Critical strengths
S1. Consistent Empirical Improvements Across Benchmarks
ProgAlign achieves consistent improvements over the closely related SGC-Net on both HICO-DET and SWIG-HOI, including unseen interaction categories. Although the gains are relatively modest, the results provide evidence that the proposed framework is effective in open-vocabulary HOI detection. The component ablations further support the practical utility of combining multi-stage feature processing with semantic alignment, even though the contributions of the specific reasoning stages remain insufficiently isolated.
S2. Practical Integration of LLM-Based Semantic Guidance
The framework integrates stage-specific textual descriptions and discriminative category-level descriptions into a unified HOI detection pipeline. Since the LLM-generated descriptions are precomputed and cached, the method does not require LLM inference during detector training or deployment. This offers a practical way to incorporate richer semantic knowledge into open-vocabulary detection, regardless of whether the proposed stage decomposition constitutes a substantial methodological advance.
Critical weaknesses
W1. Insufficient Differentiation from Prior Work
Several elements of ProgAlign have direct precedents. CLIP4HOI [1] incorporates pairwise spatial information into HOI queries; BCOM [2] combines spatial and semantic features; and InCoM-Net [3] progressively aggregates intra-instance, inter-instance, and global contexts. SICHOI [4] integrates GPT-generated spatial, posture, and situational descriptions with multi-view visual features, while SGC-Net [5] combines intermediate visual feature alignment with LLM-generated comparative descriptions.
Query augmentation also has precedents: InterProDa [6] adds prompt-distribution guidance to learnable queries, whereas ProgAlign adds image-conditioned Q_init. Their information sources differ, but both augment learnable queries with additional guidance.
These similarities do not establish that the methods are identical. ProgAlign's specific distinction appears to be its sequential spatial-contact-interaction decomposition and stage-specific alignment. However, the additional methodological value of this combination beyond existing contextual aggregation and semantic guidance remains insufficiently established. The most closely related approaches should be discussed more explicitly.
W2. Limited Evidence for the Claimed Progressive Reasoning Mechanism
The Contact StageBlock processes the original visual tokens conditioned on pooled H_spa, and the Interaction StageBlock uses image-level summaries [H_spa, H_con, G] as memory. This is a reasonable sequential conditioning architecture, but its structure alone does not establish spatial reasoning, physical contact verification, and semantic interaction inference as distinct learned functions.
Table 3(b) reports successive Full mAP improvements from 11.45 to 15.91, 16.76, and 17.15. These results establish the usefulness of adding stages, not necessarily that their designated semantic functions explain the gains. Additional feature transformations and contextual conditioning remain alternative explanations.
Image-level pooling also leaves unclear how evidence for different human-object pairs is preserved in multi-interaction images. Since the final decoder retrieves evidence from the original image features, its dependence on preceding stage representations requires further characterization. Figure 4 provides qualitative support, but does not establish the claimed semantic specialization or the role of intermediate evidence in final predictions.
W3. Insufficient Isolation of Semantic Decomposition and Stage-Specific Alignment
The ablations appropriately retain the same category descriptions and frozen CLIP text encoder. However, adding stages also increases network depth and processing capacity. The experiments therefore do not isolate the specific spatial-contact-interaction decomposition from generic multi-stage processing.
Similarly, PSA is not compared with shared semantic targets or alternative stage-prior assignments. Its targets average textual embeddings of the image's annotated HOI categories, providing image-level semantic supervision rather than direct supervision of pair-specific spatial or contact evidence. Whether the designated stage-prior correspondence matters beyond generic auxiliary alignment remains unclear.
The Table 3 baseline also removes SGC-Net's GSA and HGC modules. Its large ablation gains should therefore be distinguished from the smaller improvements over complete SGC-Net: +0.41/+0.91 Full mAP and +0.31/+0.12 Unseen mAP on HICO-DET/SWIG-HOI, respectively. These positive gains would be easier to assess with run-to-run variability, which is not reported despite the stated averaging over four runs.
W4. Unclear Contact-Importance Computation and Ablation Settings
Appendix A.3 specifies visual tokens as queries and pooled H_spa as a single key-value token. Under standard scaled dot-product attention, omitting learned projections:
A_con = softmax(QK^T / sqrt(d))
With N queries and one key, the attention logits have shape N x 1. Softmax over the key dimension therefore gives A_con[i, 1] = 1 for every query, before attention dropout. Consequently, these normalized weights cannot directly produce spatially varying patch importance.
Yet Section 3.1 derives w_con from contact attention responses, and Figure 4 shows nonuniform Contact-stage heatmaps. Residual features, pre-softmax scores, or another operation could produce such variation; the StageBlock's token outputs need not be identical. However, the exact computation connecting the stated attention mechanism to w_con and the heatmaps is missing.
Two ablation details also require clarification: which representations receive PSA supervision in the PSA-only configuration, and why I-CoT-only achieves 17.15 Full mAP in Table 3(a), whereas lambda_icot = 0 achieves 17.62 in Table 8. These ambiguities limit reproducibility and interpretation of PSA's contribution.
References
[1] Mao et al. CLIP4HOI: Towards Adapting CLIP for Practical Zero-Shot HOI Detection. NeurIPS, 2023.
[2] Wang et al. Bilateral Adaptation for Human-Object Interaction Detection with Occlusion-Robustness. CVPR, 2024. (BCOM)
[3] Seo et al. Mining Instance-Centric Vision-Language Contexts for Human-Object Interaction Detection. CVPR, 2026. (InCoM-Net)
[4] Luo et al. Discovering Syntactic Interaction Clues for Human-Object Interaction Detection. CVPR, 2024. (SICHOI)
[5] Lin et al. SGC-Net: Stratified Granular Comparison Network for Open-Vocabulary HOI Detection. CVPR, 2025.
[6] Jia et al. Orchestrating the Symphony of Prompt Distribution Learning for Human-Object Interaction Detection. AAAI, 2025. (InterProDa)
Questions for authors
Q1. Additional Contribution over Prior Work
What substantive advantages distinguish ProgAlign's stage decomposition, stage-specific alignment, and image-conditioned query generation from the spatial/contextual processing, textual guidance, and query augmentation in [1–6]?
Please connect these differences to the supporting evidence and appropriately cite and discuss the relevant studies in Related Work, adding missing references and clarifying distinctions from already cited methods. Reimplementing all referenced methods is not required.
Q2. Stage Dependencies and Semantic Specialization
How does H_spa specifically improve contact evidence, and what distinct information do H_spa, H_con, and G contribute to interaction prediction? Selective removal or replacement of memory components could help establish these dependencies.
Beyond Figure 4, is there evidence connecting each stage's designated semantics to final predictions, particularly when multiple human-object pairs occur in one image? The goal is to distinguish semantic refinement from general contextual feature processing, while acknowledging Table 3(b)'s positive results.
Q3. Controlled Evaluation of I-CoT and PSA
(1) Could I-CoT be compared with a generic multi-stage architecture of comparable depth and parameter count, or a configuration retaining the StageBlocks but modifying their conditioning? Please keep the encoder, query count, category descriptions, and training protocol comparable.
(2) Does PSA outperform alignment using a shared semantic target or altered stage-prior assignments? Such comparisons would clarify whether the intended correspondence matters beyond auxiliary semantic supervision.
Please also report variability across the experimental runs to clarify the reliability of the gains.
Q4. Attention Computation and Experimental Consistency
(1) How are w_spa and w_con calculated? Given the single-key contact attention, please identify whether w_con uses post-softmax weights, pre-softmax scores, residual features, or another operation, including aggregation and normalization dimensions. Are Figure 4's Contact-stage heatmaps generated from the same w_con used for token reweighting?
(2) Which modules and intermediate representations are retained in the PSA-only configuration, and where are the three semantic priors applied?
(3) What explains 17.15 Full mAP for I-CoT-only in Table 3(a) versus 17.62 Full mAP for lambda_icot = 0 in Table 8? Please specify any architectural, training, evaluation, or aggregation differences.
These requests are possible ways to address the concerns, not a mandatory experimental checklist. I welcome alternative analyses, explanations of existing results, or other evidence clarifying the methodological contribution, stage-specific reasoning, and technical consistency, and would reassess my recommendation accordingly.