- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SG-MoE, a Structured Geometric Mixture-of-Experts framework for monocular human-object interaction (HOI) reconstruction. The core idea is to replace a fixed decoding structure with an adaptive composition of four geometric reasoning experts: Global Interaction (standard self-attention), Human Topology and Object Topology (graph convolutions adopted from HOI-TG), and Bidirectional Geometric Interaction (cross-attention with geometric bias). These experts are combined via learned dense weights predicted by a simple gating network from the pooled representation at each decoding block. The method is evaluated on BEHAVE and InterCap datasets, reporting improvements in Chamfer distance and contact metrics over existing baselines. In summary, this paper presents a competent engineering effort that combines existing ideas in a reasonable way and obtains positive results on standard benchmarks. However, it lacks the conceptual novelty, the community impact, and the scholarly rigor expected of an ICLR publication. The contribution is incremental, the assumptions are restrictive, and the presentation is compromised by AI over-polishing.
Critical strengths
-
The paper correctly identifies that different stages of progressive HOI decoding require different types of geometric reasoning—coarse global configuration at early stages versus fine-grained local relation correction at later stages.
-
Quantitative comparisons on BEHAVE and InterCap cover multiple baselines with both Chamfer distance and contact metrics. The evaluation is reasonably comprehensive.
-
Ablations on expert composition strategies and individual expert contributions help understand the role of each component.
Critical weaknesses
- The core contribution is essentially wrapping several existing technical modules into an MoE framework with a simple gating network for weighted fusion. Every component individually is a direct reuse of existing techniques. The claimed "learning how complementary geometric operations should be composed" amounts to learning a single scalar weight per expert per block. This level of "adaptivity" is far from what the term "compositional reasoning" implies.
- Model complexity comparison is unfair: 96.3M parameters vs HOI-TG's 122.8M sounds favorable, but HOI-TG is an end-to-end framework while this paper reuses its graph operators and adds extra components. The parameter reduction likely comes from implementation differences rather than methodological advantage.
- The paper exhibits clear signs of LLM-assisted writing and The introduction and abstract spend excessive space motivating the problem while the core method description is relatively brief. This over-polishing raises doubts about whether the authors truly understand the limitations of their own method, and makes it difficult for reviewers to distinguish original thinking from LLM-generated filler.
- Writing Quality Issues: Multiple citations lack proper spacing, e.g., "(Lin et al.,2021b)". Figure 1(a) claims to illustrate that "applying the same reasoning pattern throughout decoding forces a fixed structure," but the figure does not clearly convey this contrast.
Questions for authors
-
If the core selling point is "learned composition," why is there no visualization of expert weight distributions across stages and samples? This would directly validate the central hypothesis that different stages require different reasoning patterns.
-
Eqs.(3)-(4) update human first, then object. What happens if the order is reversed? Is there any experiment justifying this asymmetric design?
-
Evaluation is limited to two indoor datasets with a restricted set of object categories. Can the method generalize to in-the-wild images or unseen object categories?
5 comments
Response to concerns - W3, W4 and Q1
W3: Writing Quality and Technical Clarity
Thank you for the feedback. We acknowledge that the Abstract and Introduction could be more concise and that the technical contributions should be presented more directly. As disclosed in our AI Use Statement, generative AI was used to assist with language editing and manuscript organization. We are responsible for the methodology, implementation, experimental design, and reported results.
We would like to further clarify that the proposed design is supported by concrete technical analyses rather than conceptual descriptions alone. In particular, we conducted additional experiments to better establish the contributions and limitations of SG-MoE:
- Geometric interaction: By retaining bidirectional cross-attention while setting the geometric bias to zero, we isolated the effect of geometry conditioning. The full model achieves 6.69% and 8.68% lower object CD on BEHAVE and InterCap, respectively, compared with the (B=0) variants (details can be found in W2 of reviewer siHc).
- Structured expert design: Progressive ablations (GI-only → GI+HT+OT → Full) reduce object CD from 7.41 to 7.20 to 6.25 on BEHAVE, and from 7.34 to 6.82 to 5.93 on InterCap, providing further evidence for the complementary roles of geometric experts. (details can be found in W2 of reviewer sQWY)
These experiments help distinguish the actual contributions of structured geometric reasoning from those of the lightweight gating mechanism, while also identifying limitations of the current formulation.
W4: Writing quality issues
Thanks for your comments.
-
Citation formatting. We carefully checked the submitted PDF, including the cited example "(Lin et al., 2021b)", and found that the spacing between the comma and year is present. Nevertheless, we will further proofread the citation formatting and improve the visual presentation of citation hyperlinks to ensure consistency and readability.
-
Clarity of Fig. 1(a). We acknowledge that the current illustration is overly schematic and does not sufficiently convey the intended distinction. The term "fixed reasoning" refers to the predefined organization of geometric operators, rather than fixed attention weights or non-adaptive feature interactions. Existing Transformer-based methods already employ input-adaptive attention.
We will revise Fig. 1(a) to make this distinction more explicit, replacing "fixed reasoning" with "predefined operator organization" and highlighting the learned block-wise fusion of geometric experts. The accompanying caption and Introduction will also be revised accordingly.
Q1: Visualization of Expert Composition
Thank you for this suggestion. Fig. 4 already visualizes the mean expert weights across the six decoding blocks, revealing distinct stage-wise preferences. However, we agree that the original visualization does not characterize variations across individual samples. We therefore additionally computed the mean and standard deviation of expert weights over the complete BEHAVE and InterCap test sets. For example, the HT weight increases from 0.326 ± 0.018 (Block 1) to 0.900 ± 0.004 (Block 5) on BEHAVE, and from 0.422 ± 0.010 to 0.951 ± 0.003 on InterCap. These results reveal pronounced block-wise differences but relatively limited sample-level variation.
We further replaced the routing weights with their block-wise test-set means at inference time. The resulting contact F1 is nearly unchanged: 65.810% vs. 65.876% on BEHAVE and 67.625% vs. 67.593% on InterCap (Full vs. Mean Router).
These observations indicate that the learned composition primarily exhibits block-specific expert preferences rather than substantial sample-dependent fluctuations. We will clarify this distinction in the revised manuscript.
Response to concerns - Q2 and Q3
Q2: Effect of Bidirectional Update Order
Thank you for these constructive comments. The motivation for sequential interaction is to allow the second cross-attention operation to leverage features already refined by the first interaction. Specifically, our default design first updates human features using object features, and then updates object features using the refined human representation.
To investigate the effect of update order, we additionally evaluated parallel and reverse-order variants on BEHAVE, using the submitted full model as the human-first reference.
| Variant | Human CD↓ | Object CD↓ | Precision↑ | Recall↑ | F1↑ |
|---|---|---|---|---|---|
| Human-first (Full) | 4.143 | 6.247 | 0.7173 | 0.6080 | 0.6581 |
| Parallel | 4.186 | 6.315 | 0.6988 | 0.5852 | 0.6370 |
| Object-first (Reverse) | 4.080 | 6.280 | 0.6966 | 0.6093 | 0.6501 |
These results support the motivation for sequential interaction, as both sequential update orders achieve better object CD and contact F1 than parallel interaction, demonstrating the benefit of within-block information propagation. Moreover, reversing the update order yields comparable performance (object CD: 6.247 vs. 6.280 cm; F1: 0.6581 vs. 0.6501), suggesting that the effectiveness of sequential BGI is relatively insensitive to the update order. The default human-first strategy achieves the best object CD and contact F1, while object-first achieves the lowest human CD and highest recall. We will include this comparison in the revised manuscript.
Q3: Generalization to In-the-Wild Images and Unseen Object Categories
Thank you for raising this important question. We evaluate SG-MoE on BEHAVE and InterCap following the established known-object-template setting to ensure fair comparisons with existing HOI reconstruction methods.
To further examine generalization beyond these benchmarks, we additionally conducted qualitative evaluations on in-the-wild images, as shown in the Appendix (Fig. 5). The results demonstrate that SG-MoE can produce plausible human--object reconstructions in selected unconstrained scenes, suggesting a certain degree of generalization beyond the training benchmarks.
Regarding unseen object categories, our framework requires a known object template, consistent with the task setting of prior methods. As already acknowledged in Appendix H (Limitations), these assumptions restrict applicability to unconstrained in-the-wild scenarios, particularly for arbitrary or previously unseen objects. We consider generalization to unseen categories an important direction for future work.
Response to concerns - W1 and W2
Thank you very much for your careful review. We wil response your questions as follows.
W1: Novelty of Structured Geometric MoE
Thank you for the comments regarding novelty. We acknowledge that GI employs standard self-attention, HT/OT build upon graph operators from HOI-TG, and the gating function follows established MoE formulations. Our contribution is not a new generic MoE routing algorithm, but a task-specific structured geometric MoE framework for monocular HOI reconstruction.
(1) Structured geometric experts rather than a generic MoE wrapper. Unlike conventional MoE formulations that primarily combine similar subnetworks, SG-MoE explicitly organizes heterogeneous operators with complementary geometric roles: global interaction (GI), human topology (HT), object topology (OT), and relative human–object geometry (BGI). These operators act on a shared reconstruction representation, and their residual updates are integrated through learned dense fusion. The objective is to combine different geometric inductive biases throughout progressive reconstruction, rather than simply increase model capacity through interchangeable experts. To the best of our knowledge, structured heterogeneous geometric expert composition remains largely unexplored in prior monocular HOI reconstruction methods.
(2) Geometry-conditioned interaction beyond existing operators. Importantly, BGI is not a direct reuse of an existing HOI refinement module. It constructs pairwise attention biases from intermediate 3D displacements and distances, which are recomputed across decoding blocks. Combined with sequential bidirectional updates, this directly couples evolving geometric estimates with cross-entity feature interaction.
(3) Experimental evidence. Our additional progressive ablations show that adding HT/OT to GI reduces object CD from 7.41 to 7.20 on BEHAVE and from 7.34 to 6.82 on InterCap. Incorporating BGI further reduces these errors to 6.25 and 5.93, respectively. More importantly, the new (B=0) ablation isolates the geometric attention bias while retaining bidirectional cross-attention. The full model reduces object CD by 6.69% on BEHAVE and 8.68% on InterCap compared with (B=0), demonstrating the independent benefit of geometry-conditioned interaction beyond standard cross-attention (details can be found in W2 of reviewer siHc)..
W2: Fairness of Model Complexity Comparison
Thank you for raising this concern. We agree that the difference in total parameter counts should not be attributed solely to the proposed SG-MoE decoder, as the two frameworks differ in their overall architectures.
A key architectural difference lies in the initialization stage. HOI-TG employs an initial reconstruction module that predicts human body/hand parameters and an initial object pose, which are used to construct geometry-aware queries for subsequent refinement. In contrast, SG-MoE only uses a pretrained ResNet-50 backbone for image feature extraction and initializes geometric queries directly from canonical human and object templates, without requiring an additional initial human–object reconstruction stage.
We further audited the parameter counts by component:
As shown, our decoder contains more parameters than HOI-TG's reconstruction module. However, the overall model has fewer parameters, primarily reflecting differences in the surrounding architecture and initialization pipeline. We therefore do not claim that the proposed expert composition itself reduces parameter count.
We will clarify this architectural distinction, provide the component-level parameter breakdown, and revise the complexity discussion to distinguish system-level model size from decoder-level parameter efficiency.