acceptodds
Under review as a conference paper at ICLR 2027

SG-MoE: Structured Geometric Mixture-of-Experts for Monocular Human-Object Interaction Reconstruction

Abstract

Monocular human–object interaction reconstruction requires recovering both individual surfaces and their coherent spatial configuration from a single image, which is challenging due to severe depth ambiguity and mutual occlusion. Existing methods typically rely on a fixed combination of interaction and geometric reasoning mechanisms throughout progressive reconstruction, despite the evolving demands of different decoding stages. We introduce SG-MoE, a structured geometric mixture-of-experts framework that adaptively composes complementary geometric reasoning operations during reconstruction. The proposed experts capture global human–object interaction, entity-specific topology, and evolving relative geometry, enabling the decoder to refine human and object representations according to the current reconstruction state. We further introduce a bidirectional geometric interaction mechanism that couples intermediate geometry with cross-entity feature reasoning. Experiments on BEHAVE and InterCap demonstrate consistent improvements in human and object reconstruction as well as contact recovery over state-of-the-art methods, while maintaining favorable model complexity and inference efficiency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
HOI reconstruction, Mixture-of-Expert,Adaptive Geometric Computation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cYv8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SG-MoE, a Structured Geometric Mixture-of-Experts framework for monocular human-object interaction (HOI) reconstruction. The core idea is to replace a fixed decoding structure with an adaptive composition of four geometric reasoning experts: Global Interaction (standard self-attention), Human Topology and Object Topology (graph convolutions adopted from HOI-TG), and Bidirectional Geometric Interaction (cross-attention with geometric bias). These experts are combined via learned dense weights predicted by a simple gating network from the pooled representation at each decoding block. The method is evaluated on BEHAVE and InterCap datasets, reporting improvements in Chamfer distance and contact metrics over existing baselines. In summary, this paper presents a competent engineering effort that combines existing ideas in a reasonable way and obtains positive results on standard benchmarks. However, it lacks the conceptual novelty, the community impact, and the scholarly rigor expected of an ICLR publication. The contribution is incremental, the assumptions are restrictive, and the presentation is compromised by AI over-polishing.

Critical strengths
  1. The paper correctly identifies that different stages of progressive HOI decoding require different types of geometric reasoning—coarse global configuration at early stages versus fine-grained local relation correction at later stages.

  2. Quantitative comparisons on BEHAVE and InterCap cover multiple baselines with both Chamfer distance and contact metrics. The evaluation is reasonably comprehensive.

  3. Ablations on expert composition strategies and individual expert contributions help understand the role of each component.

Critical weaknesses
  1. The core contribution is essentially wrapping several existing technical modules into an MoE framework with a simple gating network for weighted fusion. Every component individually is a direct reuse of existing techniques. The claimed "learning how complementary geometric operations should be composed" amounts to learning a single scalar weight per expert per block. This level of "adaptivity" is far from what the term "compositional reasoning" implies.
  2. Model complexity comparison is unfair: 96.3M parameters vs HOI-TG's 122.8M sounds favorable, but HOI-TG is an end-to-end framework while this paper reuses its graph operators and adds extra components. The parameter reduction likely comes from implementation differences rather than methodological advantage.
  3. The paper exhibits clear signs of LLM-assisted writing and The introduction and abstract spend excessive space motivating the problem while the core method description is relatively brief. This over-polishing raises doubts about whether the authors truly understand the limitations of their own method, and makes it difficult for reviewers to distinguish original thinking from LLM-generated filler.
  4. Writing Quality Issues: Multiple citations lack proper spacing, e.g., "(Lin et al.,2021b)". Figure 1(a) claims to illustrate that "applying the same reasoning pattern throughout decoding forces a fixed structure," but the figure does not clearly convey this contrast.
Questions for authors
  1. If the core selling point is "learned composition," why is there no visualization of expert weight distributions across stages and samples? This would directly validate the central hypothesis that different stages require different reasoning patterns.

  2. Eqs.(3)-(4) update human first, then object. What happens if the order is reversed? Is there any experiment justifying this asymmetric design?

  3. Evaluation is limited to two indoor datasets with a restricted set of object categories. Can the method generalize to in-the-wild images or unseen object categories?

5 comments
AuthorsOpenReview ↗

Response to concerns - W1 and W2

Thank you very much for your careful review. We wil response your questions as follows.

W1: Novelty of Structured Geometric MoE

Thank you for the comments regarding novelty. We acknowledge that GI employs standard self-attention, HT/OT build upon graph operators from HOI-TG, and the gating function follows established MoE formulations. Our contribution is not a new generic MoE routing algorithm, but a task-specific structured geometric MoE framework for monocular HOI reconstruction.

(1) Structured geometric experts rather than a generic MoE wrapper. Unlike conventional MoE formulations that primarily combine similar subnetworks, SG-MoE explicitly organizes heterogeneous operators with complementary geometric roles: global interaction (GI), human topology (HT), object topology (OT), and relative human–object geometry (BGI). These operators act on a shared reconstruction representation, and their residual updates are integrated through learned dense fusion. The objective is to combine different geometric inductive biases throughout progressive reconstruction, rather than simply increase model capacity through interchangeable experts. To the best of our knowledge, structured heterogeneous geometric expert composition remains largely unexplored in prior monocular HOI reconstruction methods.

(2) Geometry-conditioned interaction beyond existing operators. Importantly, BGI is not a direct reuse of an existing HOI refinement module. It constructs pairwise attention biases from intermediate 3D displacements and distances, which are recomputed across decoding blocks. Combined with sequential bidirectional updates, this directly couples evolving geometric estimates with cross-entity feature interaction.

(3) Experimental evidence. Our additional progressive ablations show that adding HT/OT to GI reduces object CD from 7.41 to 7.20 on BEHAVE and from 7.34 to 6.82 on InterCap. Incorporating BGI further reduces these errors to 6.25 and 5.93, respectively. More importantly, the new (B=0) ablation isolates the geometric attention bias while retaining bidirectional cross-attention. The full model reduces object CD by 6.69% on BEHAVE and 8.68% on InterCap compared with (B=0), demonstrating the independent benefit of geometry-conditioned interaction beyond standard cross-attention (details can be found in W2 of reviewer siHc)..

W2: Fairness of Model Complexity Comparison

Thank you for raising this concern. We agree that the difference in total parameter counts should not be attributed solely to the proposed SG-MoE decoder, as the two frameworks differ in their overall architectures.

A key architectural difference lies in the initialization stage. HOI-TG employs an initial reconstruction module that predicts human body/hand parameters and an initial object pose, which are used to construct geometry-aware queries for subsequent refinement. In contrast, SG-MoE only uses a pretrained ResNet-50 backbone for image feature extraction and initializes geometric queries directly from canonical human and object templates, without requiring an additional initial human–object reconstruction stage.

We further audited the parameter counts by component:

ComponentSG-MoEHOI-TG
Decoder / Reconstruction module71.76M56.20M
Other components24.57M66.62M
Total96.33M122.81M

As shown, our decoder contains more parameters than HOI-TG's reconstruction module. However, the overall model has fewer parameters, primarily reflecting differences in the surrounding architecture and initialization pipeline. We therefore do not claim that the proposed expert composition itself reduces parameter count.

We will clarify this architectural distinction, provide the component-level parameter breakdown, and revise the complexity discussion to distinguish system-level model size from decoder-level parameter efficiency.

AuthorsOpenReview ↗

Response to concerns - W3, W4 and Q1

W3: Writing Quality and Technical Clarity

Thank you for the feedback. We acknowledge that the Abstract and Introduction could be more concise and that the technical contributions should be presented more directly. As disclosed in our AI Use Statement, generative AI was used to assist with language editing and manuscript organization. We are responsible for the methodology, implementation, experimental design, and reported results.

We would like to further clarify that the proposed design is supported by concrete technical analyses rather than conceptual descriptions alone. In particular, we conducted additional experiments to better establish the contributions and limitations of SG-MoE:

  • Geometric interaction: By retaining bidirectional cross-attention while setting the geometric bias to zero, we isolated the effect of geometry conditioning. The full model achieves 6.69% and 8.68% lower object CD on BEHAVE and InterCap, respectively, compared with the (B=0) variants (details can be found in W2 of reviewer siHc).
  • Structured expert design: Progressive ablations (GI-only → GI+HT+OT → Full) reduce object CD from 7.41 to 7.20 to 6.25 on BEHAVE, and from 7.34 to 6.82 to 5.93 on InterCap, providing further evidence for the complementary roles of geometric experts. (details can be found in W2 of reviewer sQWY)

These experiments help distinguish the actual contributions of structured geometric reasoning from those of the lightweight gating mechanism, while also identifying limitations of the current formulation.

W4: Writing quality issues

Thanks for your comments.

  • Citation formatting. We carefully checked the submitted PDF, including the cited example "(Lin et al., 2021b)", and found that the spacing between the comma and year is present. Nevertheless, we will further proofread the citation formatting and improve the visual presentation of citation hyperlinks to ensure consistency and readability.

  • Clarity of Fig. 1(a). We acknowledge that the current illustration is overly schematic and does not sufficiently convey the intended distinction. The term "fixed reasoning" refers to the predefined organization of geometric operators, rather than fixed attention weights or non-adaptive feature interactions. Existing Transformer-based methods already employ input-adaptive attention.

We will revise Fig. 1(a) to make this distinction more explicit, replacing "fixed reasoning" with "predefined operator organization" and highlighting the learned block-wise fusion of geometric experts. The accompanying caption and Introduction will also be revised accordingly.

Q1: Visualization of Expert Composition

Thank you for this suggestion. Fig. 4 already visualizes the mean expert weights across the six decoding blocks, revealing distinct stage-wise preferences. However, we agree that the original visualization does not characterize variations across individual samples. We therefore additionally computed the mean and standard deviation of expert weights over the complete BEHAVE and InterCap test sets. For example, the HT weight increases from 0.326 ± 0.018 (Block 1) to 0.900 ± 0.004 (Block 5) on BEHAVE, and from 0.422 ± 0.010 to 0.951 ± 0.003 on InterCap. These results reveal pronounced block-wise differences but relatively limited sample-level variation.

We further replaced the routing weights with their block-wise test-set means at inference time. The resulting contact F1 is nearly unchanged: 65.810% vs. 65.876% on BEHAVE and 67.625% vs. 67.593% on InterCap (Full vs. Mean Router).

These observations indicate that the learned composition primarily exhibits block-specific expert preferences rather than substantial sample-dependent fluctuations. We will clarify this distinction in the revised manuscript.

AuthorsOpenReview ↗

Response to concerns - Q2 and Q3

Q2: Effect of Bidirectional Update Order

Thank you for these constructive comments. The motivation for sequential interaction is to allow the second cross-attention operation to leverage features already refined by the first interaction. Specifically, our default design first updates human features using object features, and then updates object features using the refined human representation.

To investigate the effect of update order, we additionally evaluated parallel and reverse-order variants on BEHAVE, using the submitted full model as the human-first reference.

VariantHuman CD↓Object CD↓Precision↑Recall↑F1↑
Human-first (Full)4.1436.2470.71730.60800.6581
Parallel4.1866.3150.69880.58520.6370
Object-first (Reverse)4.0806.2800.69660.60930.6501

These results support the motivation for sequential interaction, as both sequential update orders achieve better object CD and contact F1 than parallel interaction, demonstrating the benefit of within-block information propagation. Moreover, reversing the update order yields comparable performance (object CD: 6.247 vs. 6.280 cm; F1: 0.6581 vs. 0.6501), suggesting that the effectiveness of sequential BGI is relatively insensitive to the update order. The default human-first strategy achieves the best object CD and contact F1, while object-first achieves the lowest human CD and highest recall. We will include this comparison in the revised manuscript.

Q3: Generalization to In-the-Wild Images and Unseen Object Categories

Thank you for raising this important question. We evaluate SG-MoE on BEHAVE and InterCap following the established known-object-template setting to ensure fair comparisons with existing HOI reconstruction methods.

To further examine generalization beyond these benchmarks, we additionally conducted qualitative evaluations on in-the-wild images, as shown in the Appendix (Fig. 5). The results demonstrate that SG-MoE can produce plausible human--object reconstructions in selected unconstrained scenes, suggesting a certain degree of generalization beyond the training benchmarks.

Regarding unseen object categories, our framework requires a known object template, consistent with the task setting of prior methods. As already acknowledged in Appendix H (Limitations), these assumptions restrict applicability to unconstrained in-the-wild scenarios, particularly for arbitrary or previously unseen objects. We consider generalization to unseen categories an important direction for future work.

Reviewer_sQWYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SG-MoE, a structured geometric mixture-of-experts framework that adaptively combines global interaction, human/object topology, and relative geometry through MoE for 3D human and object reconstruction. Experiments demonstrate that accuracy achieves SOTA on 3D reconstruction and contact estimation.

Critical strengths
  1. Naively implementing MoE to HOI reconstruction would have lacked novelty and task-specific methodology. However, this paper specifically designs MoE that focuses on important yet difficult problem of adaptively capturing 1) global interaction, 2) human topology, 3) object topology, 4) relative geometry. This is very important step forward for HOI reconstruction and the performance demonstrates this.
  2. Consuming fewer parameter and inference time while allowing substantially better performance makes this methodology more promising.
  3. Figure 4 is interesting observation that human topology weight is increasing throughout blocks while initial block often focus on object topology with GI and BGI to be minorly weighted.
Critical weaknesses
  1. In Line 067, the paper claims that existing methods apply a predefined reasoning structure for HOI reconstruction. However, most of the existing methods including CONTHO, HOI-TG, CrossHOI utilizes Transformer which is technically not predefined reasoning structure but rather attention based adaptive reasoning structure. Please explain why the paper states in this way. Also, the main difference between existing methods and this paper is regular Transformer vs MoE. How can this be more better formulated in the introduction and motivation of this paper
Questions for authors
  1. Please state if code will be released.
  2. Please state and mention previous 3D HOI paper (e.g, CONTHO, HOI-TG, CrossHOI, Score-HOI) regarding which paper covered GI, HT, OT, BGI and in what method.
  3. What is intermediate coordinates/representations?
  4. What does "A key challenge is that the relative human–object geometry itself evolves during reconstruction." mean in Line 089? How can relative geometry evolve during reconstruction? The relative geometry is constant and deterministic as represented by ground-truth 3D geometry.
  5. Is there any reason that BGI performs sequential interaction modeling that updates human feature with object feature and then updates object feature with human feature? While there are some explanation in L270-L275, it is not fully convincing why this design is better than parallel design that models human and object feature simultaneously.
  6. How much does edge-length loss contribute to the final performance?
  7. What is surface supervision in L303?
  8. What is the benefit of not needed explicit contact supervision during training? All of the contact are gathered after 5cm distance threshold from 3D meshes afterall, which does not require annotation burden.
  9. While authors state that HOI-TG checkpoint and code is not publicly available in L364-L365, the code and checkpoint for BEHAVE dataset is publicly available from their official GitHub repository.
  10. The viewpoint of Figure 3 should be consistent. However, ScoreHOI's viewpoint seems different from those of HOI-TG and ScoreHOI in (a). Please revise this if possible.
  11. In the effect of expert composition, is it common to compare weight composition with uniform averaging, top-2 routing, static learned weights? Also, is the expert composition just contribution by MoE paper rather than contribution by this paper?
  12. It is difficult to know the performance contribution from experiments by w/o GI, w/o HT, w/o OT, w/o BGI. As GI is a simple Transformer with global interaction learning, HT&OT to be human or object specific learning, and BGI to be final interaction learning, it would be great to have ablation study with 1) SG-MoE (w/ only GI w/o HT, OT, BGI), SG-MoE (w/ GI, HT, OT w/o BGI), SG-MoE. This would show better picture of the contribution of each learning.
  13. Some related works are missing: InteractVLM (CVPR 2025, Dwivedi et al.), HOI-Gaussian (CVPR 2025, Wen et al.), TeHOR (CVPR 2026, Nam et al.)
6 comments
AuthorsOpenReview ↗

Response to concerns - W1 and Q1-Q3

Thank you very much for your constructive comments. We will address your concerns as follows.

W1: Clarification of the Motivation and Difference from Transformer-based Methods

Thank you for this insightful comment. We agree that the phrase "predefined reasoning structure" was not sufficiently precise. Existing methods such as CONTHO, HOI-TG, and CrossHOI employ Transformer-based architectures that dynamically adjust feature interactions through input-dependent attention. Our intention was not to suggest that their attention patterns are fixed, but rather that the types and organization of geometric reasoning operations are largely specified by their architectures.

The key distinction of SG-MoE is therefore not simply replacing a Transformer with MoE. In fact, SG-MoE retains a shared Transformer layer in each decoding block. Beyond this shared contextualization, we explicitly organize four complementary geometric operators, i.e., global interaction (GI), human topology (HT), object topology (OT), and bidirectional geometric interaction (BGI), as heterogeneous experts. Their residual updates are combined through learned block-wise dense fusion, rather than following a predefined sequence or fixed combination of geometric operations.

Thus, while conventional Transformer attention adaptively models feature dependencies, SG-MoE additionally learns how heterogeneous geometric operations are weighted and integrated during progressive reconstruction. We will revise the Introduction and Fig. 1(a) to clarify this distinction, replacing the ambiguous description of "predefined reasoning structures" with a more precise discussion of the organization and learned composition of geometric operators.

Q1: About the code.

We will release the code, checkpoints and training scripts after acceptance.

Q2: Relationship between our Geometric experts and prior HOI methods.

Thank you for this suggestion. We clarify the relationship between our four geometric experts and prior HOI reconstruction methods.

GI, HT, and OT build upon established geometric reasoning operations. Specifically, GI employs standard self-attention for global human–object interaction, while HT and OT adopt the graph-based topology operators introduced in HOI-TG (CVPR 2025). HOI-TG combines self-attention with graph residual blocks to capture global dependencies and mesh topology. Our contribution is not to introduce these individual operators, but to organize them as heterogeneous experts with learned dense composition.

Other representative methods incorporate related interaction mechanisms. CONTHO (CVPR 2024) employs contact-guided cross-attention using predicted human–object contact maps. CrossHOI (CVPR 2026) introduces bidirectional cross-view feature interaction and contact-guided refinement. ScoreHOI (ICCV 2025) incorporates learned interaction priors and physical constraints through score-guided diffusion and contact-driven refinement.

BGI differs from these approaches by directly encoding pairwise human–object 3D displacements and distances as attention biases, recomputed from intermediate geometry at each decoding block. It further performs sequential human–object feature updates without explicit contact prediction or supervision.

Therefore, the contributions of SG-MoE lie in (1) the learned dense composition of heterogeneous geometric operators and (2) the geometry-conditioned bidirectional interaction mechanism, rather than the novelty of each individual expert. We will expand the Related Work and Method sections to explicitly acknowledge the corresponding components in prior methods and clarify these distinctions.

Q3: The intermediate coordinates and representations.

Thank you for the question. In SG-MoE, intermediate representations refer to the progressively updated token features within the decoder, which encode visual information and evolving geometric estimates.

Specifically, we reserve the last three channels of each human and object token for intermediate 3D coordinates, representing 431 human vertices and 64 object points, respectively. At each decoding block, BGI extracts these coordinates from the shared Transformer output and computes pairwise relative displacements and distances to construct geometry-conditioned attention biases (Eq. 2).

After expert fusion, the coordinate channels are supervised using the corresponding ground-truth 3D coordinates (Eqs. 5–6). The resulting representation is propagated to subsequent blocks, allowing the geometric estimates and feature interactions to be progressively refined.

Importantly, these intermediate coordinates are model-predicted estimates rather than ground-truth inputs. We will clarify their definition, supervision, and role in BGI in Sec. 3.1 and Sec. 3.3.

AuthorsOpenReview ↗

Response to converns - Q4-Q7

Q4: Clarification of evolving relative geometry.

We agree that the ground-truth human–object geometry is fixed for a given image. Our intended meaning is that the model's intermediate estimates of relative human–object geometry are progressively updated during decoding, rather than the underlying 3D configuration itself changing.

Specifically, at decoder block (b), the estimated relative geometry is computed from the intermediate human and object coordinates, r^ij(b)=x^h,i(b)−x^o,j(b).\hat{\mathbf r}_{ij}^{(b)}=\hat{\mathbf x}_{h,i}^{(b)}-\hat{\mathbf x}_{o,j}^{(b)}. As these coordinate estimates are refined across successive blocks, their relative displacements and distances may also change. BGI therefore recomputes the geometry-conditioned attention biases at each block to reflect the current estimated configuration.

We will revise the statement in L089 to "A key challenge is that the estimated relative human–object geometry is progressively refined during reconstruction" to avoid implying that the ground-truth geometry itself evolves.

Q5: Sequential vs. Parallel Interaction

We thank you for this insightful question. The motivation for sequential interaction is to enable information propagation between the two cross-attention operations within each block. Specifically, BGI first updates human features using object features, and then updates object features using the refined human representation. In contrast, parallel interaction updates both entities using their pre-update features.

To investigate the effect of the update strategy, we compared sequential, parallel, and reverse-order variants on BEHAVE.

BGI VariantHuman CD↓Object CD↓Precision↑Recall↑F1↑
Sequential (H→O)4.1436.2470.71730.60800.6581
Parallel4.1866.3150.69880.58520.6370
Reverse (O→H)4.0806.2800.69660.60930.6501

These results support the motivation for sequential interaction, as both sequential update orders achieve better object CD and contact F1 than parallel interaction, demonstrating the benefit of within-block information propagation. Moreover, reversing the update order yields comparable performance (object CD: 6.247 vs. 6.280 cm; F1: 0.6581 vs. 0.6501), suggesting that the effectiveness of sequential BGI is relatively insensitive to the update order. The default human-first strategy achieves the best object CD and contact F1, while object-first achieves the lowest human CD and highest recall. We will include this comparison in the revised manuscript.

Q6: Ablation on edge-length loss

The edge-length loss regularizes the reconstructed human mesh by penalizing differences between predicted and ground-truth mesh edge lengths. To evaluate its contribution, we compared the full model with a variant trained without the edge-length loss.

VariantHuman CD↓Object CD↓Precision↑Recall↑F1↑
Full (w/ Edge Loss)4.1436.2470.71730.60800.6581
w/o Edge Loss4.1736.2330.69790.58550.6368

Compared with the variant without edge-length loss, the full model reduces human CD by 0.030 cm and improves contact precision, recall, and F1 by 1.94, 2.25, and 2.13 percentage points, respectively. Object CD is slightly higher (6.247 vs. 6.233 cm).

These results suggest that edge-length regularization is beneficial to human reconstruction and contact recovery in this comparison, although it does not consistently improve all metrics. We will include this analysis and clarified the role of edge-length regularization in the revised manuscript.

Q7: Clarification of surface supervision

Thank you for pointing out this ambiguity. The term "surface supervision" refers to the edge-length loss in Eq. 7, which regularizes the reconstructed human mesh by constraining its edge lengths. We do not employ any additional surface-specific supervision. We will replace "surface supervision" with "edge-length regularization" to make the description more precise.

AuthorsOpenReview ↗

Response to concerns - Q8-Q11

Q8: Benefit of Avoiding Explicit Contact Supervision

Thank you for this question. We agree that contact labels can be automatically derived from ground-truth 3D meshes using a distance threshold, and therefore do not introduce additional manual annotation costs. Our intention was not to claim reduced annotation burden. The main motivation is instead to avoid relying on explicit contact prediction and binary contact supervision during reconstruction. Unlike contact-guided methods that predict contact maps to guide feature interaction, BGI directly incorporates continuous pairwise 3D displacements and distances into cross-attention. This allows geometric interactions to be modeled without explicitly classifying contact regions or introducing an additional contact-prediction loss.

Importantly, we do not claim that removing contact supervision is inherently superior. Rather, our results demonstrate that accurate contact recovery can be achieved using geometric reconstruction supervision alone. We will clarify this distinction and avoid presenting the absence of contact supervision as an annotation-cost advantage.

Q9: Clarification of HOI-TG Code and Checkpoint Availability

Thank you for pointing this out. We apologize for the inaccurate statement. The official HOI-TG implementation and BEHAVE pretrained checkpoint are publicly available. We used the released BEHAVE checkpoint to generate the HOI-TG qualitative comparisons in Figs. 3 and 5.

Our intended statement was that the official InterCap checkpoint is not publicly available, which is why HOI-TG was not included in our InterCap qualitative comparisons. We will revise the corresponding sentences to distinguish the availability of code and checkpoints across datasets.

Q10: Clarification of the Viewpoint in Fig. 3(a)

Thank you for your careful review. We have rechecked the visualization in Fig. 3(a) and confirmed that all methods are rendered from the same viewpoint. The apparent viewpoint difference arises from ScoreHOI's inaccurate object rotation estimation in this particular case, rather than inconsistent rendering settings. We will clarify this in the figure caption to avoid confusion.

Q11: Expert Composition Strategies and Their Novelty

Thank you for this question. The composition variants in Table 2 are designed to validate the design choices of SG-MoE, rather than to claim novelty in the routing strategies themselves. These are representative alternatives for examining different aspects of expert composition. Specifically, uniform averaging examines whether learned expert weighting is beneficial; static learned weights evaluate the effect of input-conditioned fusion; and Top-2 routing provides a comparison with sparse expert selection.

As shown in Table 2, the full SG-MoE consistently achieves better overall performance than these variants on both datasets, supporting our choice of learned block-wise dense composition for heterogeneous geometric experts.

Regarding novelty, we do not claim that MoE routing or weighted expert fusion is new. Our contribution lies in organizing complementary geometric operations into heterogeneous experts for progressive HOI reconstruction, which has received limited attention in prior work. In particular, BGI explicitly incorporates evolving relative 3D geometry into bidirectional cross-entity attention, whose independent benefit is further demonstrated by the (B=0) ablation reported in our response to W2 of review siHc.

Thus, these experiments provide empirical justification for our composition design, while the methodological contribution lies in the task-specific structured geometric experts and their integration into progressive reconstruction, rather than a new generic MoE routing algorithm. We will clarify this distinction in the revised manuscript.

Reviewer_siHcOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces SG-MoE, a method for monocular human-object interaction reconstruction. Given an RGB image, the masks and a known object template it predicts the SMPL-H mesh vertices and the 6-DoF object pose. The method builds on the token layout and progressive decoder of CONTHO [1] / HOI-TG [2]. To this end, in each of six decoder blocks a shared Transformer layer is followed by four "experts": global self-attention (GI), the HOI-TG graph operators on the human (HT) and object (OT) meshes, and a new Bidirectional Geometric Interaction (BGI) expert, which biases human-to-object and object-to-human cross-attention with the pairwise distances of the current coordinate estimates (Eq. 2-4). These experts are routed with softmax weights predicted from the pooled block state. On BEHAVE [3] and InterCap [4], SG-MoE outperforms CONTHO, HOI-TG, ScoreHOI [5] and CrossHOI [6] in all metrics.

[1] Nam et al., Joint Reconstruction of 3D Human and Object via Contact-Based Refinement Transformer, CVPR 2024.

[2] Wang et al., End-to-End HOI Reconstruction Transformer with Graph-based Encoding, CVPR 2025.

[3] Bhatnagar et al., BEHAVE: Dataset and Method for Tracking Human Object Interactions, CVPR 2022.

[4] Huang et al., InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interaction, GCPR 2022.

[5] Li et al., ScoreHOI: Physically Plausible Reconstruction of Human-Object Interaction via Score-Guided Diffusion, ICCV 2025.

[6] Geng et al., CrossHOI: Learning Cross-View Representations for Monocular 3D Human-Object Interaction Reconstruction, CVPR 2026.

Critical strengths

S1. Consistent gains on both benchmarks, including contact metrics (with small caveats, see bellow).

S2. The ablations isolate the contribution of each expert and of the composition rule, with every Tab. 2 variant trained independently (L414-L416); one variant is missing (see W2). The model is also fast (0.068 s per sample, L375-L377).

Critical weaknesses

W1. The central claim, state-adaptive composition, is not shown. Static learned weights and the full model differ only in whether the weights depend on the input, so their gap in Tab. 2 is the only measure of adaptivity. Moreover, Fig. 4 contradicts Fig. 1(a) and the intro (L051-L052). Namely, the BGI module, which should matter more in the late stage, is almost zeroed-out in L3-L5 while HT dominates. Finally, Fig. 4 shows mean weights per block over the whole dataset, making it unclear whether the router actually varies per sample. Please report the per-sample spread of the router weights and the result of replacing each router by its mean weights at test time.

W2. BGI, the main novelty, is not properly evaluated. HT and OT are the HOI-TG operators (L228-L229) and GI is plain self-attention (L222), so the geometric bias of BGI is the one component claimed as new (L108-L110). BGI itself follows CONTHO's refinement transformer [1], which already cross-attends human and object vertex features gated by estimated contact. In this work, BGI replaces that hard contact mask with a soft distance bias from the intermediate coordinates. However, this is never isolated. The w/o BGI row removes the cross-attention passes together with the bias (L785-L788), so the gain attributed to BGI (L454-L456) may come from plain cross-entity attention. Please add a BGI variant with B = 0 in Eq. 3-4.

W3. Two statements about the baselines are wrong, and the Tab. 1 provenance is unstated. The Fig. 1(b) caption says SG-MoE "uses fewer parameters ... than the listed baselines" (L070-L071), but CONTHO is listed at 82.8M against 96.3M. L364-L365 states that code and checkpoints for HOI-TG are not publicly available; the official code with train/test scripts and a BEHAVE checkpoint is at https://github.com/ZhenrongWang/HOI-TG, and the paper itself shows HOI-TG reconstructions (Fig. 3, Fig. 5, video) and times it "using the same machine" (L373). Tab. 1 does not say which rows were re-run and which were copied. Please correct the caption, explain how the HOI-TG numbers and visualizations were produced, and give the provenance of each Tab. 1 row.

W4. "Object reconstruction" is pose estimation of a known template. The object tokens predict 64 template-aligned points from which rotation and translation are recovered (L156, L280-L282), and the object mesh is the posed canonical template (L746-L747). However, the abstract (L025), contribution 3 (L112), the headline (L099-L100) and the results text (L357, L443, L449-L450, L454) report gains in object reconstruction, and the baselines are said to recover object shapes (L366-L367), which none of them does.

Questions for authors
  1. (W1) What is the per-sample standard deviation of the router weights per block?
  2. (W1) An aditional cheap test whether per-sample routing is needed, would be be to replace each router by its dataset-mean weights at test time and compute same metrics.
  3. (W2) What are CD and contact F1 with BGI kept but B = 0 in Eq. 3-4?
  4. (W3) Were the HOI-TG parameter count, runtime and qualitative/quantitative results produced with the official code or a re-implementation? The official repository releases a BEHAVE checkpoint only, so which weights were used on InterCap?
  5. (W3) Which Tab. 1 rows were re-run and which copied?
  6. (W1) Is one model trained per dataset or one for both? If it's per dataset, how does the BEHAVE model perform on InterCap and vice versa? This would explain whether the learned composition is an interaction prior or benchmark-specific.
  7. (clarification) Are the test-time masks ground truth or predicted (L188)? If these are GT, please rerun with using some of the SotA mask estimators.
  8. (clarification) Eq. 7 lists three terms while B.1 adds a joint-coordinate loss (L716). What is the "final" training objective?
5 comments
AuthorsOpenReview ↗

Response to concerns

Thank you very much for taking the time to review and for your support. We try our best to address your questions.

W1: State-Adaptive Composition We thank the reviewer for the constructive comments. We address the three concerns as follows.

(1) Evidence for state-adaptive composition. Table 2 compares two end-to-end training formulations that share the same heterogeneous experts but differ in whether their mixture weights are input-conditioned. It demonstrates a performance difference between these trained configurations, but does not by itself characterize the sample-level behavior of the learned router. We therefore examined that behavior directly. SG-MoE uses reconstruction-state-conditioned, block-wise dense fusion, and the experiments below show that the converged router learns primarily block-specific preferences, rather than relying on large sample-to-sample weight fluctuations.

(2) Interpretation of Fig. 4 and BGI contributions. We respectfully clarify that the motivation in L51–52 emphasizes the increasing need for fine-grained geometric refinement, rather than requiring BGI to dominate later stages. Moreover, the routing coefficients do not directly represent the magnitude of expert updates. As defined in Eq. (1), our fusion is (Y=S+∑egeΔe),where(Δe=Ee(S)−S).(Y=S+\sum_e g_e\Delta_e), where (\Delta_e=E_e(S)-S). Thus, the actual update depends on both the routing coefficient and the expert residual. To better assess the contribution of BGI to the fused representation, we measured the weighted residual update magnitude for each expert, computed as

Ue=1N∑n=1N∣ge(n)(Ee(S(n))−S(n))∣F,U_e=\frac{1}{N}\sum_{n=1}^{N}|g_e^{(n)}(E_e(S^{(n)})-S^{(n)})|_F, where |.|_F denotes Frobenius norm and we adopt N=1000 test samples per dataset. On BEHAVE (Block 5), although BGI receives a routing weight of only 0.0515, its weighted update norm is 9.47, reaching 35.1% of HT's (27.01). Similarly, on InterCap (Block 4), BGI receives a weight of 0.0483, while its weighted update norm reaches 41.3% of HT's (13.07 vs. 31.65). These results indicate that small routing coefficients do not necessarily imply negligible updates to the fused representation. We note that these ratios measure update magnitudes, rather than direct contributions to reconstruction accuracy.

(3) Per-sample routing statistics and mean-router evaluation. (Q1 and Q2)

Following your suggestion, we computed the mean and standard deviation of expert weights over the complete test sets. The results show more pronounced differences across blocks than across samples. For example, the HT weight changes from 0.326 ± 0.018 (Block 1) to 0.900 ± 0.004 (Block 5) on BEHAVE, and from 0.422 ± 0.010 to 0.951 ± 0.003 on InterCap.

We further replaced each router with its test-set mean weights at inference time, without retraining:

DatasetSettingHuman CD↓Object CD↓F1↑
BEHAVEFull4.14326.24720.6581
Mean Router4.14366.24710.6588
InterCapFull4.27115.93330.6763
Mean Router4.26565.93300.6759

The nearly identical performance indicates that sample-dependent routing variations have limited influence on inference in the evaluated settings.

In summary, the additional analyses show that the learned expert composition exhibits pronounced block-specific preferences, while sample-level routing variations contribute little to inference performance. Meanwhile, the weighted-update analysis demonstrates that BGI cannot be considered negligible solely from its routing coefficients. We will clarify these distinctions in the revised manuscript and provide the complete routing statistics in the appendix.

AuthorsOpenReview ↗

Response to concerns - W2

W2: Contribution and novelty of BGI.

We thank you for the valuable suggestion. We agree that the original w/o BGI ablation evaluates the complete BGI module but does not isolate the contribution of its geometric attention bias. We address this concern with an additional controlled ablation and clarify the distinction from CONTHO.

(1) Isolating the contribution of geometric attention bias. (Q3)

Following your suggestion, we introduced a variant with (B=0) in Eqs. (3)–(4), while preserving both directions of cross-attention and the remaining architecture. The (B=0) variants were independently trained with the same scheme of SG-MoE. The results are summarized below.

DatasetVariantHuman CD↓Object CD↓CP↑CR↑F1↑
BEHAVEBGI (B=0)4.2546.6950.6940.5730.628
Full4.1436.2470.7170.6080.658
InterCapBGI (B=0)4.4216.4970.7310.6140.667
Full4.2715.9330.7380.6240.676

Compared with (B=0), the full model reduces object Chamfer distance by 6.69% on BEHAVE and 8.68% on InterCap, while improving contact F1 by 3.0 and 0.90 percentage points, respectively. Since both variants retain the same bidirectional cross-attention structure, these results provide direct evidence that the geometric attention bias offers additional benefits beyond standard cross-entity attention.

(2) Distinction from CONTHO.

Thank you for pointing this out. We would like to clarify the distinction from CONTHO. While CONTHO also employs cross-entity attention, BGI differs in its geometric representation, attention modulation, and progressive interaction mechanism.

First, CONTHO predicts vertex-level human and object contact maps and uses them to mask non-contacting vertex features before cross-attention. In contrast, BGI directly constructs a pairwise geometric bias B_{ij} from the relative 3D displacement and distance between each human–object point pair. Rather than selecting features according to predicted contact regions, BGI incorporates these continuous geometric relations directly into the attention logits, allowing cross-entity interactions to be conditioned on relative spatial configurations without explicitly masking non-contact regions.

Second, CONTHO uses estimated contact maps to guide the refinement of initially reconstructed meshes. BGI instead recomputes pairwise geometric biases from the evolving intermediate coordinates at each decoding block, coupling geometric estimation with feature interaction throughout progressive reconstruction. It further adopts a sequential bidirectional update, where the object representation is updated using the already refined human features.

Thus, BGI is not simply a soft replacement for contact-based masking. It introduces continuous pairwise geometry-conditioned interaction within progressive reconstruction. Together with the (B=0) ablation, these distinctions clarify the proposed mechanism and demonstrate the independent benefit of its geometric attention bias.

We will incorporate the additional ablation and clarify these architectural distinctions in the revised manuscript.

AuthorsOpenReview ↗

Response to concerns - W3 and W4

W3: Correction of baseline statements and result provenance.

A3: Thank you for identifying these inaccuracies and apologize for the unclear descriptions.

(1) Model complexity comparison.

We acknowledge that the statement in Fig. 1(b) is inaccurate. SG-MoE (96.3M) has more parameters than CONTHO (82.8M), but fewer than HOI-TG (122.8M) and ScoreHOI (451.7M). We will revise the caption to accurately reflect this comparison.

(2) HOI-TG code availability and qualitative comparisons. (Q4)

Thank you for your carefully review. You are correct that HOI-TG has released its official implementation and BEHAVE checkpoint. Our original statement was incorrect. We used the released BEHAVE checkpoint to reproduce HOI-TG results and generate its qualitative comparisons in Figs. 3 and 5. However, an official InterCap checkpoint is not provided, so we did not include HOI-TG in the InterCap qualitative comparisons. Similarly, although CrossHOI has a public GitHub repository, its implementation and pretrained checkpoints have not been released, preventing us from generating corresponding qualitative results. We will revise the description to distinguish these cases clearly.

(3) Provenance of quantitative results.

We acknowledge that Table 1 did not clearly distinguish reproduced results from those taken from prior publications. Our evaluation follows a consistent protocol: baselines with publicly available pretrained checkpoints were re-evaluated using their released models (CONTHO, HOI-TG on BEHAVE), while results without available checkpoints were taken from the corresponding publications. Specifically, HOI-TG was re-evaluated on BEHAVE using its official checkpoint, whereas its InterCap results and the CrossHOI results were taken from the published papers. We verified the dataset splits and evaluation definitions against the corresponding publications to ensure consistency. Besides, the HOI-TG parameter count and inference time were measured using its available implementation on the same hardware as our model.

We will explicitly distinguish re-evaluated results from those cited from the original publications, and correct the corresponding descriptions in Sec. 4.2.

W4: Clarification of object reconstruction

A4: Thank you for pointing out this ambiguity. Our work follows the known-object-template setting commonly adopted in monocular HOI reconstruction, including CONTHO, HOI-TG, and ScoreHOI. In this setting, the object geometry is provided, and the objective is to recover its 3D pose and spatial relationship with the human, rather than reconstruct an unknown object shape. Accordingly, we use the term "HOI reconstruction" following established literature.

Specifically, SG-MoE predicts 64 template-corresponding 3D points, from which the object rotation and translation are estimated to transform the canonical mesh. The reported object Chamfer distance evaluates the accuracy of this posed mesh against the ground truth after joint human–object alignment. Therefore, improvements in object CD reflect more accurate posed-object geometry and relative human--object configurations, rather than unknown-shape recovery or a direct reduction in absolute 6DoF pose error.

We agree that several object-specific descriptions could be more precise. We will explicitly clarify the known-template assumption and revise the corresponding statements in the abstract, contributions, and experimental analysis. We will revise the description of the baseline reconstructions accordingly.

Comments on the paper

2 comments
Area_Chair_qtuEOpenReview ↗

Please read the response from authors and discuss

Dear reviewer fellows,

The author rebuttal has been posted. The rebuttal period is a critical part of our review process, allowing authors to clarify misunderstandings and address specific concerns.

Please take a moment to read the author response, engage in the discussion by replying to the authors' responses, and update your review if the rebuttal changes your assessment.

Your active participation is vital to ensuring a fair and thorough evaluation of the submissions. A constructive discussion helps us make the most informed decisions.

Thank you for your time and valuable contributions to the review process.

Best regards,

Your AC

AuthorsOpenReview ↗

Updated Manuscript and Appreciation

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair for facilitating the discussion and encouraging constructive exchanges. We also greatly appreciate all reviewers for their careful evaluations, valuable feedback, and thoughtful discussions.

We have now uploaded the revised manuscript and appendix, incorporating the additional experiments, technical clarifications, and corrections discussed in our responses. All revisions are highlighted in red for easy reference.

We believe these revisions have strengthened the experimental analysis and improved the clarity of our presentation. We welcome any further questions or comments and look forward to continued discussion.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.