acceptodds
Under review as a conference paper at ICLR 2027

LEXIS: Latent Proximal Interaction Signatures for 3D HOI from an Image

Abstract

Reconstructing 3D Human–Object Interaction from an RGB image is essential for perceptive systems and it requires capturing the subtle physical coupling between the body and objects. Current methods rely on sparse, binary contact cues. These miss the continuous proximity and dense spatial relationships inherent in interactions. We address this with InterFields, which encode dense, continuous proximity and the direction to the interacting geometry across the entire body and object surfaces. Inferring such fields from a single image is ill-posed. Our intuition is that interaction patterns are structured by the action and object geometry. We capture this structure in LEXIS, a novel discrete manifold of interaction signatures learned via a quantized autoencoder. We then develop LEXIS-Flow, a flow-matching framework that uses LEXIS manifold to estimate human and object meshes alongside their InterFields. These InterFields guide sampling that improves physical plausibility without post-hoc optimization. On Open3DHOI and BEHAVE, LEXIS-Flow outperforms SotA baselines in reconstruction, contact and physical plausibility. It generalizes better and the reconstructions are perceived as more realistic, moving us closer to holistic 3D scene understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
human-object interaction, 3D reconstruction, interaction representation, contact representation, latent manifold, flow matching, guided sampling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MeCiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LEXIS-Flow for reconstructing 3D human–object interactions from a single RGB image and an off-the-shelf object shape. It learns discrete interaction signatures through a quantized autoencoder, representing interactions with dense surface-to-surface distances and directions rather than binary contact alone. A flow-matching model jointly estimates the human state, object pose, and interaction code, while field-consistency and object-mask guidance refine the reconstruction during sampling. The same framework can also refine existing human and object estimates. Experiments on Open3DHOI and BEHAVE report improvements over the evaluated baselines in reconstruction accuracy and interaction quality.

Critical strengths

The use of dense distance-and-direction fields is well motivated. These fields describe spatial relationships beyond the contacting regions and provide a way to check agreement between the predicted interaction and the current human–object geometry during sampling. Table 1 supports the usefulness of this representation in both fitting and generative reconstruction. This is the most interesting aspect of the paper: it addresses a limitation of binary contact rather than simply changing the reconstruction backbone.

The reported reconstruction results are strong, particularly for refinement on Open3DHOI. The text-conditioned model achieves an object Chamfer distance of 20.06 cm, compared with 30.11 cm for InteractVLM++, and a contact F1 of 0.467, compared with 0.411 for TeHOR. The main refinement competitors receive the same initialization, which makes these comparisons more informative than comparisons between systems with different starting estimates. These results provide evidence that the proposed refinement is useful in practice.

Critical weaknesses

My main concern is what the LEXIS code actually represents. Section 3.1 and Eq. (1) describe an encoder that receives body articulation and the object’s canonical shape, but no explicit human–object relative rotation or translation. The decoder is nevertheless supervised to reconstruct the interaction fields of the particular training example. Holding the body pose and canonical object shape fixed while moving the object relative to the body changes those fields without changing the stated encoder inputs. Expressing the fields in canonical coordinates does not remove this dependence on relative placement. Unless relative pose enters through an undocumented transformation of the features or query points, the model appears to predict a typical interaction conditioned on pose and shape, rather than encode the observed interaction. Such a prior could still be useful, but the distinction matters for the paper’s central representation claim and for interpreting its guidance mechanism.

The experiments do not sufficiently isolate the contributions of direction prediction and the discrete latent representation. In Table 5, adding directions to distance-only guidance increases stability from 76.3% to 77.5%. The much larger improvement from 71.2% to 88.5%, attributed to introducing directions in the accompanying discussion, instead compares no guidance with the full combination of distance, direction, and mask guidance. The missing distance-plus-mask variant is therefore important: without it, the contribution of directions in the final system remains unclear. Similarly, Table 4 compares field reconstruction errors across latent models but does not establish whether FSQ improves final HOI reconstruction over a matched continuous autoencoder. The results support the full system, but do not yet establish which parts of the proposed representation account for its gains.

Questions for authors

Where exactly does human–object relative pose enter LEXIS-Net? Please specify the coordinate transformations used for the object features, field query points, and supervision targets. Can two configurations with identical body articulation and canonical object shape but different relative object placements produce different encoded signatures?

What happens when mask guidance is combined with distance-only InterField guidance, without directions? Also, does FSQ improve downstream reconstruction over a continuous autoencoder with comparable capacity and the same training data? These comparisons would help establish whether the claimed representation choices are responsible for the final gains.

Please reconcile the text-conditioning results in Table 3 and Table S.1. Table 3 reports an object CD reduction of 1.11 cm from adding text, whereas Table S.1 implies a 0.31 cm difference between the null-token input and the default Claude Opus description. The appendix states that the no-text rows use the same model with a null token. Were different subsets, sampling settings, or evaluation protocols used?

Section S.2.5 applies a similarity transform to the human but only the corresponding rigid transform to the object, then states that all metrics are computed on the aligned surfaces. Are Contact, Collision, and Stability evaluated before or after this alignment? Applying different scale transformations can change the relative human–object geometry, so this detail could affect the interpretation of the interaction scores.

Reviewer_svxgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles single-image 3D human-object interaction (HOI) reconstruction. It replaces sparse, binary contract with InterFields and it gives each body and object point the distance and unit direction to the nearest point on the other surface. It learns LEXIS, a discrete FSQ latent space that decodes to body pose and InterFields. It then trains LEXIS-Flow with a flow-matching model to jointly sample body, object pose and LEXIS tokens with masked-guided refinement. LEXIS-Flow shows performance gains over SotA baselines on Open3DHOI and BEHAVE in Chamfer distance, Contact F1, collision and its proposed stability metric.

Critical strengths
  1. The paper is well-motivated. InterFields extending contact with dense and continuous proximity and directions cues is a natural and helpful representation.
  2. The proposed method shows large, controlled improvement on in-the-wild 3D HOI. On Open3D HOIT, all refinement baselines start from the same initialization and the comparison shows that the paper's method reaches the best performance.
  3. Fast, plug-in refinement through in-sampler guidance. This mechanism brings run-time improvement inside the flow sampling loop.
Critical weaknesses
  1. The central claim is not supported by clean attribution. The paper argues that dense InterFields and the LEXIS prior drive the gains. However, the only module ablation (Table 5) is on BEHAVE and covers only the guidance terms. There, InterField guidance alone (8.20 cm CDobj, 77.5% stability) does not beat HOI-TG (7.98 cm, 77.8%; Table 3). The win appears only once mask guidance is added. LEXIS itself is never ablated on downstream metrics. Table 4 measures only the autoencoder design and the FSQ advantage is expected from FSQ's known training stability. Can the authors provide a decomposed performance gain on Open3DHOI results from each module?

  2. Limited representational novelty. ARCTIC (Fan et al. 2023) already estimated InterFields from a single image so the three claimed differences (lines 129-133) are only partly justified. From the results it seems that the paper's main addition (the direction term) gives only small gains.

Questions for authors

Please see the questions in the weakness sections.

(Clarification questions)

  1. It seems that the training set ProciGen is generated from BEHAVE and InterCap training set interactions (Xie et al., 2024), so the claims in Figure S.2 that the LEXIS-Net 'never saw BEHAVE' is inaccurate.
  2. Does the stability gain in Table 3 (88.5 vs. 77.8) still hold with a stricter contact threshold (smaller than 5cm)?
1 comment

Thank the authors for the detailed responses, which has addressed majority of my concerns. I have adjusted the rating.

Reviewer_77BiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the task of single-image 3D human-object interaction reconstruction, where the object shape is provided by an off-the-shelf model. The method extends binary contact cues into bidirectional interaction fields (InterFields) that describe the distance and direction from surface points to the interacting object. The proposed LEXIS-Net employs a finite scalar quantization autoencoder to learn a compact latent representation and decodes it into body pose and interaction fields. LEXIS-Flow then jointly estimates the human state, object pose, and latent interaction representation via image-conditioned flow matching. During sampling, the consistency between the decoded interaction fields and the current geometry, combined with object mask constraints, provides gradient-based guidance without requiring a separate post-hoc optimization stage. Experiments on the Open3DHOI and BEHAVE datasets show improvements in reconstruction accuracy, contact quality, and physical plausibility metrics. The evaluation also includes ablations on the representation and guidance, an initialization sensitivity test, and a perceptual study.

Critical strengths

Well-motivated interaction representation. Dense distance-and-direction fields can capture spatial relationships beyond binary contact and guide reconstruction during sampling. This provides a meaningful way to incorporate interaction geometry into the reconstruction process.

Notable empirical improvements. The paper reports consistent performance gains on the Open3DHOI and BEHAVE datasets. In particular, the shared-initialization comparison on Open3DHOI helps disentangle optimization gains from initialization quality, where the object Chamfer distance improves from 30.11 cm to 20.06 cm relative to the strongest corresponding baseline. These results support the practical value of the method.

Critical weaknesses

The contribution of the direction component is insufficiently isolated. Table 5 shows the additional gain from direction information, but a variant that excludes direction and uses only distance plus a mask is missing. This makes it difficult to assess the specific extent to which the direction component contributes to the performance improvement of the full system.

Some details of the latent sampling process need clarification. In particular, it is currently unclear whether the flow matching operates on pre-quantization or post-quantization latents, and when quantization is applied during sampling. A clearer description or sampling pseudocode for this core methodological component would improve reproducibility and strengthen credibility.

Questions for authors

Does LEXIS-Flow operate on pre-quantization or quantized latents? Please clarify when bounding and rounding are applied during sampling, and how intermediate states are passed to the frozen decoder.

How much does directional guidance contribute when mask guidance is also enabled? A comparison between distance-plus-mask guidance and the full distance-plus-direction-plus-mask guidance, under the same setting as Table 5, would help assess the direction component’s contribution.

1 comment
Reviewer_77BiOpenReview ↗

Thank you for the response

Thank you for the response and additional experiments. I will maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.