- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents LEXIS-Flow for reconstructing 3D human–object interactions from a single RGB image and an off-the-shelf object shape. It learns discrete interaction signatures through a quantized autoencoder, representing interactions with dense surface-to-surface distances and directions rather than binary contact alone. A flow-matching model jointly estimates the human state, object pose, and interaction code, while field-consistency and object-mask guidance refine the reconstruction during sampling. The same framework can also refine existing human and object estimates. Experiments on Open3DHOI and BEHAVE report improvements over the evaluated baselines in reconstruction accuracy and interaction quality.
Critical strengths
The use of dense distance-and-direction fields is well motivated. These fields describe spatial relationships beyond the contacting regions and provide a way to check agreement between the predicted interaction and the current human–object geometry during sampling. Table 1 supports the usefulness of this representation in both fitting and generative reconstruction. This is the most interesting aspect of the paper: it addresses a limitation of binary contact rather than simply changing the reconstruction backbone.
The reported reconstruction results are strong, particularly for refinement on Open3DHOI. The text-conditioned model achieves an object Chamfer distance of 20.06 cm, compared with 30.11 cm for InteractVLM++, and a contact F1 of 0.467, compared with 0.411 for TeHOR. The main refinement competitors receive the same initialization, which makes these comparisons more informative than comparisons between systems with different starting estimates. These results provide evidence that the proposed refinement is useful in practice.
Critical weaknesses
My main concern is what the LEXIS code actually represents. Section 3.1 and Eq. (1) describe an encoder that receives body articulation and the object’s canonical shape, but no explicit human–object relative rotation or translation. The decoder is nevertheless supervised to reconstruct the interaction fields of the particular training example. Holding the body pose and canonical object shape fixed while moving the object relative to the body changes those fields without changing the stated encoder inputs. Expressing the fields in canonical coordinates does not remove this dependence on relative placement. Unless relative pose enters through an undocumented transformation of the features or query points, the model appears to predict a typical interaction conditioned on pose and shape, rather than encode the observed interaction. Such a prior could still be useful, but the distinction matters for the paper’s central representation claim and for interpreting its guidance mechanism.
The experiments do not sufficiently isolate the contributions of direction prediction and the discrete latent representation. In Table 5, adding directions to distance-only guidance increases stability from 76.3% to 77.5%. The much larger improvement from 71.2% to 88.5%, attributed to introducing directions in the accompanying discussion, instead compares no guidance with the full combination of distance, direction, and mask guidance. The missing distance-plus-mask variant is therefore important: without it, the contribution of directions in the final system remains unclear. Similarly, Table 4 compares field reconstruction errors across latent models but does not establish whether FSQ improves final HOI reconstruction over a matched continuous autoencoder. The results support the full system, but do not yet establish which parts of the proposed representation account for its gains.
Questions for authors
Where exactly does human–object relative pose enter LEXIS-Net? Please specify the coordinate transformations used for the object features, field query points, and supervision targets. Can two configurations with identical body articulation and canonical object shape but different relative object placements produce different encoded signatures?
What happens when mask guidance is combined with distance-only InterField guidance, without directions? Also, does FSQ improve downstream reconstruction over a continuous autoencoder with comparable capacity and the same training data? These comparisons would help establish whether the claimed representation choices are responsible for the final gains.
Please reconcile the text-conditioning results in Table 3 and Table S.1. Table 3 reports an object CD reduction of 1.11 cm from adding text, whereas Table S.1 implies a 0.31 cm difference between the null-token input and the default Claude Opus description. The appendix states that the no-text rows use the same model with a null token. Were different subsets, sampling settings, or evaluation protocols used?
Section S.2.5 applies a similarity transform to the human but only the corresponding rigid transform to the object, then states that all metrics are computed on the aligned surfaces. Are Contact, Collision, and Stability evaluated before or after this alignment? Applying different scale transformations can change the relative human–object geometry, so this detail could affect the interpretation of the interaction scores.
Thank the authors for the detailed responses, which has addressed majority of my concerns. I have adjusted the rating.