acceptodds
Under review as a conference paper at ICLR 2027

COSMI: COmpositional Synthesis of Multi-Object Interactions

Abstract

Generative models of human-object interaction are bounded by the data that exists: everyday activities involve several objects, but most captured datasets record one at a time, as multi-object capture is combinatorially expensive. Our observation is that interactions are local, so single-object captures already contain the parts of multi-object activities. We compose them: contact-consistent clips of single interactions, mirrored to balance the hands, transfer between bodies, and a language model and geometric checks admit only the pairings that are plausible, semantically and physically. Therefore, the dataset grows combinatorially with the clips rather than recording time. The COSMI dataset holds 222k sequences and 275 hours with up to five objects, nearly thirty times the largest multi-object capture, and can be extended by adding datasets or even hand-object recordings. On this data we train the COSMI method, a text-to-interaction diffusion transformer that follows how the data is built: weight-shared object slots generate a variable number of objects, predicted relative to the body parts that move them. On a benchmark with an unseen object and unseen interaction combinations, models trained on the dataset generalize to the unseen combinations. COSMI outperforms baselines in text alignment and contact accuracy, where its margin is largest on the unseen object. Code, models, and the dataset pipeline will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
human-object interaction, text-to-motion, interaction synthesis, diffusion models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qyNROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents COSMI, a pipeline for composing multi-object human interactions from single-object motion data, and a text-conditioned model for generating such interactions. The pipeline extracts and labels atomic motion clips from several existing datasets, then uses action and object annotations, composition constraints, and geometric filters to construct feasible combinations. The resulting dataset contains 221,677 sequences involving 36 objects and up to five objects per sequence. The proposed diffusion Transformer uses shared object slots, part assignment, object-relative motion, and contact-aware inference guidance. The authors evaluate the dataset and model using physical and semantic metrics, comparisons with adapted baselines, held-out object pairings, and ablations. The results indicate improvements on several reported measures, although the evidence for semantic plausibility and the interpretation of the contact-accuracy results would benefit from further validation.

Critical strengths
  1. The paper offers a practical approach to scaling multi-object interaction data. Composing interactions from existing single-object motion data addresses a meaningful data bottleneck, and the reported dataset scale and range of object counts make the resource potentially useful for studying compositional generalization. This is a central reason I view the contribution positively.

  2. The generation model is designed around the structure of the task. Shared object slots, part assignment, object-relative motion, and contact-aware guidance are appropriate choices for variable-object interactions. The comparisons and ablations provide useful evidence about how these components contribute, supporting the paper’s methodological value beyond the dataset alone.

Critical weaknesses
  1. The semantic plausibility of the composed dataset is not independently validated. The pipeline uses corrected action labels, per-action object and head/gaze flags, composition rules, and geometric filters. These checks can establish compatibility under the chosen rules and help reject physically implausible combinations, but they do not show that people judge the accepted sequences to be semantically plausible interactions. This matters because the scale and usefulness of the composed dataset are central claims of the paper. A modest human audit of accepted sequences would materially strengthen that claim.

  2. The reported per-marker “B-Acc” contact metric is difficult to interpret without class-sensitive statistics. B-Acc evaluates marker contact labels across 138 markers, while contact states may be sparse. Without reporting the positive-label prevalence or positive-class metrics, it is unclear how much of the score reflects correctly identifying actual contacts versus correctly predicting non-contact states. The paper also reports T-Acc, which is a useful complementary measure, but the missing statistics limit how strongly the B-Acc results support the claimed contact advantage.

Questions for authors

Q1: Could you report a human audit of a representative sample of sequences accepted by the current composition pipeline, including the fraction judged semantically plausible? This would help assess the semantic-quality claim for the existing dataset.

Q2: What fraction of marker–object labels in the test set are positive? Could you report a majority-negative baseline and positive-class precision/recall or balanced accuracy, and clarify whether COSMI’s advantage persists on positive contact labels?

3 comments

R3 – W1/Q1 (user study of semantic plausibility)

  • We ran the requested study: 55 participants rated a total of 1,000 composed sequences and 113 implausible controls (App. A.2, Tab. 7, Fig. 7). The participants watched rendered videos of composed sequences with their captions and decided whether each interaction is plausible. Each participant rated a random subset of at least 25 sequences. The 1,000 sequences are sampled from the dataset, covering varied object combinations, and each is rated once or twice. As a check, we mixed in control sequences that the composition rejects as implausible, such as drinking from a cup and from a mug at the same time. A sequence is plausible if most of its votes say so, and a tie if its votes are split.
SequencesPlausibleTieImplausiblePlausible (%)
One object1501462297.3
Two objects3503424497.7
Three objects3002915497.0
Sequential2001971298.5
All composed1,000976121297.6
Implausible controls113121100.9
  • The participants judge 976 of the 1,000 composed sequences plausible and only 12 implausible. At least 97 % are plausible in every group. No composed sequence was rejected by two participants, and the few rejected ones are unusual rather than impossible, such as drinking from a bowl while walking.
  • They reject 110 of the 113 controls, which shows that they told plausible from implausible interactions. Only one control is judged plausible, and two are ties.

R3 – W2/Q2 (class statistics of the contact metrics)

  • The reading is correct: contact is sparse at marker level. Only around 9 % of the marker labels of the test set are positive, so predicting no contact everywhere already reaches a B-Acc of 0.909. We computed the requested statistics for every method on the same labels (new App. A.6, Tab. 10). B-Acc (body contact accuracy) and T-Acc are the values of Tab. 2, and the other columns are pooled over all labels.
  • At marker level, the advantage of COSMI is larger on the positive labels. COSMI is the best in every statistic, with an F1 of 0.824 against 0.573 for the best baseline. B-Acc indeed hides how few contacts HOI-Diff and Dex2HOI detect, since both are level with the all-negative baseline in it.
MethodB-Acc ↑Bal. acc. ↑Precision ↑Recall ↑F1 ↑IoU ↑
All-negative baseline0.9090.500–0.0000.0000.000
MDM0.9340.7060.6890.4330.5320.362
PriorMDM0.9360.7350.6800.4940.5730.401
HOI-Diff0.8910.5270.2320.0810.1200.064
Dex2HOI0.9120.5920.5210.2030.2930.171
COSMI (ours)0.9680.9170.7940.8570.8240.701
  • At frame level, COSMI stays the best when contact and no-contact frames are weighted equally. Here contact is the majority class, with around 90 % of the labels positive, so the balanced accuracy is more illustrative, because it weights contact and lack of it equally. COSMI keeps the highest balanced accuracy, 0.946 against 0.922 for the best baseline, and also the highest recall and F1.
MethodT-Acc ↑Bal. acc. ↑Precision ↑Recall ↑F1 ↑
All-negative baseline0.1130.500–0.0000.000
MDM0.9010.8900.9860.8890.935
PriorMDM0.9380.9220.9890.9360.962
HOI-Diff0.7560.7480.9620.7540.846
Dex2HOI0.7520.7530.9640.7400.838
COSMI (ours)0.9800.9460.9880.9900.989

Revised manuscript

We hope that our reply addresses the reviewer's concerns on the semantic plausibility of the composed sequences and on the interpretation of the contact metrics. We invite the reviewer to check the following updates in the PDF:

  • User study: L246–L247 and p21–22 (App. A.2): Table 7, Figure 7.
  • Contact statistics: L441 and p23–24 (App. A.6): Table 10.

We thank the reviewer for their valuable remarks and await any further feedback.

Thanks for the detailed response. I will keep my positive rating.

We thank the reviewer for the constructive review and the continued support.

Reviewer_LbLjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents the COSMI dataset, a multi-object interaction dataset created by combining human-object interaction datasets, containing 222k sequences with up to five objects. The multi-object interaction sequences are formed by combining multiple local hand-object and/or body-object interactions and by mirroring; the resulting sequences are annotated by a Qwen3 LLM. It further introduces the COSMI method, a new human-multi-object interaction generation method that introduces human-to-multi-object feature interaction via attention and feature part assignment, joint-relative prediction, auxiliary losses, and contact-consistent guidance. The method achieves superior performance on the majority of metrics in the COSMI dataset compared to MDM, PriorMDM, and HIMO.

Critical strengths

The paper introduces the COSMI dataset, which utilizes multi-object composition to compose multiple human-object interaction datasets, resulting in a large dataset with a significantly higher number of sequences and hours than previous multi-object datasets. The composition pipeline directly addressing the data scarcity issue in the multi-object interaction domain.

Critical weaknesses
  1. All experiments are in the COSMI dataset, with the training data, test ground-truths, and evaluator all coming from the same data composition pipeline. There is no cross-dataset evaluation that proves training/fine-tuning on COSMI could improve performance on real-world multi-object datasets (such as HIMO).

  2. Weak evaluations for the COSMI method: Only baselines from 2023-2024 (MDM, PriorMDM, HIMO) are compared against COSMI. It is unclear why later baselines weren't compared; for example, SyncDiff (He et al., 2025), HOIDiff (Peng et al., 2025), and HOIMask (Ji et al., 2026).

Questions for authors
  1. Can the authors add at least one cross-dataset transfer experiment that assesses the effect of training on COSMI on other multi-object HOI tasks?
  2. Can the authors compare the COSMI method to HOI baselines in 2025-26?
  3. Can the authors add clarifications for "275 hours" / "nearly thirty times the largest captured multi-object dataset"? From my understanding, this size calculation is based on the combination of multiple atomic clips, where the same atomic clip might appear more than one time. The 275 hours number is not directly comparable to, for example, 9.44 hours of real-world clips of HIMO.
3 comments

R2 – W1/Q1 (evaluation outside COSMI, cross-dataset transfer)

  • On HIMO data, pretraining on COSMI lowers the object jerk by more than 40 % and the foot sliding by 8 to 9 %. We add the requested transfer experiment, detailed in the general reply: our model pretrained on COSMI and fine-tuned on HIMO is compared with the same model trained on HIMO alone, on HIMO's test split. Like HIMO, COSMI is based on real captures, and the model retains the prior learned from the composed sequences, the coordination of body and objects and smooth object motion, when fine-tuned on HIMO.
  • The learned metrics of HIMO's evaluator do not improve because of the domain gap: the two datasets capture different interactions. HIMO is tabletop manipulation whose verbs, such as pick up, assemble and cut, do not occur in COSMI, which instead composes walking, sitting, and hand, foot and head interactions.

R2 – W2/Q2 (baselines from 2025–26)

  • We add two baselines from 2025–26: HOI-Diff (CVPRW'25) and Dex2HOI (arXiv'26). Of the named methods, only HOI-Diff has public code; SyncDiff and HOIMask have not released any as of November 2026. Dex2HOI, a full-body hand-object interaction generator, is added as a second recent method.
  • Both are extended to five object slots with their original design, and both are larger than COSMI (43.5M and 54.7M parameters against 30.4M). Both are designed for one or two rigid objects, so we extend them to five object slots with articulated objects, each as a single model for all tiers. HOI-Diff keeps its body representation, affordance model and guidance, and Dex2HOI its wrist-relative object representation, losses and training recipe (App. D.8).

All-objects tier, in the format of Tab. 2 (diversity and penetration best when closest to the data, sweep when closest to 1):

MethodFID ↓Top-1 ↑Top-3 ↑MM-Dist ↓Div. →Foot sl. ↓Jerk ↓T-Acc ↑B-Acc ↑Pen. (%) →Hinge err. ↓Sweep →
Data–0.7400.9190.7404.311.8790.366––16.86––
MDM0.1480.6190.8611.0894.113.2900.4950.9010.93419.810.5470.846
PriorMDM0.0770.6640.8760.9334.181.8740.3900.9380.93618.550.5570.707
HOI-Diff (new)0.1560.6380.8751.0984.232.5950.5560.7560.89121.970.5451.380
Dex2HOI (new)0.2400.5580.7791.3554.131.7430.3430.7520.91218.860.6421.225
COSMI (ours)0.0490.6960.9000.8274.151.3950.4090.9800.96816.730.4370.665
  • COSMI is ahead of both in FID, R-precision, MM-Dist, foot sliding, contact accuracy and hinge error, with a T-Acc of 0.980 against 0.756 and 0.752. It is also the closest to the penetration of the data. HOI-Diff is competitive in R-precision and Dex2HOI has the smoothest object motion, but both have the least accurate contacts of all methods. Both baselines are in Tabs. 2, 3 and 4 and described in Sec. 5.

R2 – Q3 (275 hours with recurring clips against the 9.44 captured hours of HIMO)

  • The 275 hours count distinct sequences. As detailed in the general reply, every composed sequence is distinct: no combination of clips repeats, and sequences that share a clip still differ. Like the 9.44 hours of HIMO, the 275 hours thus count distinct multi-object sequences, composed instead of captured together. "Nearly thirty times" compares these hours of distinct sequences, not hours of captured motion.
  • These hours cover far more combinations than captured ones. The composition coverage far exceeds captured data (general reply): 1,361 combinations of two or more objects against 145 in HIMO. Each combination of interactions is realized about 20 times with different captured motion.
  • Composed hours complement captured ones. We do not claim that composed hours replace captured ones. The composition pipeline scales and extends the coverage of the captures, so it is not limited to the current source datasets. For example, HIMO can enter as a source dataset, which would extend its dependent interactions, such as pouring and cutting, from a nearly static body to sitting and walking.

Revised manuscript

We hope that our reply addresses the reviewer's concerns on the evaluation outside COSMI, on recent baselines, and on the comparison of composed with captured hours. We invite the reviewer to check the following updates in the PDF:

  • Cross-dataset transfer: pretraining on COSMI before training on HIMO, L477–L480 and p23 (App. A.4).
  • Recent baselines: HOI-Diff and Dex2HOI in Tables 2 to 4, described in L450–L455 and on p39 (App. D.8).
  • Coverage: combination counts of COSMI against HIMO, L252–L256.

We thank the reviewer for their valuable remarks and await any further feedback.

Thank you for the author's detailed response. The authors answered my questions regarding dataset transfer and added new baselines for comparison. These largely resolve my concerns about weaknesses one and two, as well as questions one and two.

However, I'm not totally convinced by the answer to my question 3. Although the sequences are distinct, the number of hours is not directly comparable to the numbers in previous datasets. I still encourage the authors to clarify this distinction in the manuscript.

Overall, I would like to raise my rating to weak acceptance. Thanks again for the authors' efforts.

We thank the reviewer for the follow-up and for raising the rating. As suggested, we have updated the manuscript to clarify the distinction between composed and captured hours: the abstract and introduction now compare COSMI with captured data by coverage of object combinations (1,361 against 145 in HIMO) instead of by hours, Sec. 3 states that composed hours are not directly comparable to captured ones, and Tab. 1 marks them as composed.

Reviewer_P2dROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

COSMI addresses the scarcity of multi-object human–object interaction data through a compositional data pipeline and a unified generative model. The pipeline combines local body motions and associated object trajectories, applying semantic and geometric filtering to produce approximately 222K sequences involving up to five objects. A shared diffusion Transformer generates human and object motions using joint-relative representations and contact guidance. Experiments evaluate generation quality and compositional generalization on the proposed dataset, with additional evaluation on HIMO.

Critical strengths
  1. The data pipeline offers a practical way to expand existing interaction recordings into diverse multi-object compositions. Its combination of motion transfer, semantic compatibility checks, and geometric filtering addresses an important data bottleneck.
  2. A single model supports varying numbers of objects through shared object representations. The joint-relative motion representation provides a well-motivated mechanism for maintaining coordination between body movements and object trajectories.
Critical weaknesses
  1. The composition pipeline mainly combines spatially separable interaction primitives. The rule rejecting object pairs within 1 cm of each other (Appendix C.7) excludes operations requiring direct object–object contact, limiting the dataset’s coverage and practical value for functionally coupled multi-object manipulation.
  2. The dataset expands 1,491 atomic clips into approximately 222K sequences through fixed composition rules, with frequent reuse of local motion components. This redundancy and its effect on learned diversity are not quantified, and generalization to unseen motion primitives remains untested.
Questions for authors
  1. The effect of mirroring on local hand–object contact and penetration should be clarified, particularly for asymmetric objects. The small template-alignment residuals reported in Appendix C.5 do not directly quantify contact preservation.
  2. The HIMO experiment retrains the generator on HIMO (Appendix A.3), leaving the benefit of the composed dataset for captured interactions unclear. The authors should clarify what existing evidence supports improved generation in real, functionally coupled scenarios.
  3. The physical-validity claims should be reconciled with the permissive contact criterion in Appendix C.7 and the reported penetration exceeding 2 cm in 34.60% of dataset frames and 33.96% of generated frames (Table 2).
  4. The mixed HIMO results in Table 7 require further discussion, particularly the worse two-object FID than HIMO-Gen (4.896 vs. 1.481) and the lowest multimodality in both tiers, despite improvements in text alignment and three-object FID.
5 comments

R1 – W1 (1 cm rule, functionally coupled manipulation)

  • Sources are independent by design. COSMI composes single-object captures, so any two composed objects come from different source captures. For example, eating an apple with the right hand and drinking from a cup with the left are unrelated, so a contact between the apple and the cup would be a composition artifact. The 1 cm rule rejects exactly these artifacts.
  • The composition pipeline can support multi-object sources. Nothing prevents a multi-object capture from entering the composition as one source clip, since every object follows the body part that moves it (Eq. 1). For example, pouring from a teapot into a cup would enter as one source clip. The 1 cm rule would not apply between the teapot and the cup, which come from the same source capture.
  • Using multi-stage and multi-object activities as sources is a natural future direction. Dependent actions such as pouring need a source capture that contains both objects. Sec. 6 names multi-stage activities and multi-object sources as the most promising future direction, with pouring from a teapot into a cup as an example of a multi-object source. For example, adding HIMO as a source dataset would extend its multi-object tabletop manipulation, captured with a nearly static body, to sitting and walking.

R1 – W2 (reuse of atomic clips, learned diversity, unseen motion primitives)

Reuse of atomic clips

  • The reuse is quantified, and it does not make the sequences redundant. A clip is used in 307 sequences on average and in 13 at the median, or in 530 and 91 when a mirrored clip is counted with its original. The most-used clip, in 6,984 sequences, is a sitting clip on the black chair, which is in the test set with all its sequences and thus absent from training. As detailed in the general reply, no combination of clips repeats, the composition coverage far exceeds captured data, and each of the 11,038 combinations of interactions is realized about 20 times with different clips. The effect of the reuse on the learned diversity and on unseen motion primitives is measured below.

Learned diversity

  • Generated diversity matches the data. The generations of COSMI reach a diversity of 4.15±0.324.15 \pm 0.32, statistically indistinguishable from the 4.31 of the data (Tab. 2).
  • Learnable diversity grows with new sources. Neither the clips nor the rules of the composition pipeline are fixed, so every new rule or capture adds motion primitives and combinations. A new interaction type enters as a new rule, defined by contacts, pose constraints, and reference joints. New clips can come from unselected sequences and objects of the current source datasets, from further full-body datasets, and from other domains: hand-only recordings with a fitted body, as already done with HOT3D (Fig. 4, App. A.9), and multi-object captures such as HIMO, which we plan as a future direction.

Unseen motion primitives

  • COSMI generalizes to unseen geometry and motion primitives. The COSMI test set covers both an unseen geometry and unseen motion primitives. It holds out the black chair with all its atomic clips, so no motion recorded on it is seen in training. MDM, PriorMDM, and Dex2HOI lose four to nine points of temporal contact accuracy (T-Acc) on the black chair, and no baseline exceeds 0.895, whereas COSMI reaches 0.975, within one point of its value on the whole test set (Tab. 4).
  • Models generalize to unseen combinations. On five held-out combinations whose parts remain in training, MDM, PriorMDM, and COSMI match or exceed their T-Acc on the whole test set, with one drop of three points (Tab. 4). A model that just memorized its training combinations could not do this.

R1 – Q1 (effect of mirroring on contact and penetration, asymmetric objects)

We now measure the contact directly: every one of the 1,172 mirrored one-handed and one-footed clips is compared with its original frame by frame (Tab. 13 in App. C.5).

ClipsCountFrames in contact (%), originalFrames in contact (%), mirroredDistance change (mm)Penetrating frames (%), originalPenetrating frames (%), mirrored
Nearly symmetric objects (residual below 0.2 mm)42100.0100.00.052.54.7
Other objects91999.199.00.220.20.2
Hold poses (single frame)211100.0100.00.120.91.9
All1,17299.299.10.220.20.3
  • Mirrored clips keep the contact and the penetration of their originals: 99.1 % of the frames in contact against 99.2 %, and 0.3 % penetrating frames against 0.2 %. The distance of the acting hand or foot to the object surface changes by 0.22 mm on average. A frame counts as penetrating when the body is more than 2 cm inside the object, and the rate rises only for the water bottle, whose eight clips reach a depth of 2.6 cm, just past this threshold, and for the hold poses, from two to four of 211.
  • Asymmetric objects keep both as well: 99.0 % of the frames in contact against 99.1 %, and 0.2 % penetrating frames before and after. These are the values of the other objects in the table, which include all asymmetric ones, with residuals of up to 4.0 mm.
  • Only our mirroring keeps the contacts at the corresponding point of the object: 98.9 %, against 55.5 % and 17.7 % when the object is reflected. App. C.5 now ablates our local mirroring against local and global reflection, which both replace the mesh by its reflected version, thus changing the geometry of the object (Tab. 12, Fig. 16). On the three objects with the largest residuals, our mirroring still keeps 98.0 %.

R1 – Q2 (benefit of the composed data for captured interactions)

  • On captured data, pretraining on COSMI lowers the object jerk by more than 40 % and the foot sliding by 8 to 9 %. The HIMO experiment of the submission indeed evaluates the method and not the data, so we add this transfer experiment, detailed in the general reply. HIMO is a captured dataset of functionally coupled manipulation, such as pouring and cutting, and the comparison is with the same model trained on HIMO alone. Like HIMO, COSMI is based on real captures, and the model retains the prior learned from the composed sequences, the coordination of body and objects and smooth object motion, when fine-tuned on HIMO.
  • The learned metrics of HIMO's evaluator do not improve because of the domain gap: the two datasets capture different interactions. HIMO is tabletop manipulation whose verbs, such as pick up, assemble and cut, do not occur in COSMI, which instead composes walking, sitting, and hand, foot and head interactions.
  • On the COSMI benchmark, our method generalizes to unseen combinations of interactions and to an unseen object. The five combinations never occur in training while their parts do, so the model learns to compose interactions and does not memorize the training combinations. The black chair is completely unseen, held out with all its sequences, and we attribute the generalization to it to the number and variety of the composed sequences.

R1 – Q3 (physical validity, contact criterion, penetration in Tab. 2)

  • The contact criterion is only a rejection threshold: 99.1 % of the interacted objects keep the contacts of their rule in at least 90 % of their frames. Half of the frames is the limit below which a composition is rejected, and on the test set the median object keeps its contacts in every frame (App. A.1).
  • The penetration of Tab. 2 was inflated: the corrected rates are 16.86 % for the data and 16.73 % for COSMI, against 34.60 % and 33.96 % in the submission. The cause is a bug in the distance grids, detailed in the general reply. Most of what remains is inherited from the captures and not added by composition: we compare the composed sequences with the captured clips their objects come from, on the same frames and by the role of the object (test set, Tab. 6 in App. A.1).
Role of the objectObjects (%)Frames with penetration deeper than 2 cm (%), capturedFrames with penetration deeper than 2 cm (%), composedFrames with penetration deeper than 2 cm (%), generated by COSMI
Seat16.876.374.970.2
Worn18.4–11.710.2
Held37.63.23.84.9
Hold pose (single frame)24.2–2.52.9
Foot2.92.92.916.3
All100–16.916.7
  • Seats cause three quarters of the penetration, and they penetrate as often in the captures (76.3 %) as in the composed sequences (74.9 %). The legs of the fitted body sink into the seat, and the penetration is shallow, with a median depth of 3.4 cm. Worn objects, attached to bodies of different shapes, cause half of the rest, and held and foot objects penetrate about as often as in their captures. The generations of COSMI follow the data in every role except the foot objects.
  • The physical-validity claim refers to the feasibility checks, and the revised paper now states it this way. The checks (App. C.7) reject the intersections that composition can introduce: between two objects, of an object or a hand with the lower body, and between the arms. The text in the paper is updated to reflect it more clearly.

R1 – Q4 (HIMO results of Tab. 9, Tab. 7 of the submission: two-object FID, multimodality)

  • The two-object FID gap reflects the sensitivity of HIMO's evaluator to the hand representation: with full hand poses, the FID of COSMI drops from 4.896 to 0.230, against 1.481 for HIMO-Gen. Our representation stores each hand as the 24 leading PCA coefficients of the MANO pose space, which explain 99.3 % of the hand variance of COSMI and 83 % of that of HIMO. The trailing components that we omit carry the rest, and the learned evaluator of HIMO is highly sensitive to them. Retraining the same model with full hand poses is the only change between the two COSMI rows of the table (two / three objects, HIMO's evaluator).
MethodFID ↓Top-3 ↑MM-Dist ↓MModal. ↑
Data0.018 / 0.1810.799 / 0.6993.566 / 3.770–
HIMO-Gen1.481 / 4.7710.637 / 0.5353.649 / 5.0871.786 / 0.756
COSMI, PCA hands (submission)4.896 / 2.8150.730 / 0.5824.436 / 4.3660.691 / 0.741
COSMI, full hand poses0.230 / 0.8330.804 / 0.7243.487 / 3.6140.837 / 0.853
  • With full hand poses, COSMI has the best FID, top-3 R-precision and MM-Dist in both tiers. The interaction itself does not change with the hand representation: the temporal contact accuracy of the two models differs by at most 0.003 (0.744 against 0.741 for two objects, 0.693 against 0.691 for three).
  • Multimodality becomes the highest with three objects (0.853) and stays below the baselines with two (0.837). It trades off against text alignment: HIMO-Gen varies more for one caption (1.786) at a much lower R-precision (0.637 against 0.804). The new row is in Tab. 9 and discussed in App. A.4 and Sec. 5.

Revised manuscript

We hope that our reply addresses the reviewer's concerns on the coverage and reuse of the composed data, the effect of mirroring, the physical validity of the data, and the results on HIMO. We invite the reviewer to check the following updates in the PDF:

  • Mirroring: contact ablation and frame-by-frame check of the mirrored clips, p33–34: Tables 12 and 13, Figure 16.
  • Penetration: corrected values in Tables 2, 3 and 8, and penetration by object role with contact retention, p20: Table 6.
  • Physical validity: the claim now refers to the feasibility checks, L77–L79 and L524–L525.
  • HIMO: full hand poses and pretraining on COSMI, L475–L480 and p21–23: Table 9.
  • Coverage and future work: combination counts against HIMO, L252–L256, and multi-object captures as sources, L534–L537.

We thank the reviewer for their valuable remarks and await any further feedback.

Comments on the paper

2 comments

We thank the reviewers for their careful reading and constructive feedback. Points are labelled by reviewer (R; R1 = P2dR, R2 = LbLj, R3 = qyNR) and by weakness (W) or question (Q), in the order of the reviews, and all section, table and figure numbers refer to the revised paper.

Positive feedback

The reviewers highlight the following strengths:

  • Presentation (R1, R2, R3): the paper is clear and easy to follow.
  • Composition (R1, R2, R3): a practical way to scale multi-object interaction data that addresses an important data bottleneck.
  • Dataset scale (R2, R3): a number of sequences, hours and objects per sequence beyond earlier multi-object datasets.
  • Method (R1, R3): a single model for a varying number of objects through shared object slots, with a well-motivated joint-relative representation (R1) and a design that follows the structure of the task (R3).
  • Evaluation (R3): comparisons and ablations that support the value of the method beyond the dataset alone.

Additions to the paper

The manuscript and supplementary video are updated. The revised version contains the following new results, which the replies below summarize:

  • Recent baselines (R2 – W2/Q2; Tabs. 2 to 4, Sec. 5, App. D.8): HOI-Diff (2025) and Dex2HOI (2026), extended to five object slots and to articulated objects. In the all-objects tier, COSMI remains best in FID, R-precision, MM-Dist, foot sliding, contact accuracy and hinge error.
  • COSMI data as pretraining for HIMO (R2 – W1/Q1, R1 – Q2; App. A.4, Sec. 5): pretraining on COSMI before training on HIMO lowers the object jerk by more than 40 % and the foot sliding by 8 to 9 % on HIMO's test split.
  • Explanation of mixed results on HIMO (R1 – Q4; Tab. 9, App. A.4): the two-object FID gap to HIMO-Gen comes from the sensitivity of the learned evaluator of HIMO to the hand representation. With full hand poses in place of the compressed PCA representation, COSMI reaches an FID of 0.230 against 1.481.
  • User study of semantic plausibility (R3 – W1/Q1; App. A.2, Tab. 7, Fig. 7): 55 participants judge 976 of 1,000 composed sequences plausible and only 12 implausible, with the votes tied on the remaining 12.
  • Detailed evaluation of contacts (R3 – W2/Q2; App. A.6, Tab. 10): we evaluate the contacts of our method and of all baselines in detail, adding the share of contact labels, a no-contact baseline, the balanced accuracy, and precision, recall, F1 and IoU on the contact labels. On the contact labels alone, the advantage of COSMI is larger than in B-Acc, with an F1 of 0.824 against 0.573 for the best baseline.
  • Effect of mirroring on contact (R1 – Q1; App. C.5): every mirrored clip is compared with its original frame by frame. The acting hand or foot stays in contact in 99.1 % of the frames, against 99.2 % in the originals, and the penetration is nearly unchanged (Tab. 13). We also ablate our mirroring against naive reflecting of the object mesh: with ours, 98.9 % of the contacts land on the symmetric point of the object, against at most 55.5 % with a reflected mesh (Tab. 12, Fig. 16).
  • Penetration by object role and contact retention (R1 – Q3; App. A.1, Tab. 6): seats cause three quarters of the penetration of the data, as often in the original captured clips as in the composed sequences in COSMI. On the test set, 99.1 % of the interacted objects keep the contacts of their rule in at least 90 % of their frames.
  • Coverage (R1 – W2, R2 – Q3; Sec. 3): 1,361 combinations of two or more objects and 11,038 combinations of body parts, actions and objects, against 145 object combinations in HIMO.
  • Retargeting to humanoids (new qualitative result; Sec. 5, App. A.11, Figs. 6 and 12): composed sequences retargeted with their objects to the Unitree G1 and H2 humanoids with dexterous hands. We also add videos of the retargeted sequences to the supplementary material.

Shared concerns

Two concerns are raised by both R1 and R2, so we answer them here.

R1 – W2, R2 – Q3 (reuse of atomic clips in the composed sequences)

  • Every composed sequence is distinct. Each compatible pair of a target and a source clip is composed only once, so no two composed sequences contain the same atomic clips (Sec. 3, Statistics). Sequences that share a clip still differ in body part through mirroring (the other hand or foot), target motion (walking, sitting, or standing), other objects, order in a sequential composition, and body shape. For example, eating an apple with the right hand also appears with the left hand, while sitting or walking, while holding another object, before or after another interaction, and on different body shapes.
  • Composition coverage far exceeds captured data. We count combinations at the level of interactions, not of captured clips: an interaction is a body part, an action, and an object, such as eating an apple with the right hand. From 36 objects and 151 such interactions selected from the source datasets, COSMI composes 1,361 combinations of two or more objects and 11,038 combinations of interactions, against 145 combinations of 53 objects in HIMO, the largest multi-object capture. Capturing every combination is combinatorially expensive.
  • Each combination is realized with different captured motion. Each interaction has about ten atomic clips from 3.7 subjects in the source datasets, which differ in style and timing. Each combination of interactions is therefore realized about 20 times with different clips (222K sequences for 11K combinations).

R1 – Q2, R2 – W1/Q1 (benefit of the composed data on captured data, cross-dataset transfer)

We add the requested transfer experiment: the model trained on COSMI is fine-tuned on HIMO with the recipe and budget of training from scratch and evaluated on HIMO's test split (two / three objects).

MethodObject jerk ↓Foot sliding ↓Top-3 ↑ (HIMO evaluator)FID ↓ (HIMO evaluator)
HIMO data0.147 / 0.1190.194 / 0.1400.799 / 0.699–
Ours trained on HIMO0.375 / 0.3770.222 / 0.2080.730 / 0.5824.90 / 2.82
Ours trained on COSMI and fine-tuned on HIMO0.216 / 0.2010.202 / 0.1920.712 / 0.5665.22 / 2.65
  • Pretraining on COSMI lowers the object jerk on HIMO by more than 40 % and the foot sliding by 8 to 9 %. Both gains hold with two and with three objects, and both metrics move closer to those of the HIMO captures. Like HIMO, COSMI is based on real captures: every sequence combines captured interactions, and the resulting data is natural and physically plausible enough to improve the model performance on another, captured dataset. The model learns the coordination of body and objects and smooth object motion from the composed sequences, and retains this prior when fine-tuned on HIMO.
  • The learned metrics of HIMO's evaluator do not improve because of the domain gap: the two datasets capture different interactions. HIMO is tabletop manipulation with a nearly static body and resting objects, and its verbs, such as pick up, assemble and cut, do not occur in COSMI, which instead composes walking, sitting, and hand, foot and head interactions.

Correction of the penetration metric (Tabs. 2, 3 and 8, App. B.2)

  • We found a bug in the distance grids that inflated the penetration reported in the submission. The signed distance grids from which the penetration for the method evaluation was computed derived their sign from surface normals. This is reliable only for watertight meshes, whereas several objects are open scans, mostly the furniture of BEHAVE, so a body sitting on or standing next to such an object was counted as deep inside it.
  • The penetration of all methods is recomputed on the same samples, and no other metric changes. With the corrected grids, the penetration of the data drops from 34.60 to 16.86 % and that of COSMI from 33.96 to 16.73 %. Nothing else in the evaluation relies on these grids, and all penetration numbers below and in the revised paper use the corrected ones.
  • COSMI is the closest to the penetration of the data in every tier. As the original captures themselves have penetration, methods are now ranked by the distance of their rate to that of the data. COSMI has 16.73 against 16.86 % for all objects, 6.94 against 6.91 % for two and 25.80 against 25.39 % for three, whereas some baselines now fall below the data with two and three objects (Tab. 3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.