- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper adapts an existing human-object interaction motion to unseen object properties, such as mass, scale, center of mass and friction, without property-labeled motion data. HOI-UP describes the reference with 17 interpretable HOI-codes per interaction phase, asks a VLM (Claude Opus 5) for phase-wise target ranges of these codes under the new properties, and realizes them by optimizing spline corrections to the reference under contact, balance and joint-load terms. A fine-tuned InterMimic controller then tracks the plan in Isaac Gym under the target properties, serving as both refinement and executability test. An evaluation protocol compares adaptations of the same reference across property levels without ground truth. On 30 OMOMO references with 240 variations, HOI-UP achieves the highest tracking success, contact preservation and property response among the compared baselines.
Critical strengths
-
Formulates a clear and practically motivated task, adapting an existing property-agnostic HOI reference to unseen object properties without property-labeled data, with a coherent pipeline whose components each answer a stated sub-problem.
-
Introduces an evaluation protocol for adaptation without ground truth, combining property response, reference preservation, naturalness, biomechanical demand and executability, together with a human-strength-informed actuation profile validated against literature joint moments.
-
Reports consistent gains in executability and contact preservation over controller- and motion-adaptation baselines under a shared actuation profile, with kinematic and vocabulary ablations and a candid limitations section.
Critical weaknesses
-
The central hypothesis is never tested. The paper rests on the claim that a VLM prior can replace property-labeled motion data and yield human-like adaptation, but no experiment checks either half. Adaptations are never compared with real motions captured under changed properties, although FORCE contains exactly such paired data and is used only to extract slope directions. The one metric grounded in human data, Eff, compares only the sign of each change, not its magnitude, and the three controller-adaptation baselines score higher on it than HOI-UP. Whether the VLM is needed at all is untested, since a rule table derived from the same biomechanics literature that selected the 17 codes could produce the same phase-wise ranges and is not compared. Whether a human prior is needed is also untested, since no physics-driven alternative is run, such as task-driven RL with an effort or balance cost that does not track the original reference. The claim that biomechanical objectives alone collapse strategies across property levels is asserted without an experiment.
-
The evaluation favors the method by construction. Minimality, Eff and Mon are computed on the 12 literature codes, which are a subset of the 17 codes the method optimizes directly, while the controller-adaptation baselines never optimize them. Only R² uses independent features. The motion-adaptation baseline CoMo-HOI predicts its edits with a text-only Qwen2.5-7B, whereas HOI-UP uses Claude Opus 5 with rendered frames, so the gap in Table 1 mixes method and model. The plan-level and simulation-level results are reported on different sequence sets, so the effect of tracking on the adaptation cannot be read off.
-
The scope is narrower than stated and partly exceeds what the method can do. Hand positions are fixed object-local anchors, so the method cannot change where the object is held, which directly limits the center-of-mass axis. All 30 references, whose default masses range from 0.51 to 9.82 kg, are adapted to 10, 20 and 30 kg, so a 0.5 kg trashcan is adapted to 30 kg, beyond the lifting limit referenced by the paper’s own RNLE metric, where a human would change strategy rather than posture. Objects are large floor-level items and interactions are six floor-based phases, while the title and abstract claim human-object interaction in general. None of this appears in the limitations.
-
Positioning and novelty. The pipeline, a foundation model producing a structured high-level specification, a kinematic step filling it in, and a physics tracker refining it, is the structure of MLLM-driven mocap-free motion synthesis, which is not cited. Within this structure the VLM contributes only phase-wise ranges for 17 scalars, the physics simulator is used to score plans and never to inform them, and there is no feedback from execution to the plan.
Questions for authors
-
No code, prompts or reproducibility statement are provided, and the VLM is a closed API. Will the code, the full prompt, the 240 VLM outputs, the optimization settings and the list of the 30 sequences be released?
-
How is the VLM queried? Temperature, number of samples per pair, variance across repeated queries, and the fraction of code ranges left at the reference value due to low confidence. What changes with text-only input and without the rendered frames?
-
What do a literature rule table for the target ranges and a physics-driven baseline that does not track the original reference achieve under the same protocol?
-
FORCE contains motions under several resistance levels. How do adapted motions compare with the real higher-resistance motions in both direction and magnitude of the 17 codes?
-
Why was CoMo-HOI driven by a text-only Qwen2.5-7B rather than the same VLM, and how was the LLM-implemented baseline validated?
8 comments
-
Revised Eff metric. Eff compares mass-dependent motion-feature trends with FORCE, a mass-conditioned HOI motion-capture dataset. We made three changes to this metric.
- Filtering failed trials from FORCE. The submitted version also used failed trials in which the subject fails to lift a heavy object. The revised version uses only successful captures.
- Fair feature selection. Comparing HOI-code trends disadvantages InterMimic-style baselines, which do not use these codes. We therefore compare changes in motion features that none of the methods including HOI-UP uses.
- Direction- and magnitude-aware comparison. Whereas the submitted version measured only the agreement in the direction of feature changes, the revised version compares both the direction () and the magnitude () of feature changes with FORCE.
Results: Compared with the successful FORCE captures, HOI-UP achieves the highest agreement in both the direction () and the magnitude () of mass-dependent motion-feature changes. This indicates that HOI-UP yields adaptations closer to real captures than the other methods, without using captured property-labeled data.
Nevertheless, both the user study and the FORCE comparison can depend on subject preferences, and human adaptation to object-property changes can be multimodal. Therefore, we moderate our claims on human-likeness.
Contribution of HOI-code adaptation
We agree that the contribution of HOI-code matching was reported only at the kinematic stage through the No HOI-code ablation (Tab. 2), leaving its contribution to the end-to-end results unclear. To address this weakness, we added HOI-UP$_{\text{NoHOICode}}$ to Tab. 1 (App. A.9.5). It removes only the HOI-code objective from kinematic HOI optimization and uses the same RL fine-tuning and evaluation as HOI-UP.
HOI-UP$_{\text{NoHOICode}}$ shows lower physical executability (success rate, completion rate, HOI error) and naturalness (Jit$_{\text{HOI}}_{\text{100}}$ but lower Eff and R$^2_{\text{all}}^2_{\text{all}}$). This confirms that the VLM provides the prior knowledge required for object-property-dependent adaptations through HOI-code adaptation.
Confounding the method with the LLM in the CoMo-HOI comparison
We agree that the previous comparison confounded the method with the model. To disentangle them, we re-ran CoMo-HOI with the same Claude Opus 5 (effort high) as HOI-UP, keeping the text-only input of the original CoMo design.
With Claude Opus 5, the success rate of CoMo-HOI improves from 23.25% to 40.42%, but it remains far below that of HOI-UP (82.75%). CoMo-HOI (Claude Opus 5) also shows lower direction () and magnitude () agreement with FORCE than HOI-UP. Moreover, even with text-only input, HOI-UP outperforms CoMo-HOI at the kinematic stage in contact F1 (0.734 vs. 0.646), Comp$_{\text{HOI}}$ (8.69 vs. 33.41), Mon (42.58 vs. 42.14), and (19.70 vs. 11.18). Upgrading the foundation model alone therefore does not close the gap, which comes from the method rather than the VLM model.
Different evaluation sets in Tab. 1 and Tab. 2
We recompute Tab. 2 on 270 conditions: the 240 conditions of Tab. 1 and 30 additional conditions with unchanged object properties.
The results show that the two ablations each achieve the best values on one side at a large cost on the other. No HOI-code adaptation optimizes only hand contact, foot contact, and balance, and thus achieves the best naturalness and biomechanical demand (foot-floor error, , C$_{\text{100}}$). However, its is the lowest among all methods, indicating a weak response to property changes. Conversely, HOI-code Only follows only the VLM-predicted HOI-code targets and attains the best preservation, Mon, and R$^2_{\text{all}}$. However, its contact F1 drops to 0.733 and its foot-floor error is the largest, making its plans difficult to execute. HOI-UP ranks first in and second in , Jit$_{\text{HOI}}$, foot-floor error, , and Comp$_{\text{HOI}}$. This suggests that HOI-UP produces kinematic plans that satisfy both property response and physical plausibility. This balance carries over after RL fine-tuning, where HOI-UP achieves the highest success rate and contact F1 and the lowest foot-floor error (Tab. 1). That is, the balance between property response and executability, rather than the best value on any single kinematic metric, determines the final performance.
Limitations
Fixed hand positions. Our research scope in this work is how the pose of a given reference adapts when object properties change. For example, when the CoM changes, one can respond by changing the grasp location, but also by modifying the arm and body pose while keeping the grasp location. As shown in Fig. 3(b), balance can be maintained by pulling the heavier side closer to the body. We leave adapting the grasp strategy jointly with object properties while following the human affordance to future work.
Masses beyond the RNLE limit. Following RNLE, LI 1 corresponds to the lowest risk, risk increases as LI exceeds 1, and LI > 3 is the highest-risk category. Since the load constant in the RNLE’s multiplicative formula is 23 kg, the 30 kg condition exceeds the lowest-risk range by construction. For the 30 kg conditions, the mean RNLE LI is 2.184, and 7.62% of 30kg conditions exceed 3 among sequences. Although this indicates increased lifting demand, such motions are few, and LI above 3 does not imply that a motion is infeasible, since RNLE estimates the risk of repeated lifting over a work shift. We instead check physical feasibility in simulation under the HS profile. The HS profile does not precisely model the joint strength of an average human, but it is grounded in the literature and provides a fair setting across baselines. Under a 0.8$\times$ HS profile, HOI-UP still achieves the highest success rate, preservation, and property response, indicating that its gains are not specific to the default HS profile (Tab. 19). Even when the motion is feasible, humans may change the strategy at heavy loads, whereas HOI-UP keeps the grasp locations and phase order of the reference. We leave strategy-level adaptation to future work.
Large floor-level objects and six floor-based phases. We agree that the evaluation scope is whole-body manipulation of large OMOMO objects on the floor, segmented into six phases. To quantify this scope, we assigned all action verbs in the OMOMO text annotations (13,230) to phases. 86.0% fall into the six phases, and grasping (2.1%) and non-interaction (2.2%) are also covered by our phase labels (90.3% in total). The remaining 9.6% are out of scope: manipulating objects with the feet (5.5%), pivoting objects on the floor (3.9%), and rotating objects on a table (0.2%). Moreover, phase segmentation is a replaceable component, and the subsequent stages require only the phase boundaries, open-vocabulary labels, and contact states.
Generalization beyond the evaluation scope. We additionally apply HOI-UP to hand-held tools that are manipulated or swung in the air from IMHD$^2$ and HODome, and add four representative results to the App. A.14 (Fig. 19) and Part 3 of the supplementary video. The VLM-based HOI adaptation and kinematic HOI optimization handle open-vocabulary phases beyond the six phases (“kettlebell swinging”, “bat swinging”, “tennis swinging”, and “pan cooking”) and objects absent from OMOMO without additional training, as HOI-UP involves no object-specific training even for OMOMO. In the optimized motions, heavier kettlebells and baseball bats are held lower and move more slowly. When the CoM shifts away from the hand, the tennis racket swings over a smaller range with a lower head speed, and the pan is tossed more gently.
Positioning and Novelty
Sec. 2 already discussed LLM-based motion generation and editing (InterDreamer, CoMo). We additionally cite and discuss FreeMotion and Wu et al., which follow a planning kinematics physics structure. HOI-UP shares a similar structure with these works, but we do not claim this structure as our main contribution. Instead, we propose the task of adapting existing HOI references to unseen physical object properties without property-labeled motion data, together with the stage designs for this task:
- We use the VLM to infer how the reference should change under the target properties, rather than to follow explicit instructions.
- We extend the literature-motivated HOI-codes with interaction codes that describe whole-body support and object-motion states.
- We jointly optimize HOI-code matching with reference preservation, contact, joint load, and balance.
- We propose the HS profile, which approximates human strength limits, and fine-tune the controller under the target object properties and this HS profile.
Central Hypothesis: VLM prior can replace property-labeled motion data
HOI-UP aims to adapt an original HOI to unseen physical object properties. Because property-labeled data are currently limited in variety and scale, this can be viewed as an out-of-distribution problem. We use a VLM prior to perform zero-shot adaptation to object properties. Tab. 1 shows that the VLM prior of HOI-UP yields strong property response (first in Effdir and Effmag, second in R$^2_{\text{all}}$). For human-likeness, our additional user study mentioned below shows that HOI-UP is preferred over the other baselines. However, preferred strategies vary across individuals, and the numbers of subjects in the user study and in FORCE are small. Therefore we will moderate the hypothesis that a VLM prior fully replaces property-labeled data.
Rule-table baseline. We agree that literature-based rules can determine some HOI-codes. For example, RNLE can be used to compute an upper bound on the horizontal human–object distance as a function of mass when the other task conditions are fixed. However, such relations cover only a few codes, mostly for mass, and specify bounds or directions rather than phase-wise target ranges. We found few quantitative relations for CoM offset, object-floor friction, or scale. A rule table would therefore need to specify, for each HOI-code, interaction phase, and object property, the direction and amount of change conditioned on the posture and contacts of the reference. Most of these entries would be the authors’ heuristic choices rather than values derived from the literature, so such a comparison would mainly reflect how the table is designed. Therefore, we use the VLM as a provider of prior knowledge on how HOI-codes should change under object-property adaptation. The VLM predicts quantitative HOI-code changes for the property change while accounting for posture, contacts, and interaction phases of the reference. The VLM comparison supports the role of this reference-specific context: at the kinematic stage, the text-only variant, which does not observe the rendered reference, shows lower Effdir (52.54 vs. 56.70), Effmag (20.11 vs. 22.63), and Mon (42.58 vs. 45.12) than HOI-UP (Tab. 17, App. A.12).
Physics- and task-driven RL. Our task is to adapt a given HOI to new object properties while preserving its intent (trajectory, contacts, and phase order). Task-driven RL without reference tracking instead generates new motions, which would require reward designing for each reference-property pair and could not be evaluated with the same-reference protocol (e.g., Minimality, Comp$_{\text{HOI}},HOIerror).Ourexperimentsincludetheclosestalternativesunderthisevaluationprotocol:InterMimic+ZMPaddsabalancerewardtoRL,andHOI−UP_{\text{NoHOICode}}$ optimizes the motion with contact, balance, and joint-load terms but without the VLM targets. Both show weaker property responses than HOI-UP (Effdir 48.89 and 51.80 vs. 56.47; Effmag 16.54 and 16.24 vs. 21.82; Rall2 17.71 and 12.74 vs. 21.91), indicating that physical and biomechanical objectives alone yield limited property-dependent adaptation (Tab. 1). RL trained with task rewards alone is already known to produce motions that are not human-like (DeepMimic, AMP), so physics-based HOI methods typically rely on motion priors. Even a data-driven prior such as AMP would require property-labeled motion data to learn property-specific styles, which HOI-UP does not need.
Central Hypothesis: Human-likeness evaluation & HOI-Code-based metrics
We agree that our property-response metrics (Eff, Mon, R$^2_{\text{all}}$) measure only whether motions change with object properties and whether these changes are monotonic in the magnitude of the property change. They do not establish whether humans would make the same adaptations, so we conducted two additional analyses.
User study. 16 anonymous participants completed two tasks. (Sec. 5.4, Fig. 6, App. A.11)
Results: HOI-UP was preferred in 67–84% of comparisons and ranked first in ordering accuracy (81%). HOI-UP$_{\text{HOICodeOnly}}$ and InterMimic+HOI-code, which ranked second and third in ordering accuracy, were less preferred than HOI-UP in the comparison task. This suggests that they respond monotonically without accounting for human-likeness.