acceptodds
Under review as a conference paper at ICLR 2027

HOI-UP: Adapting Human-Object Interaction Motion to Unseen Object Properties without Property-Labeled Motion Data

Abstract

How humans interact with an object depends on its physical properties, yet widely used Human-Object Interaction (HOI) datasets provide little explicit supervision for property-dependent adaptation. We aim to adapt an existing HOI reference to unseen object properties without property-labeled motion data. Existing property-aware methods such as FORCE and PA-HOI require additional motion data for new property conditions, while motion editing approaches such as CoMo rely on discrete pose-code editing that compromises the precise contacts required for physical HOI execution. Biomechanical objectives, used as the sole adaptation guidance in motion optimization or physics-based imitation, yield similar motion strategies across object-property levels rather than human-like variations. In this paper, we introduce HOI-UP, a framework for adapting existing HOI references to unseen object properties. A vision-language model (VLM) predicts property-dependent changes to compact, literature-informed and empirically validated HOI-codes. Kinematic optimization adapts the reference toward these targets in an anatomically plausible HOI motion space. RL-based tracking then refines the resulting plans in physics simulation for execution under the target properties. We further introduce an evaluation protocol that jointly assesses property response, reference preservation, naturalness, biomechanical demand, and physical executability across object property changes. Our motions adapted to unseen properties follow human-like posture changes and vary monotonically along the property axis, while achieving the highest tracking success rate and the lowest tracking errors under constrained actuator powers.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
human-object interaction, human motion generation, motion editing and adaptation, vision-language model prior, zero-shot adaptation, physics-based character control

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MiiVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adapts an existing human-object interaction motion to unseen object properties, such as mass, scale, center of mass and friction, without property-labeled motion data. HOI-UP describes the reference with 17 interpretable HOI-codes per interaction phase, asks a VLM (Claude Opus 5) for phase-wise target ranges of these codes under the new properties, and realizes them by optimizing spline corrections to the reference under contact, balance and joint-load terms. A fine-tuned InterMimic controller then tracks the plan in Isaac Gym under the target properties, serving as both refinement and executability test. An evaluation protocol compares adaptations of the same reference across property levels without ground truth. On 30 OMOMO references with 240 variations, HOI-UP achieves the highest tracking success, contact preservation and property response among the compared baselines.

Critical strengths
  • Formulates a clear and practically motivated task, adapting an existing property-agnostic HOI reference to unseen object properties without property-labeled data, with a coherent pipeline whose components each answer a stated sub-problem.

  • Introduces an evaluation protocol for adaptation without ground truth, combining property response, reference preservation, naturalness, biomechanical demand and executability, together with a human-strength-informed actuation profile validated against literature joint moments.

  • Reports consistent gains in executability and contact preservation over controller- and motion-adaptation baselines under a shared actuation profile, with kinematic and vocabulary ablations and a candid limitations section.

Critical weaknesses
  • The central hypothesis is never tested. The paper rests on the claim that a VLM prior can replace property-labeled motion data and yield human-like adaptation, but no experiment checks either half. Adaptations are never compared with real motions captured under changed properties, although FORCE contains exactly such paired data and is used only to extract slope directions. The one metric grounded in human data, Eff, compares only the sign of each change, not its magnitude, and the three controller-adaptation baselines score higher on it than HOI-UP. Whether the VLM is needed at all is untested, since a rule table derived from the same biomechanics literature that selected the 17 codes could produce the same phase-wise ranges and is not compared. Whether a human prior is needed is also untested, since no physics-driven alternative is run, such as task-driven RL with an effort or balance cost that does not track the original reference. The claim that biomechanical objectives alone collapse strategies across property levels is asserted without an experiment.

  • The evaluation favors the method by construction. Minimality, Eff and Mon are computed on the 12 literature codes, which are a subset of the 17 codes the method optimizes directly, while the controller-adaptation baselines never optimize them. Only R² uses independent features. The motion-adaptation baseline CoMo-HOI predicts its edits with a text-only Qwen2.5-7B, whereas HOI-UP uses Claude Opus 5 with rendered frames, so the gap in Table 1 mixes method and model. The plan-level and simulation-level results are reported on different sequence sets, so the effect of tracking on the adaptation cannot be read off.

  • The scope is narrower than stated and partly exceeds what the method can do. Hand positions are fixed object-local anchors, so the method cannot change where the object is held, which directly limits the center-of-mass axis. All 30 references, whose default masses range from 0.51 to 9.82 kg, are adapted to 10, 20 and 30 kg, so a 0.5 kg trashcan is adapted to 30 kg, beyond the lifting limit referenced by the paper’s own RNLE metric, where a human would change strategy rather than posture. Objects are large floor-level items and interactions are six floor-based phases, while the title and abstract claim human-object interaction in general. None of this appears in the limitations.

  • Positioning and novelty. The pipeline, a foundation model producing a structured high-level specification, a kinematic step filling it in, and a physics tracker refining it, is the structure of MLLM-driven mocap-free motion synthesis, which is not cited. Within this structure the VLM contributes only phase-wise ranges for 17 scalars, the physics simulator is used to score plans and never to inform them, and there is no feedback from execution to the plan.

Questions for authors
  1. No code, prompts or reproducibility statement are provided, and the VLM is a closed API. Will the code, the full prompt, the 240 VLM outputs, the optimization settings and the list of the 30 sequences be released?

  2. How is the VLM queried? Temperature, number of samples per pair, variance across repeated queries, and the fraction of code ranges left at the reference value due to low confidence. What changes with text-only input and without the rendered frames?

  3. What do a literature rule table for the target ranges and a physics-driven baseline that does not track the original reference achieve under the same protocol?

  4. FORCE contains motions under several resistance levels. How do adapted motions compare with the real higher-resistance motions in both direction and magnitude of the 17 codes?

  5. Why was CoMo-HOI driven by a text-only Qwen2.5-7B rather than the same VLM, and how was the LLM-implemented baseline validated?

8 comments

Central Hypothesis: VLM prior can replace property-labeled motion data

HOI-UP aims to adapt an original HOI to unseen physical object properties. Because property-labeled data are currently limited in variety and scale, this can be viewed as an out-of-distribution problem. We use a VLM prior to perform zero-shot adaptation to object properties. Tab. 1 shows that the VLM prior of HOI-UP yields strong property response (first in Effdir\text{Eff}_{\text{dir}} and Effmag\text{Eff}_{\text{mag}}, second in R$^2_{\text{all}}$). For human-likeness, our additional user study mentioned below shows that HOI-UP is preferred over the other baselines. However, preferred strategies vary across individuals, and the numbers of subjects in the user study and in FORCE are small. Therefore we will moderate the hypothesis that a VLM prior fully replaces property-labeled data.

Rule-table baseline. We agree that literature-based rules can determine some HOI-codes. For example, RNLE can be used to compute an upper bound on the horizontal human–object distance as a function of mass when the other task conditions are fixed. However, such relations cover only a few codes, mostly for mass, and specify bounds or directions rather than phase-wise target ranges. We found few quantitative relations for CoM offset, object-floor friction, or scale. A rule table would therefore need to specify, for each HOI-code, interaction phase, and object property, the direction and amount of change conditioned on the posture and contacts of the reference. Most of these entries would be the authors’ heuristic choices rather than values derived from the literature, so such a comparison would mainly reflect how the table is designed. Therefore, we use the VLM as a provider of prior knowledge on how HOI-codes should change under object-property adaptation. The VLM predicts quantitative HOI-code changes for the property change while accounting for posture, contacts, and interaction phases of the reference. The VLM comparison supports the role of this reference-specific context: at the kinematic stage, the text-only variant, which does not observe the rendered reference, shows lower Effdir\text{Eff}_{\text{dir}} (52.54 vs. 56.70), Effmag\text{Eff}_{\text{mag}} (20.11 vs. 22.63), and Mon (42.58 vs. 45.12) than HOI-UP (Tab. 17, App. A.12).

Physics- and task-driven RL. Our task is to adapt a given HOI to new object properties while preserving its intent (trajectory, contacts, and phase order). Task-driven RL without reference tracking instead generates new motions, which would require reward designing for each reference-property pair and could not be evaluated with the same-reference protocol (e.g., Minimality, Comp$_{\text{HOI}},HOIerror).Ourexperimentsincludetheclosestalternativesunderthisevaluationprotocol:InterMimic+ZMPaddsabalancerewardtoRL,andHOI−UP, HOI error). Our experiments include the closest alternatives under this evaluation protocol: InterMimic+ZMP adds a balance reward to RL, and HOI-UP_{\text{NoHOICode}}$ optimizes the motion with contact, balance, and joint-load terms but without the VLM targets. Both show weaker property responses than HOI-UP (Effdir\text{Eff}_{\text{dir}} 48.89 and 51.80 vs. 56.47; Effmag\text{Eff}_{\text{mag}} 16.54 and 16.24 vs. 21.82; Rall2R^2_{\text{all}} 17.71 and 12.74 vs. 21.91), indicating that physical and biomechanical objectives alone yield limited property-dependent adaptation (Tab. 1). RL trained with task rewards alone is already known to produce motions that are not human-like (DeepMimic, AMP), so physics-based HOI methods typically rely on motion priors. Even a data-driven prior such as AMP would require property-labeled motion data to learn property-specific styles, which HOI-UP does not need.

Central Hypothesis: Human-likeness evaluation & HOI-Code-based metrics

We agree that our property-response metrics (Eff, Mon, R$^2_{\text{all}}$) measure only whether motions change with object properties and whether these changes are monotonic in the magnitude of the property change. They do not establish whether humans would make the same adaptations, so we conducted two additional analyses.

  1. User study. 16 anonymous participants completed two tasks. (Sec. 5.4, Fig. 6, App. A.11)

    • Comparison task (15 questions): given HOI-UP and a random baseline, participants selected the motion that responds more clearly to the property change and the motion that adapts in a more human-like manner.
    • Ordering task (20 questions): participants reordered shuffled motions of one method at different levels of one property.

    Results: HOI-UP was preferred in 67–84% of comparisons and ranked first in ordering accuracy (81%). HOI-UP$_{\text{HOICodeOnly}}$ and InterMimic+HOI-code, which ranked second and third in ordering accuracy, were less preferred than HOI-UP in the comparison task. This suggests that they respond monotonically without accounting for human-likeness.

  1. Revised Eff metric. Eff compares mass-dependent motion-feature trends with FORCE, a mass-conditioned HOI motion-capture dataset. We made three changes to this metric.

    • Filtering failed trials from FORCE. The submitted version also used failed trials in which the subject fails to lift a heavy object. The revised version uses only successful captures.
    • Fair feature selection. Comparing HOI-code trends disadvantages InterMimic-style baselines, which do not use these codes. We therefore compare changes in motion features that none of the methods including HOI-UP uses.
    • Direction- and magnitude-aware comparison. Whereas the submitted version measured only the agreement in the direction of feature changes, the revised version compares both the direction (Effdir\text{Eff}_{\text{dir}}) and the magnitude (Effmag\text{Eff}_{\text{mag}}) of feature changes with FORCE.

    Results: Compared with the successful FORCE captures, HOI-UP achieves the highest agreement in both the direction (Effdir\text{Eff}_{\text{dir}}) and the magnitude (Effmag\text{Eff}_{\text{mag}}) of mass-dependent motion-feature changes. This indicates that HOI-UP yields adaptations closer to real captures than the other methods, without using captured property-labeled data.

Nevertheless, both the user study and the FORCE comparison can depend on subject preferences, and human adaptation to object-property changes can be multimodal. Therefore, we moderate our claims on human-likeness.

Contribution of HOI-code adaptation

We agree that the contribution of HOI-code matching was reported only at the kinematic stage through the No HOI-code ablation (Tab. 2), leaving its contribution to the end-to-end results unclear. To address this weakness, we added HOI-UP$_{\text{NoHOICode}}$ to Tab. 1 (App. A.9.5). It removes only the HOI-code objective from kinematic HOI optimization and uses the same RL fine-tuning and evaluation as HOI-UP.

HOI-UP$_{\text{NoHOICode}}$ shows lower physical executability (success rate, completion rate, HOI error) and naturalness (Jit$_{\text{HOI}},foot−floorerror,contactF1)thanHOI−UP.WithoutHOI−codeadaptation,itobtainsbetterRNLELIandC, foot-floor error, contact F1) than HOI-UP. Without HOI-code adaptation, it obtains better RNLE LI and C_{\text{100}}$ but lower Eff and R$^2_{\text{all}}.Inconclusion,withoutHOI−codes,theplanscollapsetosimilarmotionsthatoptimizebalance,contact,andjointloadinguniformlyregardlessofobjectproperties(thelowestR. In conclusion, without HOI-codes, the plans collapse to similar motions that optimize balance, contact, and joint loading uniformly regardless of object properties (the lowest R^2_{\text{all}}$). This confirms that the VLM provides the prior knowledge required for object-property-dependent adaptations through HOI-code adaptation.

Confounding the method with the LLM in the CoMo-HOI comparison

We agree that the previous comparison confounded the method with the model. To disentangle them, we re-ran CoMo-HOI with the same Claude Opus 5 (effort high) as HOI-UP, keeping the text-only input of the original CoMo design.

With Claude Opus 5, the success rate of CoMo-HOI improves from 23.25% to 40.42%, but it remains far below that of HOI-UP (82.75%). CoMo-HOI (Claude Opus 5) also shows lower direction (Effdir\text{Eff}_{\text{dir}}) and magnitude (Effmag\text{Eff}_{\text{mag}}) agreement with FORCE than HOI-UP. Moreover, even with text-only input, HOI-UP outperforms CoMo-HOI at the kinematic stage in contact F1 (0.734 vs. 0.646), Comp$_{\text{HOI}}$ (8.69 vs. 33.41), Mon (42.58 vs. 42.14), and Rall2R^2_{\text{all}} (19.70 vs. 11.18). Upgrading the foundation model alone therefore does not close the gap, which comes from the method rather than the VLM model.

Different evaluation sets in Tab. 1 and Tab. 2

We recompute Tab. 2 on 270 conditions: the 240 conditions of Tab. 1 and 30 additional conditions with unchanged object properties.

The results show that the two ablations each achieve the best values on one side at a large cost on the other. No HOI-code adaptation optimizes only hand contact, foot contact, and balance, and thus achieves the best naturalness and biomechanical demand (foot-floor error, Mˉ\bar{M}, C$_{\text{100}}$). However, its Effmag\text{Eff}_{\text{mag}} is the lowest among all methods, indicating a weak response to property changes. Conversely, HOI-code Only follows only the VLM-predicted HOI-code targets and attains the best preservation, Mon, and R$^2_{\text{all}}$. However, its contact F1 drops to 0.733 and its foot-floor error is the largest, making its plans difficult to execute. HOI-UP ranks first in Effmag\text{Eff}_{\text{mag}} and second in Effdir\text{Eff}_{\text{dir}}, Jit$_{\text{HOI}}$, foot-floor error, Mˉ\bar{M}, and Comp$_{\text{HOI}}$. This suggests that HOI-UP produces kinematic plans that satisfy both property response and physical plausibility. This balance carries over after RL fine-tuning, where HOI-UP achieves the highest success rate and contact F1 and the lowest foot-floor error (Tab. 1). That is, the balance between property response and executability, rather than the best value on any single kinematic metric, determines the final performance.

Limitations

Fixed hand positions. Our research scope in this work is how the pose of a given reference adapts when object properties change. For example, when the CoM changes, one can respond by changing the grasp location, but also by modifying the arm and body pose while keeping the grasp location. As shown in Fig. 3(b), balance can be maintained by pulling the heavier side closer to the body. We leave adapting the grasp strategy jointly with object properties while following the human affordance to future work.

Masses beyond the RNLE limit. Following RNLE, LI ≤\leq 1 corresponds to the lowest risk, risk increases as LI exceeds 1, and LI > 3 is the highest-risk category. Since the load constant in the RNLE’s multiplicative formula is 23 kg, the 30 kg condition exceeds the lowest-risk range by construction. For the 30 kg conditions, the mean RNLE LI is 2.184, and 7.62% of 30kg conditions exceed 3 among sequences. Although this indicates increased lifting demand, such motions are few, and LI above 3 does not imply that a motion is infeasible, since RNLE estimates the risk of repeated lifting over a work shift. We instead check physical feasibility in simulation under the HS profile. The HS profile does not precisely model the joint strength of an average human, but it is grounded in the literature and provides a fair setting across baselines. Under a 0.8$\times$ HS profile, HOI-UP still achieves the highest success rate, preservation, and property response, indicating that its gains are not specific to the default HS profile (Tab. 19). Even when the motion is feasible, humans may change the strategy at heavy loads, whereas HOI-UP keeps the grasp locations and phase order of the reference. We leave strategy-level adaptation to future work.

Large floor-level objects and six floor-based phases. We agree that the evaluation scope is whole-body manipulation of large OMOMO objects on the floor, segmented into six phases. To quantify this scope, we assigned all action verbs in the OMOMO text annotations (13,230) to phases. 86.0% fall into the six phases, and grasping (2.1%) and non-interaction (2.2%) are also covered by our phase labels (90.3% in total). The remaining 9.6% are out of scope: manipulating objects with the feet (5.5%), pivoting objects on the floor (3.9%), and rotating objects on a table (0.2%). Moreover, phase segmentation is a replaceable component, and the subsequent stages require only the phase boundaries, open-vocabulary labels, and contact states.

Generalization beyond the evaluation scope. We additionally apply HOI-UP to hand-held tools that are manipulated or swung in the air from IMHD$^2$ and HODome, and add four representative results to the App. A.14 (Fig. 19) and Part 3 of the supplementary video. The VLM-based HOI adaptation and kinematic HOI optimization handle open-vocabulary phases beyond the six phases (“kettlebell swinging”, “bat swinging”, “tennis swinging”, and “pan cooking”) and objects absent from OMOMO without additional training, as HOI-UP involves no object-specific training even for OMOMO. In the optimized motions, heavier kettlebells and baseball bats are held lower and move more slowly. When the CoM shifts away from the hand, the tennis racket swings over a smaller range with a lower head speed, and the pan is tossed more gently.

Positioning and Novelty

Sec. 2 already discussed LLM-based motion generation and editing (InterDreamer, CoMo). We additionally cite and discuss FreeMotion and Wu et al., which follow a planning →\rightarrow kinematics →\rightarrow physics structure. HOI-UP shares a similar structure with these works, but we do not claim this structure as our main contribution. Instead, we propose the task of adapting existing HOI references to unseen physical object properties without property-labeled motion data, together with the stage designs for this task:

  • We use the VLM to infer how the reference should change under the target properties, rather than to follow explicit instructions.
  • We extend the literature-motivated HOI-codes with interaction codes that describe whole-body support and object-motion states.
  • We jointly optimize HOI-code matching with reference preservation, contact, joint load, and balance.
  • We propose the HS profile, which approximates human strength limits, and fine-tune the controller under the target object properties and this HS profile.
Reviewer_hySjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HOI-UP, a framework for adapting existing human-object interaction (HOI) motions to changes in object mass, scale, center of mass, and object-floor friction without property-labeled motion supervision. The method consists of three stages. First, a VLM predicts target ranges for phase-wise HOI-codes based on the reference motion and property changes. Kinematic optimization then adjusts the motion to satisfy these targets while considering contact, balance, biomechanical constraints, and reference preservation. Finally, an InterMimic controller is fine-tuned under the target object properties to improve physical execution. The authors evaluate the method on 30 reference sequences and 240 reference-property conditions, measuring execution success, contact quality, motion preservation, and property response. Additional ablations examine the role of HOI-codes and the kinematic optimization objectives.

Critical strengths
  1. The framework provides a way to adapt existing HOI motions to different object properties without matched demonstrations. I find the separation between property adaptation and physically constrained motion optimization useful. The method uses phase-wise HOI-code targets while accounting for contact and balance, rather than relying on the predicted targets alone. This allows existing motion references to be reused under different physical conditions.

  2. The experiments show improvements in execution and contact quality, with useful supporting ablations. In Table 1, aggregate success increases from 78.00% for InterMimic+ZMP to 82.75%, while contact F1 improves from 0.889 to 0.916. The shared actuation and training settings make this comparison meaningful. Table 2 also shows that removing HOI-code matching weakens property response, while optimizing only the codes degrades contact quality. These results support combining adaptation targets with interaction constraints.

Critical weaknesses

The contribution of HOI-code matching to the final results is not fully isolated. Table 2 includes a useful No HOI-code ablation, showing that code matching improves property response at the kinematic planning stage. However, this variant is not evaluated after the RL fine-tuning used in Table 1. It is therefore unclear how much of the final improvement comes from code matching rather than the other optimization terms and controller adaptation.

Neither InterMimic+ZMP nor HOI-code Only provides this control, since they change other parts of the pipeline. Evaluating No HOI-code with the same controller fine-tuning would help establish whether the planning-stage improvements carry over to final execution and how much they contribute to the reported performance.

Questions for authors

What happens to the No HOI-code ablation if it is taken through the same RL fine-tuning as the full method? In particular, do the differences in execution success, contact quality, and property response persist?

1 comment

Contribution of HOI-code on end-to-end not fully isolated

We agree that the contribution of HOI-code matching was reported only at the kinematic stage through the No HOI-code ablation (Tab. 2), leaving its contribution to the end-to-end results unclear. To address this weakness, we added HOI-UP$_{\text{NoHOICode}}$ to Tab. 1 (App. A.9.5). Its plans are optimized excluding only the HOI-code objective from kinematic HOI optimization, and are executed with the same RL fine-tuning and evaluation setting as HOI-UP (240 pairs × 5 rollouts).

Table 1: End-to-end comparison under various target properties. (a) Controller adaptation and (b) Motion adaptation followed by InterMimic fine-tuning. All methods share the same HS profile.

MethodSuccess (%) ↑Completion (%) ↑HOI error (mm) ↓Jit$_{\text{HOI}}$ (m s⁻³) ↓Foot-floor (cm) ↓Contact F1 ↑τˉsim\bar{\tau}_{\text{sim}} (N m) ↓C100C_{100} (joints) ↓OWAS ↓RNLE LI ↓Minimality ↓Effdir\text{Eff}_{\text{dir}} (%) ↑Effmag\text{Eff}_{\text{mag}} (%) ↑Mon (%) ↑Rall2R^2_{\text{all}} (%) ↑
(a) Controller adaptation
InterMimic77.4287.6979.0709.210.3550.890‾\underline{\text{0.890}}12.782.9412.1190.9030.64944.7914.4937.4514.66
InterMimic+ZMP78.0087.3081.8698.420.2890.88912.442.9302.0970.8830.65948.8916.5438.8117.71
InterMimic + HOI-code73.8385.5685.3743.110.4150.86613.612.9582.116‾\underline{\text{2.116}}0.8910.65047.9215.0639.8315.86
(b) Motion adaptation
CoMo-HOI40.4260.7595.4743.900.3370.78015.182.9422.2050.816‾\underline{\text{0.816}}0.62152.8021.01‾\underline{\text{21.01}}40.7025.66
HOI-UP$_{\text{HOICodeOnly}}$42.8365.74104.1768.000.5330.82514.993.0022.1950.8480.52555.55‾\underline{\text{55.55}}21.01‾\underline{\text{21.01}}42.9021.63
HOI-UP$_{\text{NoHOICode}}$78.50‾\underline{\text{78.50}}88.00‾\underline{\text{88.00}}78.0‾\underline{\text{78.0}}706.150.218‾\underline{\text{0.218}}0.890‾\underline{\text{0.890}}12.772.8012.2100.7980.560‾\underline{\text{0.560}}51.8016.2442.41‾\underline{\text{42.41}}12.74
HOI-UP (ours)82.7591.5376.7702.60‾\underline{\text{702.60}}0.1850.91612.69‾\underline{\text{12.69}}2.865‾\underline{\text{2.865}}2.1700.8340.58556.4721.8241.1021.91‾\underline{\text{21.91}}

HOI-UP$_{\text{NoHOICode}}$ shows lower physical executability (success rate, completion rate, HOI error) and naturalness (Jit$_{\text{HOI}},foot−floorerror,contactF1)thanHOI−UP.Moreover,becauseitoptimizesonlythebalance,contact,andloadobjectiveswithoutadaptingHOI−codestothegivenobjectproperties,itobtainsbetterRNLELIandC, foot-floor error, contact F1) than HOI-UP. Moreover, because it optimizes only the balance, contact, and load objectives without adapting HOI-codes to the given object properties, it obtains better RNLE LI and C_{\text{100}}$ scores, but its property response in Eff and R$^2_{\text{all}}$ is lower.

Table 2: Kinematic ablation before RL fine-tuning. (only groups (d) and (e) of Table 2 are shown)

MethodJit$_{\text{HOI}}$ (m s⁻³) ↓Foot-floor error, mesh (cm) ↓Contact F1 ↑Mˉ\bar{M} (N m) ↓C100C_{100} (joints) ↓OWAS mean ↓RNLE LI valid range ↓Minimality ↓Comp$_{\text{HOI}}$ (cm) ↓Effdir\text{Eff}_{\text{dir}} (%) ↑Effmag\text{Eff}_{\text{mag}} (%) ↑Mon (%) ↑Rall2R^2_{\text{all}} (%) ↑
(d) Optimization objectives
No HOI-code adaptation61.730.5380.84867.902.4202.0450.7800.441‾\underline{\text{0.441}}8.69051.2417.3548.70‾\underline{\text{48.70}}14.47
HOI-code only70.802.6430.73374.362.7582.041‾\underline{\text{2.041}}0.8520.2995.55657.2621.4951.5623.36
(e) Ours
Ours (L+IL + I)61.86‾\underline{\text{61.86}}1.250‾\underline{\text{1.250}}0.82369.46‾\underline{\text{69.46}}2.4432.0560.8430.5598.684‾\underline{\text{8.684}}56.70‾\underline{\text{56.70}}22.6345.1215.98

In conclusion, without HOI-codes, the plans collapse to similar motions that optimize balance, contact, and joint loading uniformly regardless of object properties (the lowest R$^2_{\text{all}}$ in Tab. 1). The gap in property response at the kinematic stage (Tab. 2: Effmag\text{Eff}_{\text{mag}} 17.35 vs. 22.63) persists after RL fine-tuning (Tab. 1: 16.24 vs. 21.82). This confirms that the VLM provides the prior knowledge required for object-property-dependent adaptations through HOI-code adaptation.

Reviewer_vHRiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces HOI-UP, which adapts existing human-object interaction motions to changes in mass, scale, center of mass, and friction without capturing the motion adapting such change. To achieve this, a VLM predicts phase-wise changes to continuous interaction codes, then a constrained optimization realizes these changes while preserving contacts and limiting biomechanical demand. With the reference ready, RL fine-tuning on InterMimic produces simulated executions under physics. Contributions include this adaptation framework, an interaction-specific code representation, and a multidimensional evaluation protocol. Experiment validate the controller and highligh motion-editing comparisons, kinematic ablations, and qualitative examples.

Critical strengths

Integrating semantic adaptation with physical constraints and biomechanical guidance is a promising direction for human-object interaction. Compared with related work focused on compliant locomotion, this paper’s emphasis on object manipulation is particularly interesting.

The demonstration video shows some behavioral adaptations consistent with biomechanical expectations. However, the results are uneven across examples: some adaptations are subtle, and some are accompanied by noticeable degradation in motion quality.

Critical weaknesses

Scalability and downstream generalization are insufficiently evaluated. policy training is explicitly part of the proposed synthesis task; these might limit the evidence for practical, large-scale motion augmentation.

  • The method is performed on only 30 reference motions (compared to InterMimic on thousands), limiting confidence in its robustness across a broader range of human-object interactions.
  • The complete pipeline requires motion optimization and substantial target-condition controller fine-tuning and runtime, GPU-hours, are unreported for certain amount of data synthesis

The metrics for measuring the plausible editing is reasonable, but some metrics check whether motions change consistently as object properties vary, but do not establish a more concrete sense of whether humans would make the same changes.

Questions for authors

Given that CodeOnly retains InterMimic’s rewards, including contact and interaction rewards, what explains its low success rate?

In the first demonstration, the CodeOnly rollout appears to begin in an already failed configuration. In the second, the hand does not approach the appropriate grasp region. Since the controller is fine-tuned on the kinematic optimized target motions, it is unclear why the reference rewards do not help correct this behavior or why the introduced reward in HOI-UP can correct this.

These examples raise questions about the consistency of the initialization, motion references, and contact annotations used for training. Could the authors clarify how these are constructed and whether its failures arise from physically infeasible targets, conflicting reward objectives, or training and implementation issues?

5 comments

W1. Scalability and Downstream Generalization

We evaluate HOI-UP on 30 reference motions from OMOMO, yielding 270 reference–property pairs (including 30 identity pairs with unchanged properties), and execute 240 pairs with 5 rollouts in physics simulation to measure physical executability. The 30 references cover diverse objects and interaction scenarios in OMOMO and thus form a representative evaluation set (Sec. 5.1, App. A.8). This evaluation scale is comparable to that of zero-/few-shot HOI studies, and is larger when we consider the number of reference–property pairs.

MethodEvaluation ScalePhysical Object Properties
FORCE (3DV'25)30 seqsresistance 3–6 levels
AssistMimic (CVPR'26)40 clips (Inter-X 30 + HHI-Assist 10)Recipient mass ×1.2/×1.5, hip torque ×0.5, pd gain ×0.5
SplitAdapter (arXiv'26)9 conditions × 10 trialsmass 2/4/6 kg
ZeroHSI (3DV'26)20 instances × 5 seeds–
HOI-PAGE (ICML'26)21 prompts–
GenZI (CVPR'24)38 actions in 8 scenes + PROX-S 150 combinations–
InteractAnything (CVPR'25)5 prompts × 10 object categories–
CoMo (ECCV'24)20 scenarios for user study–
HOI-UP (Ours)30 reference seqs (270 pairs for kinematic eval, 240 pairs × 5 rollouts for physics-based eval)mass, scale, CoM drift, object-floor friction

The RL fine-tuning stage of HOI-UP does not train a separate controller for each reference–property pair. Instead, starting from a common HS profile checkpoint, a single policy is fine-tuned on all 240 pairs (App. A.6). HOI-UP thus forms a scalable pipeline that optimizes each reference–property condition independently and trains multiple property conditions jointly in a single policy. No per-condition policy is required, and the independent motion optimizations can be parallelized. For larger-scale training, distillation as in InterMimic or new training schemes can be adopted, and we leave such extensions and large-scale generalization studies to future work.

Such future works would enable diverse downstream applications. Under the same conditions, HOI-UP achieves a higher success rate than the InterMimic policy (82.75% vs. 77.42% in Tab. 1) and stronger property responses (Effdir\text{Eff}_\text{dir} 56.47 vs. 44.79 and Rall2R^2_\text{all} 21.91 vs. 14.66). This indicates that HOI-UP is a more reliable approach for synthesizing property-adapted HOI motions at scale. We expect such synthesized motions to support future work such as training property-conditioned HOI generators or regressors that estimate object properties from HOI motion.

W2. Computation Costs

We report the computational cost of each component below. We used two RTX 4090 and two A100 GPUs, running two optimizations and two RL fine-tuning jobs in parallel.

  • Phase tagging: 0.6 s per reference sequence (19 s for 30 sequences)
  • VLM-based HOI adaptation: 30–60 s per reference–property pair (10–20k input and 2–4k output tokens; median 2.9k output tokens).
    • The time and number of tokens vary substantially with the length and the number of phases of the reference HOI.
  • Kinematic HOI optimization: 5.8 min per reference–property pair (1× RTX 4090)
  • RL fine-tuning: 28 h for 240 reference–property pairs (1× A100 80GB)

Kinematic HOI optimization is performed independently for each reference–property pair and can be parallelized across pairs, while a single controller is fine-tuned across all the target conditions using the resulting motion plans. This structure supports scaling motion augmentation to larger collections of reference motions and target object properties.

W3. Human-likeness

We agree that our property-response metrics (Eff, Mon, Rall2R^2_{\text{all}}) measure only whether motions change with object properties and whether these changes are monotonic in the magnitude of the property change. They do not establish whether humans would make the same adaptations, so we conducted two additional analyses.

  1. User study. 16 anonymous participants completed two tasks. (Sec. 5.4, Fig. 6, App. A.11)

    • Comparison task (15 questions): given HOI-UP and a randomly selected baseline, participants selected the motion that responds more clearly to the property change and the motion that adapts in a more human-like manner.
    • Ordering task (20 questions): motions generated by a randomly selected method at different levels of one property were shuffled, and participants reordered them by level.

    Results: HOI-UP was preferred in 67–84% of comparisons and ranked first in ordering accuracy (81%). HOI-UP$_{\text{HOICodeOnly}}$ and InterMimic+HOI-code, which ranked second and third in ordering accuracy, were less preferred than HOI-UP in the comparison task. This suggests that they respond to the changes and show monotonicity in the change magnitude, without accounting for human-likeness.

User study (Sec. 5.4, Fig. 6, App. A.11), comparison task: share of answers (%) when HOI-UP is compared with each baseline.

HOI-UP vs.Clearer responseMore human-like
HOI-UPsimilarbaselineHOI-UPsimilarbaseline
InterMimic72.922.94.272.922.94.2
InterMimic+ZMP77.116.76.272.920.86.2
InterMimic+HOI-code83.310.46.281.212.56.2
CoMo-HOI79.19.311.683.79.37.0
HOI-UP$_{\text{HOICodeOnly}}$66.716.716.773.316.710.0

User study, ordering task: accuracy of reordering shuffled motions of one method by property level (%).

MethodOrdering accuracy (%) ↑
InterMimic61.2
InterMimic+ZMP62.6
InterMimic+HOI-code77.5
CoMo-HOI60.3
HOI-UP$_{\text{HOICodeOnly}}$78.1‾\underline{\text{78.1}}
HOI-UP (ours)80.7
  1. Revised Eff metric. Eff compares the trends of motion-feature changes with increasing object mass against FORCE which is a mass-conditioned HOI motion-capture dataset. For the rebuttal, we made three changes to this metric.

    • Filtering failed trials from FORCE. The submitted version compared motion features against all FORCE captures, including failed trials in which the subject attempts but fails to lift a heavy object. The revised version uses only captures labeled as successful.
    • Fair feature selection. Comparing HOI-code trends disadvantages InterMimic-style baselines, which do not use these codes. We therefore compare changes in motion features that none of the methods including HOI-UP uses.
    • Direction- and magnitude-aware comparison. Whereas the submitted version measured only the agreement in the direction of feature changes, the revised version compares both the direction (Effdir\text{Eff}_{\text{dir}}) and the magnitude (Effmag\text{Eff}_{\text{mag}}) of feature changes with FORCE.

    Results: Compared with the successful FORCE captures, HOI-UP achieves the highest agreement in both the direction (Effdir\text{Eff}_{\text{dir}}) and the magnitude (Effmag\text{Eff}_{\text{mag}}) of mass-dependent motion-feature changes. This indicates that HOI-UP yields adaptations closer to real captures than the other methods, without using captured property-labeled data.

Nevertheless, both the user study and the FORCE comparison can depend on subject preferences, and human adaptation to object-property changes can be multimodal. Therefore, we moderate our claims on human-likeness.

Q1. Why HOICodeOnly fails, and what HOI-UP fixes

HOI-UP$_{\text{HOICodeOnly}}$ is identical to HOI-UP in the RL fine-tuning stage (fine-tuning procedure, rewards, initialization rule, and HS profile), and it differs only in that kinematic HOI optimization uses the HOI-code objective alone, without the contact, biomechanical, and regularization objectives (App. A.9.4). Therefore, its failure comes not from implementation issues in RL fine-tuning, but from kinematic plans optimized with the HOI-code objective alone being difficult to track.

For example, optimizing only the HOI-codes does not guarantee that the original hand–object contacts are preserved. Indeed, contact F1 already drops from 0.823 to 0.733 at the kinematic stage (Tab. 2). In addition, when knee flexion is required, the optimization can produce a floating posture that bends only the knees instead of a natural squat. Indeed, HOI-UP$_{\text{HOICodeOnly}}$ has the largest foot-floor error (0.533 cm; Tab. 1). The resulting conflict between pose tracking and contact rewards can further hinder RL training.

HOI-UP does not add rewards, and instead, it corrects the target motion that the policy tracks. Kinematic HOI optimization adjusts HOI-code changes jointly with contact, support, and joint load, so that the motion is adapted according to the VLM prior while preserving the hand–object contacts and balance of the original motion. Such a plan, with accurate adaptation, contact, and balance, is executable and contributes substantially to RL training and the success rate. In contrast, HOI-UP$_{\text{HOICodeOnly}}$ is optimized into plans that are difficult or impossible to track, which can result in rollouts that fail from its initialization, as observed in the demonstration video.

Q2. Consistency of the initialization, motion references, and contact annotations

  • All numbers in the tables are evaluated with the same initialization rule for all methods. Each method tracks its own motion, and each rollout is initialized from the first frame of that motion: the original reference for InterMimic, InterMimic+ZMP, and InterMimic+HOI-code, and the method’s own generated or optimized plan for CoMo-HOI, HOI-UP$_{\text{HOICodeOnly}},HOI−UP, HOI-UP_{\text{NoHOICode}}$, and HOI-UP. Therefore, the initial states of the latter methods can differ from that of the original reference.
  • The baselines fall into two groups. Controller adaptation (InterMimic+ZMP and InterMimic+HOI-code) keeps the reference motion unchanged and modifies only the RL rewards. Motion adaptation (CoMo-HOI, HOI-UP$_{\text{HOICodeOnly}},HOI−UP, HOI-UP_{\text{NoHOICode}}$, HOI-UP) keeps the InterMimic RL setup fixed and changes only the motion to be tracked.
  • RL fine-tuning setup: all methods share the same stage-1 HS checkpoint, PPO configuration, reset and termination rules, and evaluation rollouts. Controller adaptation only adds its own reward terms, and the motion-adaptation methods use identical reward settings.
  • Contact annotations: all methods use those of the original reference. HOI-UP$_{\text{HOICodeOnly}}$ and HOI-UP apply global temporal rescaling during optimization and rescale the contact annotations by the same ratio, so the contact intervals of the original motion remain valid.
  • Evaluation and video: we have revised the supplementary video to describe the initialization and fine-tuning settings of each method, to avoid misunderstandings about fairness across methods.

Comments on the paper

1 comment
AuthorsOpenReview ↗

We thank you for your thorough and constructive review.

HOI-UP adapts property-agnostic HOI references to unseen physical object properties without property-labeled motion data.

To clarify the problem setting, "without property-labeled motion data" does not mean that we use unlabeled motions of objects with diverse properties. Instead, it means that no property-specific motion data are used for either adaptation or training.

  • Existing property-specific motion data (FORCE, PA-HOI) are limited in variety and scale, and we use them neither for learning nor for retrieval.
  • RL fine-tuning converts the intermediate output of the pipeline into a physically executable final motion, and it uses no property-specific motion data other than our own intermediate outputs.

In this sense, HOI-UP adapts to new object properties in a zero-shot manner, without property-specific motion data.

Revised property-response metrics. In the submitted version, Min, Eff, and Mon were computed on the 12 literature-motivated codes (L), which are a subset of the HOI-codes that HOI-UP directly optimizes. To avoid favoring HOI-UP by construction, we recompute these three metrics on the motion features independent to our HOI-codes (pelvis/heading-relative pose, object corners, pelvis height, pelvis trajectory, and speed/duration), the same features used for Rall2R^2_{\text{all}} in the submitted version, which no method uses as an optimization target (App. A.7). For Eff, we additionally exclude failed FORCE captures and evaluate both the direction (Effdir\text{Eff}_{\text{dir}}) and magnitude (Effmag\text{Eff}_{\text{mag}}) of mass-dependent changes. All tables in this rebuttal report the revised metrics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.