acceptodds
Withdrawn from ICLR 2027

ForceHOSI: Force-Aware Human–Object–Scene Interaction Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Human–object–scene interaction generation requires more than moving an object to a target while avoiding collisions: the human posture must also reflect the physical demands of the interaction. Changing an object's weight or resistance can require different postural responses even when its geometry and the task remain unchanged. We present PhysHOSI, a force-aware framework that generates scene-conditioned interactions with load- and resistance-dependent human motion. Our method models postural adaptation as a continuous response field along a force coordinate. Action, interaction type, and force jointly modulate the field's temporal weights through a tensor-conditioned low-rank parameterization. A task-preserving composition module retains support motion and refines manipulation without suppressing the learned pose response. To support training and evaluation, we construct FORCE-Scene by grounding recorded force-annotated interactions in 3D scenes, checking ground support and penetration, and forming strategy-consistent response pairs. Experiments demonstrate state-of-the-art performance on the evaluated force-aware interaction benchmark. Under the paired reconstruction protocol, PhysHOSI achieves better force-response ordering, more accurate object positioning at the task goal, and higher task success than the task baseline. Additional obstacle demonstrations illustrate how different physical demands can be associated with different ways of completing the same task.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human--Object--Scene Interaction Generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mbWfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The work tackles a problem of force-aware human-object interaction modeling with scene context. The contribution is twofold: the FORCE-Scene dataset that combines motions from the FORCE dataset and scenes from InfBaGel and the ForceHOSI method trained on this dataset. The dataset is built by unifying the lengths of all sequences and transferring the rotation deltas from sequences capturing motion with higher force level onto a corresponding base sequence to isolate the posture changes. The method consists of three major parts: motion synthesis module (adapted from InfBaGel), force-aware adaptation and action-specific composition. ForceHOSI is evaluated on Force-Scene and compared to several baselines constructed from previous work.

Critical strengths
  • The problem is well-motivated. Force-awareness is a clear gap in scene-conditioned interaction generation and a meaningful improvement over FORCE that modeled this effect in isolation from the scene.
  • Meaningful design of the method. The force-response adaptation is defined to be 0 when only the reference force is supplied, thus preserving the original motion. The idea of a separate force-aware module that adapts the motion to the force probably requires less data to train compared to an end-to-end method, which suits the limited amount of available force-aware captures.
  • Meaningful construction of baselines. InfBaGel is fine-tuned on the same data, and InfBaGel+FiLM variant that is conditioned on the force value directly is a very close competitor. Unfortunately, the work does not provide enough implementation details to reproduce the baselines.
Critical weaknesses

The two most critical weaknesses of the manuscript are the clarity and completeness of the explanation (for both dataset and method) and the quality of evaluation. Below is the more detailed explanation for each group:

Dataset:

  1. Narrow scope of force-dependent characteristics in dataset. The sequences in the dataset are resampled to be 160 frames in length to allow for rough per-frame correspondence between them. However, the downside of it is that non-posture related effects (velocity of carrying, duration of the action) are not included in the new data, thus significantly narrowing the scope of force-related characteristics that the method captures. Another consequence is that metrics that use adjacent frames (e.g. foot sliding) become incomparable across sequences because the time span between adjacent frames is not consistent.
  2. Dataset construction lacks details: how the placement within the scene is searched, what is the breakdown of the sequences kept after filtering (action, object, force), how the filtering is formally defined, etc.
  3. The supplementary material lacks video demonstration of the dataset contents.

Method:

  1. The text contains several heuristics that are not introduced or formally defined (e.g. f0,pf_{0,p}, sps_p, αk\alpha_k in schedule, MaM_a and BaB_a). No architecture details are provided, e.g. which contacts are used for ctc_t, how is the force token ff constructed. Sec. 3.5 Line 260 claims that "candidate routes are selected by a separate component" that is not introduced anywhere.
  2. The method relies on the substantial amount of hand-crafted parameters (action specific constants and whole modules for refinement). Such design hurts the generalization and is not discussed in the limitations.
  3. The quality of writing has room for improvement. The work is notation heavy, yet the text does not clearly introduce most of the used terms and the notation is not consistent across it (e.g., pp denotes distribution, interaction type and object translation).

Evaluation:

  1. The train-test split of FORCE-Scene is not clearly defined. The text in Sec 5.1 mentions sequence-level split and Sec 5.5 and App. B add that same motion is reused for multiple scenes. This raises the question of whether the split accounts for the re-use of source motion to avoid motion leakage from train into test (i.e. same motion in different scenes is present in both splits).
  2. The tables do not report variance across multiple runs. Several gaps between methods are quite narrow and given the stochastic nature of motion generation could vanish when averaged across several runs.
  3. The ablation shows that the force response field introduces minimal improvement (0.005 in Ordering and 0.013 in Direction). The motion generator is already informed about the force via text prompt, thus the further correction adds little. These together with the lack of reported variance prevent the faithful estimate of the actual improvement.
  4. No evaluation of auto-regressive generation. The evaluation is only run on per-window basis, meaning that every 14 frames the method is conditioned on 2 GT frames. InfBaGel performs auto-regressive generation starting only with the very first two frames of the sequence in the original work. The manuscript introduces continuous rollout in the appendix but never evaluates it. This in turn means that for Table 3 the endpoint error essentially evaluates the last prediction of the model, because the last window is conditioned on the 2 GT frames.
  5. The baselines lack implementation details. The high-level description of the baseline methods sounds reasonable, but the text lacks any details for most of them.
  6. The supplementary material lacks video demonstration of the method results.
Questions for authors

The list of weaknesses is quite extensive, but some could be addressed with clarifications. In particular:

  1. What are the missing heuristics and how were they estimated (Method.1)
  2. The exact train-test split (Evaluation.1)
  3. Do the gaps between methods stay after averaging across runs (Evaluation.2)
  4. What is the result of the method with force removed from a text prompt and refiner input (to fully isolate contribution of force-response adaptation, Evaluation.3)
  5. What are the results of the evaluation for continuous rollout (Evaluation.4)
  6. Were CHOIS and HOSIG baselines retrained on the same data and how are the force and scene provided as input for them (Evaluation.5)
  7. Will the code be released to clarify all the heuristics and missing implementation details (Dataset.2, Method.1, Evaluation.1, Evaluation.5)
Reviewer_tQ6nOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ForceHOSI generates force-dependent human–object interactions in scenes by combining a learned force-response model with scene-conditioned motion generation. It also introduces FORCE-Scene, which places existing FORCE recordings into scene contexts. Evaluation measures force-dependent pose changes and scene-interaction quality.

Critical strengths
  1. Modeling force-dependent posture together with scene constraints is a useful direction. The method improves force-response ordering and object–scene compatibility over the force FiLM baseline.
  2. FORCE-Scene provides a useful setting for studying this combined problem. The paper clearly distinguishes derived scene placements from independent source recordings.
Critical weaknesses
  1. The quantitative gains are limited or mixed across metrics. In Table 2, direction agreement improves only from 0.248 to 0.249 and pose error from 17.08° to 16.97° over force FiLM, while FORCE with grounding achieves a lower pose error of 15.72°. Ordering improves more clearly, but the results do not yet establish a substantial overall benefit from the added model complexity.
  2. The static examples make it difficult to judge the perceptual effect of force conditioning and the temporal quality of the generated interactions. Supplementary videos showing the dataset and matched baseline comparisons across force levels would help establish whether the numerical gains translate into visibly better motion.
Questions for authors

Could you provide supplementary videos showing dataset examples and matched outputs from ForceHOSI, force FiLM and FORCE with grounding under different force levels? These would help assess the practical differences and temporal artifacts.

Reviewer_aZzsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors present an important problem for human-object interaction (HOI) generation in that the forecasted motion and interaction should be affected by both:

  • (i) the object's resistance (eg a human's posture when pushing a box should be affected by the box's mass and friction between the box and floor),
  • and (ii) the surrounding scene (eg a human may elect to go around a couch than through or over it when pushing a box).

To address both of these factors, the authors developed ForceHOSI, which incorporates the force condition from FORCE [1] into the human-object-scene interaction (HOSI) generation framework from InfBaGEL [2].

[1] Zhang, et al. FORCE: Physics-Aware Human-Object Interaction. 3DV, 2025.
[2] Zou, et al. InfBaGel: Human-Object-Scene Interaction Generation with Dynamic Perception and Iterative Refinement. ICLR, 2026.

Critical strengths

The authors present an important problem for human-object scene interaction (HOSI) generation. The paper is clear and easy to follow, with well defined objectives and contributions.

Critical weaknesses

While the results show improvements in kinematic metrics, there is limited evaluation on (W1) the combined effects of the force and scene conditions and (W2) generated diversity.


(W1) Modeling and evaluation of the combined effects of (i) and (ii) are limited.
Overall, to address both (i) and (ii), ForceHOSI seems to combine the contributions of FORCE and InfBaGel (which addresses (i) and (ii) respectively) largely as separate, decoupled effects, with limited modeling of the interaction between the force and scene conditions. This is important because the context changes when the two are combined, as they are not independent factors. For example, when pushing or pulling an object while turning (eg to change route for scene adaption), you are actually pushing and pulling simultaneously, generating rotational force (torque). This torque also depends on the angle of the turn. And to further adapt to object mass and friction, taking a wider angle with larger overall distance traveled may result in less resistance than a steep turn.

The examples in Fig 3 and Fig 5 show how different object mass affects ForceHOSI's generated postures, but it is unclear how the generated routes are affected by the different object mass, and how they differ from the routes generated by InfBaGel, where the object mass is not taken into consideration. It is also unclear how this posture adaptation compares to the case in FORCE, where the scene is not taken into consideration. Including additional videos of the generated outputs in the appendix would be helpful.

There is one example in Fig 4, which shows that both the generated posture and route adapt to object mass, in that the human steps over the barrier when carrying a light load (0.5 kg) and moves around the barrier in a hunched over/squatted position when carrying a heavier load (10 kg). However, it is unclear if this resulting interaction is simplified to a categorical/threshold-based effect, in that the decision to go over or around the barrier is based on the discrete resistance levels from FORCE. One of the limitations of FORCE is its simplified dynamics assumptions, where only linear forces are estimated and taken into account when adjusting the human's posture when pushing, pulling, or carrying an object. In the current context however, although it may seem out of scope, using this simplified resistance may not be sufficient in supporting the current paper's claim that their output motion adapts to both resistance and scenery, as shown via changes in both posture and route selection. However, for the Fig 4 results:

  • on route selection: turning in order to go around the barrier at a sharp angle generates a lot of torque, which may actually lead to an overall resistance higher than simply stepping over the barrier, which seems to be below knee level. This contradicts the goal of adapting the forecasted motion to resistance.

  • on posture adaptation: turning or pivoting while in a squatted/bent over position (eg duckwalk) like in the bottom right of Figure 4 is more physically demanding on your joints than moving forward. And when carrying a heavy suitcase, it might be easier with your torso in a more upright or leaned back position.


(W2) The evaluation of generation diversity is limited. All reported metrics are kinematic based. Furthermore, the potential contradictions in adaptations previously discussed in Fig 4 may also depend on the relative sizes between the human, object, and obstacles. This is not explored. The current evaluation is done on the newly constructed FORCE-Scene test split only, and it is unclear how well the improved kinematic metrics translate to other scenarios. I am also unsure how the interaction type-specific coefficients f0,pf_{0,p} and sps_p in Eq (4) are currently calculated, and how they would extend to other actions.

Questions for authors

Questions included in weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.