- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The work tackles a problem of force-aware human-object interaction modeling with scene context. The contribution is twofold: the FORCE-Scene dataset that combines motions from the FORCE dataset and scenes from InfBaGel and the ForceHOSI method trained on this dataset. The dataset is built by unifying the lengths of all sequences and transferring the rotation deltas from sequences capturing motion with higher force level onto a corresponding base sequence to isolate the posture changes. The method consists of three major parts: motion synthesis module (adapted from InfBaGel), force-aware adaptation and action-specific composition. ForceHOSI is evaluated on Force-Scene and compared to several baselines constructed from previous work.
Critical strengths
- The problem is well-motivated. Force-awareness is a clear gap in scene-conditioned interaction generation and a meaningful improvement over FORCE that modeled this effect in isolation from the scene.
- Meaningful design of the method. The force-response adaptation is defined to be 0 when only the reference force is supplied, thus preserving the original motion. The idea of a separate force-aware module that adapts the motion to the force probably requires less data to train compared to an end-to-end method, which suits the limited amount of available force-aware captures.
- Meaningful construction of baselines. InfBaGel is fine-tuned on the same data, and InfBaGel+FiLM variant that is conditioned on the force value directly is a very close competitor. Unfortunately, the work does not provide enough implementation details to reproduce the baselines.
Critical weaknesses
The two most critical weaknesses of the manuscript are the clarity and completeness of the explanation (for both dataset and method) and the quality of evaluation. Below is the more detailed explanation for each group:
Dataset:
- Narrow scope of force-dependent characteristics in dataset. The sequences in the dataset are resampled to be 160 frames in length to allow for rough per-frame correspondence between them. However, the downside of it is that non-posture related effects (velocity of carrying, duration of the action) are not included in the new data, thus significantly narrowing the scope of force-related characteristics that the method captures. Another consequence is that metrics that use adjacent frames (e.g. foot sliding) become incomparable across sequences because the time span between adjacent frames is not consistent.
- Dataset construction lacks details: how the placement within the scene is searched, what is the breakdown of the sequences kept after filtering (action, object, force), how the filtering is formally defined, etc.
- The supplementary material lacks video demonstration of the dataset contents.
Method:
- The text contains several heuristics that are not introduced or formally defined (e.g. , , in schedule, and ). No architecture details are provided, e.g. which contacts are used for , how is the force token constructed. Sec. 3.5 Line 260 claims that "candidate routes are selected by a separate component" that is not introduced anywhere.
- The method relies on the substantial amount of hand-crafted parameters (action specific constants and whole modules for refinement). Such design hurts the generalization and is not discussed in the limitations.
- The quality of writing has room for improvement. The work is notation heavy, yet the text does not clearly introduce most of the used terms and the notation is not consistent across it (e.g., denotes distribution, interaction type and object translation).
Evaluation:
- The train-test split of FORCE-Scene is not clearly defined. The text in Sec 5.1 mentions sequence-level split and Sec 5.5 and App. B add that same motion is reused for multiple scenes. This raises the question of whether the split accounts for the re-use of source motion to avoid motion leakage from train into test (i.e. same motion in different scenes is present in both splits).
- The tables do not report variance across multiple runs. Several gaps between methods are quite narrow and given the stochastic nature of motion generation could vanish when averaged across several runs.
- The ablation shows that the force response field introduces minimal improvement (0.005 in Ordering and 0.013 in Direction). The motion generator is already informed about the force via text prompt, thus the further correction adds little. These together with the lack of reported variance prevent the faithful estimate of the actual improvement.
- No evaluation of auto-regressive generation. The evaluation is only run on per-window basis, meaning that every 14 frames the method is conditioned on 2 GT frames. InfBaGel performs auto-regressive generation starting only with the very first two frames of the sequence in the original work. The manuscript introduces continuous rollout in the appendix but never evaluates it. This in turn means that for Table 3 the endpoint error essentially evaluates the last prediction of the model, because the last window is conditioned on the 2 GT frames.
- The baselines lack implementation details. The high-level description of the baseline methods sounds reasonable, but the text lacks any details for most of them.
- The supplementary material lacks video demonstration of the method results.
Questions for authors
The list of weaknesses is quite extensive, but some could be addressed with clarifications. In particular:
- What are the missing heuristics and how were they estimated (Method.1)
- The exact train-test split (Evaluation.1)
- Do the gaps between methods stay after averaging across runs (Evaluation.2)
- What is the result of the method with force removed from a text prompt and refiner input (to fully isolate contribution of force-response adaptation, Evaluation.3)
- What are the results of the evaluation for continuous rollout (Evaluation.4)
- Were CHOIS and HOSIG baselines retrained on the same data and how are the force and scene provided as input for them (Evaluation.5)
- Will the code be released to clarify all the heuristics and missing implementation details (Dataset.2, Method.1, Evaluation.1, Evaluation.5)