- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents ReFM, a motion-retargeting framework that doesn't depend on the source mesh. It takes the source motion, skeletons from both source and target, and the target mesh as inputs. To make this work, the method uses an SO(3) canonicalizer to align the first-frame yaw, a frozen cross-character semantic encoder to extract clip-level representations, and an energy-guided refinement flow. This flow predicts quaternion-tangent-space corrections starting from Copy or HumanIK initializations, using the complete motion energy to pick step sizes for at most four updates (§4, Appendix B). Evaluating on 222 Mixamo test pairs under a unified 65-joint protocol, ReFM achieves a Pen score of 0.117 for both initializations. The paper provides extensive experiments, including controlled ablations, optimization baselines, bootstrap analysis, and an author-reported user study to support its performance gains. While the geometric improvements look solid, my main concern is whether these geometric gains translate well into actual dynamic motion quality and local semantic preservation.
Critical strengths
-
The progressive refinement mechanism is well-designed, and the authors provide solid evidence showing both its effectiveness and efficiency. Specifically, the acceptance rule (which includes a zero step size) guarantees that the total energy never increases across accepted states, even if individual energy terms might not improve (Appendix B, pp.13–14). In Table 2 (p.9), removing the flow component (w/o Flow) while keeping the same backbone, training data, initialization, and motion energy results in a worse Pen score of 0.137/0.136 (for Copy/HumanIK) compared to 0.117 with the full model. Furthermore, the authors compare ReFM to a direct optimization baseline using the same Copy initialization, complete energy, at most four steps, step-size candidates, and acceptance rule. Under this budget, direct optimization gets a Pen score of 0.131 and takes 4.3 seconds per motion, whereas ReFM-Copy achieves 0.117 in just 0.52 seconds (Appendix F.3, Table 6, pp.22–23). This is an impressive 8.3-fold speedup. I appreciate that this comparison is conducted under the same maximum iteration budget, which fairly demonstrates the benefit of the learned flow over a simple few-step optimization (though it is not a general advantage over optimization run to full convergence).
-
The geometric improvements are backed by thorough statistical analysis, generalization tests, and human evaluation. a) First, the penetration (Pen) metric shows clear improvement, dropping by about 16%/17% (from 0.140/0.141 for Copy/HumanIK to 0.117 in Table 1, p.7). The 95% target-character cluster-bootstrap confidence intervals for mean paired reductions are [0.0133, 0.0312]/[0.0133, 0.0347], with median reductions of 0.0051/0.0086 (Table 7, p.24). b) Second, the model generalizes well to unseen data. Out of 222 test pairs, 218 use unseen motion identities. On the 49 test pairs with unseen target characters, the Pen metric still drops from 0.090/0.091 down to 0.080 (Appendix F.5, Table 8, pp.24–25). c) Third, the user study provides helpful perceptual evidence that is independent of the encoder. In a double-blind setup with 20 participants evaluating 15 randomly sampled motions (Appendix F.6, Table 9, p.25), ReFM-HumanIK scores better mean ranks than the HumanIK baseline: 2.03 vs 2.39 for Overall motion quality, and 2.02 vs 3.30 for Semantic preservation. It is great that this user study evaluates the entire system and is independent of the frozen encoder, offering a more objective perspective. d) Finally, the semantic representation is also well-validated. Table 3's cross-character identity discrimination tests (p.12) and Table 2's combined semantic ablation also support the representation and its system-level use: removing the component (w/o Sem. Enc.) raises Pen to 0.129 for both initializations and lowers Copy's Semsim from 0.998 to 0.992.
Critical weaknesses
-
My major concern is whether reducing geometric penetration (Pen) actually translates into better overall dynamic motion quality. For motion retargeting, dynamic elements like temporal naturalness, foot-sliding/contacts, and characteristic gestures are crucial. Static figures or individual frames in the paper simply cannot show how these dynamics look in action, and unfortunately, there are no playable video comparisons provided for direct inspection. While Table 9's user study covers "Overall motion quality" (which includes temporal smoothness and visual plausibility), it only has 20 participants rating 15 motions. This is a very small sample compared to the 300+ total motions, and relative ranks alone don't establish how "natural" the motions are in an absolute sense. Also, the user study lacks statistical details like paired-difference confidence intervals or a proper treatment of repeated measurements. Moreover, the study excludes STaR/MeshRet. In Table 1, comparing ReFM-HumanIK to STaR, ReFM gets a lower Pen (0.117 vs 0.137) but shows higher curvature (Curv: 0.772 vs 0.702) and higher joint error (MSE: 0.091 vs 0.074). Curv is the mean trajectory acceleration magnitude and MSE is the averaged height-normalized Euclidean joint error. Because a target character can have multiple valid motions and the reference might also have some self-penetration, these higher error values don't automatically mean worse quality, but they do suggest a trade-off (Appendix E, pp.16–17). Without playable dynamic clips or a broader user study, it is hard to tell if the decrease in penetration is worth the increase in trajectory curvature and joint error.
-
The evidence supporting "local semantic preservation" feels quite indirect. The authors define semantic consistency as keeping the motion identity and characteristic articulation across characters (§4.2, p.5). However, Table 3 only tests cross-character same-identity matching and different-identity discrimination. It doesn't show whether the model makes reasonable local adaptations or if it accidentally damages characteristic articulation within the same motion. Furthermore, both the training loss E_sem and the evaluation metric Semsim rely on the exact same frozen semantic encoder, which the paper openly notes. In Table 2's ablation study, removing the semantic encoder (w/o Sem. Enc.) removes both the semantic conditioning and E_sem. While this shows that the semantic module is helpful overall, it doesn't directly prove that it specifically protects local semantic details. Since Table 9's user study only evaluates the final full system and not this specific ablation, we have to rely heavily on encoder-based metrics. I don't expect every module to have completely isolated novelty, but without video comparisons, it is hard to tell how well local semantic details are actually preserved under different character geometries. (To be clear, I am not saying the encoder fails, nor am I dismissing the positive human study in Table 9, but the evidence here is a bit weak.)
Questions for authors
-
Could you provide some playable video comparisons (for both typical and challenging cases) showing the source motion, Copy/HumanIK initialization, and ReFM output? Please use consistent camera angles, rendering, and playback speed, and disclose any postprocessing used. This would be incredibly helpful for evaluating dynamic naturalness, foot-sliding/contacts, and characteristic gestures. Additionally, could you clarify the unit of standard error in Table 9 and how repeated measurements (from participants and motions) were handled? It would also help if you could report paired rank differences and confidence intervals for the HumanIK and R2ET comparisons. Finally, if you have any side-by-side user study results comparing ReFM against STaR or MeshRet, please share them.
-
To better assess local semantic preservation, is there any evidence that is independent of the frozen encoder? For example, could you show side-by-side video comparisons or human ratings comparing the full model and the "w/o Sem. Enc." ablation on identical inputs? Specifically, I would like to know if the Semsim metric can differentiate between a reasonable geometric adaptation and actual distortion of a characteristic gesture. (I understand that complete isolation of semantic conditioning and E_sem isn't strictly necessary, but some encoder-independent validation would make the paper much stronger.)
4 comments
Response to Reviewer uTHX - Part II
W2/Q2: Local articulation and metric sensitivity; independent encoders; visual ablations and human-evaluation scope.
ReFM does not provide a theoretical guarantee of local semantic preservation. We thank the reviewer for highlighting this distinction. Our encoder provides clip-level semantic guidance, but a high similarity score cannot certify the preservation of every local wrist or elbow configuration. We have stated this limitation explicitly in Sec. 6.
The controlled ablation demonstrates a quantitative contribution from semantic guidance. Removing both semantic conditioning and the semantic energy raises Pen from 0.117 to 0.129 under both initializations. Semantic similarity decreases from 0.998 to 0.992 for Copy and remains 0.996 for HumanIK at the reported precision (Table 2). These results support the usefulness of guidance from the learned embedding space for the motion quality.
Independent rescoring supports clip-level motion consistency beyond the guidance encoder. We rescored unchanged ReFM outputs using two encoders that played no role in ReFM training or candidate selection. The first was trained independently with a different seed and network width; the second excluded every benchmark motion identity from pretraining across all characters. Both achieve held-out motion-pair ROC-AUC of 0.998. The high similarities below corroborate motion-identity consistency under these additional evaluators.
| Output | Independent encoder | Benchmark-disjoint encoder |
|---|---|---|
| ReFM-Copy | 0.999 | 0.997 |
| ReFM-HumanIK | 0.997 | 0.997 |
Visual examples provide complementary evidence, with limited visible differences in the semantic ablation. We have included video examples of the “w/o Sem. Enc.” variant in the supplementary materials. Our inspection reveals only limited visual differences in these examples, so they do not establish a clear perceptual advantage attributable to the semantic component. But in some cases, we did observe more severe penetration. Moreover, a dedicated human comparison of the semantic ablation requires additional data collection. The existing user study evaluates the final system. Comparing it with the ablated variant would require preparing matched, blinded comparisons, recruiting participants, and collecting fresh ratings across the motion set, which cannot be completed reliably within the short rebuttal period. We respectfully ask the reviewer to consider Table 2, the independent rescoring, and the supplementary ablation videos as complementary evidence within their stated scope.
We retain semantic guidance for its quantitative benefit and potential to support future semantic evaluation. As discussed in the paper, this encoder is an initial exploration toward scalable, cross-character evaluation of motion semantics. We hope that larger and more diverse training datasets, together with independent perceptual validation, can develop this representation into a useful complementary evaluation metric for the community. A broadly validated semantic metric remains a longer-term goal.
Thank you for the detailed responses. The clarification of the rendering protocol and user-study statistics is helpful, and I appreciate the additional independent-encoder evaluation and the more explicit discussion of the method’s limitations. I have now watched the provided videos. While the penetration improvements are visible, some examples still show motion-quality issues, which reinforce my earlier concern that reducing penetration does not necessarily translate into better dynamic naturalness or preservation of characteristic articulation.
Dear Reviewer uTHX,
Thank you for taking the time to watch the videos and for acknowledging the additional evaluations and clarifications. We agree that the remaining artifacts deserve attention: reducing penetration alone does not establish better dynamic naturalness or preservation of characteristic articulation.
We emphasize because it directly quantifies a concrete geometric failure, body self-interpenetration, without requiring a potentially problematic reference motion or subjective judgment. It therefore provides an interpretable and reproducible measure of geometric plausibility. However, we agree that this property is only one component of motion quality, which motivates our complementary evaluations.
To our best knowledge, a broadly validated automatic metric for dynamic naturalness and characteristic articulation across different character morphologies remains lacking. Existing evaluations therefore use proxies: R2ET [1] reports reference-based joint-position errors, while STaR [2] additionally evaluates trajectory curvature. These metrics assess reference fidelity and temporal smoothness, respectively. However, reference motions can themselves contain penetration, so a necessary geometric correction may increase MSE. Likewise, lower acceleration does not necessarily imply more natural motion or better preservation of an expressive gesture. STaR’s supplementary material also explicitly acknowledges penetration in the reference motions and describes MSE as an auxiliary evaluation. We therefore report Curv and MSE as useful auxiliary measurements without treating either as a definitive measure of perceptual quality.
Our user study and semantic evaluation address complementary aspects of this gap. Within the evaluated clips, participants preferred ReFM-HumanIK over its HumanIK initialization for overall motion quality: the mean rank improved from 2.39 to 2.03, with a paired difference of −0.36 and a 95% confidence interval of [−0.67, −0.05]. This interval resamples participants with their complete ratings and is conditional on the evaluated clips. These results support relative perceptual improvement within the study, while allowing that individual outputs still contain artifacts.
The additional encoders provide a complementary, representation-based assessment of motion consistency. We rescored unchanged outputs using two encoders that played no role in ReFM training or candidate selection: one trained with a different random seed and network width, and another excluding every benchmark motion identity from pretraining across all characters. Their mean source–output cosine similarities were:
| Output | Independently trained encoder | Benchmark-disjoint encoder |
|---|---|---|
| ReFM-Copy | 0.999 | 0.997 |
| ReFM-HumanIK | 0.997 | 0.997 |
Both evaluators achieved held-out motion-pair ROC-AUC of 0.998. These results support clip-level motion consistency beyond the guidance encoder.
We would also like to emphasize that ReFM is source-mesh-agnostic and therefore operates with less source information than MeshRet [3] and STaR [2], both of which use source and target meshes. MeshRet models dense source-surface interactions, while STaR incorporates geometric features from both characters. ReFM instead receives only skeletal motion and the rest-pose skeleton from the source; the target mesh remains available for geometric correction. Consequently, source-surface contacts and proximity relationships cannot be observed directly and must be inferred through skeletal motion and learned semantic guidance. This distinction matters when interpreting comparisons: our goal is to support retargeting when source geometry is unavailable, while improving geometric plausibility and retaining motion semantics under this reduced-information setting.
We thank the reviewer for emphasizing this valuable open problem. ReFM addresses part of it through source-mesh-agnostic progressive refinement guided by a learned cross-character semantic representation. Its contribution includes obtaining geometric improvements and evidence of perceptual benefit without access to the source geometry used by skinned methods, along with the proposed semantic encoder, which is viewed as an initial step toward a scalable semantic evaluation metric. We hope the reviewer will consider these contributions within their intended input setting, together with the complementary evidence and acknowledged limitations.
Sincerely,
Authors
[1] Zhang et al. Skinned motion retargeting with residual perception of motion semantics & geometry. CVPR, 2023.
[2] Yang et al. STaR: Seamless Spatial-Temporal Aware Motion Retargeting with Penetration and Consistency Constraints. CVPR, 2025.
[3] Ye, et al. Skinned motion retargeting with dense geometric interaction perception. NeurIPS, 2024.
Response to Reviewer uTHX - Part I
We thank the reviewer for the detailed comments on dynamic motion quality and local semantic preservation. We have revised the manuscript accordingly and provide point-by-point responses below, supported by additional visual evidence, statistical analysis, and an explicit discussion of the remaining limitations.
W1/Q1: Dynamic quality and contacts; video protocol; metric trade-offs and Pen reliability; user-study size, statistics, and missing baselines.
We agree with the reviewer that static figures alone cannot show how contacts and gestures evolve over time. For this reason, we included a link to playable side-by-side video comparisons in the abstract of our original submission. We kindly draw the reviewer’s attention to the project website, where typical and challenging cases can be viewed to assess temporal naturalness, foot sliding, contact timing, and gesture continuity. Each comparison shows the source motion alongside the initialization, baseline outputs, and ReFM output. We render all motions directly from their joint rotations without postprocessing and use the same target character, camera angles, and playback speed across the retargeted outputs.
Dynamic motion quality requires evaluating the reported metric trade-offs. ReFM-HumanIK reduces penetration relative to STaR but has higher Curv (0.772 versus 0.702) and MSE (0.091 versus 0.074, Table 1). Curv measures mean joint-acceleration magnitude, while MSE measures height-normalized joint-position deviation from a reference that may itself contain penetration. Neither metric alone determines whether a geometric correction improves perceived motion quality.
Table 9 reports standard errors across participants' mean ranks (from 1 to 5), expressed in rank units. For the paired comparisons below, we resampled participants with their complete ratings to obtain 95% confidence intervals conditional on those clips.
Paired intervals support gains over HumanIK and an overall-quality gain over R2ET. All three intervals exclude zero against HumanIK; against R2ET, only the overall-quality interval excludes zero.
We omitted STaR and MeshRet after finding that ranking all baselines placed an excessive burden on participants. Producing a complete ranking requires repeated comparisons among outputs, so adding alternatives increases both time and cognitive effort. Including these methods now would require collecting new rankings for the expanded set, which cannot be completed reliably within the short rebuttal period. We respectfully hope the reviewer will consider the current fair comparison as evidence of ReFM’s perceptual benefits within the evaluated setting. We have clarified this scope in Appendix F.6.