acceptodds
Under review as a conference paper at ICLR 2027

ReFM: Semantic-Aware Refinement Flow Model for Motion Retargeting

Abstract

Motion retargeting transfers motion across characters with different skeletal structures while preserving semantic intent and physical plausibility. Despite recent progress, two fundamental questions remain: (i) how can reliable source-motion semantics be learned without high-quality paired retargeting data, and (ii) how should retargeting be formulated when no reliable paired motion can serve as a definitive regression objective? Existing methods commonly preserve semantics by constraining predictions toward copied motions. However, such initializations entangle useful articulation cues with artifacts caused by mismatched skeletal proportions and body geometry. Moreover, directly regressing a final motion in one forward pass is restrictive because retargeting is inherently underdetermined, and the desired solution must balance semantic fidelity with target-specific physical and temporal constraints rather than match a unique paired target. Motivated by these limitations, we propose ReFM, a source-mesh-agnostic, energy-guided model that reformulates motion retargeting as progressive refinement. First, an canonicalizer removes redundant global-orientation variations. Second, a cross-character semantic encoder, pretrained through contrastive learning, provides a character-invariant representation for both optimization guidance and semantic evaluation. ReFM then progressively refines an initialized target motion through a learned flow guided by semantic consistency, physical plausibility, temporal coherence, and minimal motion modification. The framework is compatible with different initialization strategies, including both direct motion copying and Autodesk HumanIK, an industry-standard full-body inverse-kinematics retargeting system. Experiments show that ReFM can further refine HumanIK-transferred motions while consistently reducing self-penetration and maintaining strong semantic consistency. Extensive evaluations demonstrate that ReFM achieves a favorable balance between semantic preservation and physical plausibility. The project website is at https://anonymous-user-tq.github.io/ReFM-Anonymous/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Motion retargeting, semantic aware, contrastive learning, 3D graph transformer, progressive refinement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uTHXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ReFM, a motion-retargeting framework that doesn't depend on the source mesh. It takes the source motion, skeletons from both source and target, and the target mesh as inputs. To make this work, the method uses an SO(3) canonicalizer to align the first-frame yaw, a frozen cross-character semantic encoder to extract clip-level representations, and an energy-guided refinement flow. This flow predicts quaternion-tangent-space corrections starting from Copy or HumanIK initializations, using the complete motion energy to pick step sizes for at most four updates (§4, Appendix B). Evaluating on 222 Mixamo test pairs under a unified 65-joint protocol, ReFM achieves a Pen score of 0.117 for both initializations. The paper provides extensive experiments, including controlled ablations, optimization baselines, bootstrap analysis, and an author-reported user study to support its performance gains. While the geometric improvements look solid, my main concern is whether these geometric gains translate well into actual dynamic motion quality and local semantic preservation.

Critical strengths
  1. The progressive refinement mechanism is well-designed, and the authors provide solid evidence showing both its effectiveness and efficiency. Specifically, the acceptance rule (which includes a zero step size) guarantees that the total energy never increases across accepted states, even if individual energy terms might not improve (Appendix B, pp.13–14). In Table 2 (p.9), removing the flow component (w/o Flow) while keeping the same backbone, training data, initialization, and motion energy results in a worse Pen score of 0.137/0.136 (for Copy/HumanIK) compared to 0.117 with the full model. Furthermore, the authors compare ReFM to a direct optimization baseline using the same Copy initialization, complete energy, at most four steps, step-size candidates, and acceptance rule. Under this budget, direct optimization gets a Pen score of 0.131 and takes 4.3 seconds per motion, whereas ReFM-Copy achieves 0.117 in just 0.52 seconds (Appendix F.3, Table 6, pp.22–23). This is an impressive 8.3-fold speedup. I appreciate that this comparison is conducted under the same maximum iteration budget, which fairly demonstrates the benefit of the learned flow over a simple few-step optimization (though it is not a general advantage over optimization run to full convergence).

  2. The geometric improvements are backed by thorough statistical analysis, generalization tests, and human evaluation. a) First, the penetration (Pen) metric shows clear improvement, dropping by about 16%/17% (from 0.140/0.141 for Copy/HumanIK to 0.117 in Table 1, p.7). The 95% target-character cluster-bootstrap confidence intervals for mean paired reductions are [0.0133, 0.0312]/[0.0133, 0.0347], with median reductions of 0.0051/0.0086 (Table 7, p.24). b) Second, the model generalizes well to unseen data. Out of 222 test pairs, 218 use unseen motion identities. On the 49 test pairs with unseen target characters, the Pen metric still drops from 0.090/0.091 down to 0.080 (Appendix F.5, Table 8, pp.24–25). c) Third, the user study provides helpful perceptual evidence that is independent of the encoder. In a double-blind setup with 20 participants evaluating 15 randomly sampled motions (Appendix F.6, Table 9, p.25), ReFM-HumanIK scores better mean ranks than the HumanIK baseline: 2.03 vs 2.39 for Overall motion quality, and 2.02 vs 3.30 for Semantic preservation. It is great that this user study evaluates the entire system and is independent of the frozen encoder, offering a more objective perspective. d) Finally, the semantic representation is also well-validated. Table 3's cross-character identity discrimination tests (p.12) and Table 2's combined semantic ablation also support the representation and its system-level use: removing the component (w/o Sem. Enc.) raises Pen to 0.129 for both initializations and lowers Copy's Semsim from 0.998 to 0.992.

Critical weaknesses
  1. My major concern is whether reducing geometric penetration (Pen) actually translates into better overall dynamic motion quality. For motion retargeting, dynamic elements like temporal naturalness, foot-sliding/contacts, and characteristic gestures are crucial. Static figures or individual frames in the paper simply cannot show how these dynamics look in action, and unfortunately, there are no playable video comparisons provided for direct inspection. While Table 9's user study covers "Overall motion quality" (which includes temporal smoothness and visual plausibility), it only has 20 participants rating 15 motions. This is a very small sample compared to the 300+ total motions, and relative ranks alone don't establish how "natural" the motions are in an absolute sense. Also, the user study lacks statistical details like paired-difference confidence intervals or a proper treatment of repeated measurements. Moreover, the study excludes STaR/MeshRet. In Table 1, comparing ReFM-HumanIK to STaR, ReFM gets a lower Pen (0.117 vs 0.137) but shows higher curvature (Curv: 0.772 vs 0.702) and higher joint error (MSE: 0.091 vs 0.074). Curv is the mean trajectory acceleration magnitude and MSE is the averaged height-normalized Euclidean joint error. Because a target character can have multiple valid motions and the reference might also have some self-penetration, these higher error values don't automatically mean worse quality, but they do suggest a trade-off (Appendix E, pp.16–17). Without playable dynamic clips or a broader user study, it is hard to tell if the decrease in penetration is worth the increase in trajectory curvature and joint error.

  2. The evidence supporting "local semantic preservation" feels quite indirect. The authors define semantic consistency as keeping the motion identity and characteristic articulation across characters (§4.2, p.5). However, Table 3 only tests cross-character same-identity matching and different-identity discrimination. It doesn't show whether the model makes reasonable local adaptations or if it accidentally damages characteristic articulation within the same motion. Furthermore, both the training loss E_sem and the evaluation metric Semsim rely on the exact same frozen semantic encoder, which the paper openly notes. In Table 2's ablation study, removing the semantic encoder (w/o Sem. Enc.) removes both the semantic conditioning and E_sem. While this shows that the semantic module is helpful overall, it doesn't directly prove that it specifically protects local semantic details. Since Table 9's user study only evaluates the final full system and not this specific ablation, we have to rely heavily on encoder-based metrics. I don't expect every module to have completely isolated novelty, but without video comparisons, it is hard to tell how well local semantic details are actually preserved under different character geometries. (To be clear, I am not saying the encoder fails, nor am I dismissing the positive human study in Table 9, but the evidence here is a bit weak.)

Questions for authors
  1. Could you provide some playable video comparisons (for both typical and challenging cases) showing the source motion, Copy/HumanIK initialization, and ReFM output? Please use consistent camera angles, rendering, and playback speed, and disclose any postprocessing used. This would be incredibly helpful for evaluating dynamic naturalness, foot-sliding/contacts, and characteristic gestures. Additionally, could you clarify the unit of standard error in Table 9 and how repeated measurements (from participants and motions) were handled? It would also help if you could report paired rank differences and confidence intervals for the HumanIK and R2ET comparisons. Finally, if you have any side-by-side user study results comparing ReFM against STaR or MeshRet, please share them.

  2. To better assess local semantic preservation, is there any evidence that is independent of the frozen encoder? For example, could you show side-by-side video comparisons or human ratings comparing the full model and the "w/o Sem. Enc." ablation on identical inputs? Specifically, I would like to know if the Semsim metric can differentiate between a reasonable geometric adaptation and actual distortion of a characteristic gesture. (I understand that complete isolation of semantic conditioning and E_sem isn't strictly necessary, but some encoder-independent validation would make the paper much stronger.)

4 comments
AuthorsOpenReview ↗

Response to Reviewer uTHX - Part I

We thank the reviewer for the detailed comments on dynamic motion quality and local semantic preservation. We have revised the manuscript accordingly and provide point-by-point responses below, supported by additional visual evidence, statistical analysis, and an explicit discussion of the remaining limitations.

W1/Q1: Dynamic quality and contacts; video protocol; metric trade-offs and Pen reliability; user-study size, statistics, and missing baselines.

We agree with the reviewer that static figures alone cannot show how contacts and gestures evolve over time. For this reason, we included a link to playable side-by-side video comparisons in the abstract of our original submission. We kindly draw the reviewer’s attention to the project website, where typical and challenging cases can be viewed to assess temporal naturalness, foot sliding, contact timing, and gesture continuity. Each comparison shows the source motion alongside the initialization, baseline outputs, and ReFM output. We render all motions directly from their joint rotations without postprocessing and use the same target character, camera angles, and playback speed across the retargeted outputs.

Dynamic motion quality requires evaluating the reported metric trade-offs. ReFM-HumanIK reduces penetration relative to STaR but has higher Curv (0.772 versus 0.702) and MSE (0.091 versus 0.074, Table 1). Curv measures mean joint-acceleration magnitude, while MSE measures height-normalized joint-position deviation from a reference that may itself contain penetration. Neither metric alone determines whether a geometric correction improves perceived motion quality.

Table 9 reports standard errors across participants' mean ranks (from 1 to 5), expressed in rank units. For the paired comparisons below, we resampled participants with their complete ratings to obtain 95% confidence intervals conditional on those clips.

Mean rank differenceOverall qualityPenetration handlingSemantic preservation
ReFM - HumanIK−0.36 [−0.67, −0.05]−0.31 [−0.60, −0.02]−1.28 [−1.57, −1.01]
ReFM - R2ET−0.62 [−0.94, −0.29]−0.24 [−0.48, +0.06]−0.11 [−0.35, +0.13]

Paired intervals support gains over HumanIK and an overall-quality gain over R2ET. All three intervals exclude zero against HumanIK; against R2ET, only the overall-quality interval excludes zero.

We omitted STaR and MeshRet after finding that ranking all baselines placed an excessive burden on participants. Producing a complete ranking requires repeated comparisons among outputs, so adding alternatives increases both time and cognitive effort. Including these methods now would require collecting new rankings for the expanded set, which cannot be completed reliably within the short rebuttal period. We respectfully hope the reviewer will consider the current fair comparison as evidence of ReFM’s perceptual benefits within the evaluated setting. We have clarified this scope in Appendix F.6.

AuthorsOpenReview ↗

Response to Reviewer uTHX - Part II

W2/Q2: Local articulation and metric sensitivity; independent encoders; visual ablations and human-evaluation scope.

ReFM does not provide a theoretical guarantee of local semantic preservation. We thank the reviewer for highlighting this distinction. Our encoder provides clip-level semantic guidance, but a high similarity score cannot certify the preservation of every local wrist or elbow configuration. We have stated this limitation explicitly in Sec. 6.

The controlled ablation demonstrates a quantitative contribution from semantic guidance. Removing both semantic conditioning and the semantic energy raises Pen from 0.117 to 0.129 under both initializations. Semantic similarity decreases from 0.998 to 0.992 for Copy and remains 0.996 for HumanIK at the reported precision (Table 2). These results support the usefulness of guidance from the learned embedding space for the motion quality.

Independent rescoring supports clip-level motion consistency beyond the guidance encoder. We rescored unchanged ReFM outputs using two encoders that played no role in ReFM training or candidate selection. The first was trained independently with a different seed and network width; the second excluded every benchmark motion identity from pretraining across all characters. Both achieve held-out motion-pair ROC-AUC of 0.998. The high similarities below corroborate motion-identity consistency under these additional evaluators.

OutputIndependent encoderBenchmark-disjoint encoder
ReFM-Copy0.9990.997
ReFM-HumanIK0.9970.997

Visual examples provide complementary evidence, with limited visible differences in the semantic ablation. We have included video examples of the “w/o Sem. Enc.” variant in the supplementary materials. Our inspection reveals only limited visual differences in these examples, so they do not establish a clear perceptual advantage attributable to the semantic component. But in some cases, we did observe more severe penetration. Moreover, a dedicated human comparison of the semantic ablation requires additional data collection. The existing user study evaluates the final system. Comparing it with the ablated variant would require preparing matched, blinded comparisons, recruiting participants, and collecting fresh ratings across the motion set, which cannot be completed reliably within the short rebuttal period. We respectfully ask the reviewer to consider Table 2, the independent rescoring, and the supplementary ablation videos as complementary evidence within their stated scope.

We retain semantic guidance for its quantitative benefit and potential to support future semantic evaluation. As discussed in the paper, this encoder is an initial exploration toward scalable, cross-character evaluation of motion semantics. We hope that larger and more diverse training datasets, together with independent perceptual validation, can develop this representation into a useful complementary evaluation metric for the community. A broadly validated semantic metric remains a longer-term goal.

Thank you for the detailed responses. The clarification of the rendering protocol and user-study statistics is helpful, and I appreciate the additional independent-encoder evaluation and the more explicit discussion of the method’s limitations. I have now watched the provided videos. While the penetration improvements are visible, some examples still show motion-quality issues, which reinforce my earlier concern that reducing penetration does not necessarily translate into better dynamic naturalness or preservation of characteristic articulation.

Dear Reviewer uTHX,

Thank you for taking the time to watch the videos and for acknowledging the additional evaluations and clarifications. We agree that the remaining artifacts deserve attention: reducing penetration alone does not establish better dynamic naturalness or preservation of characteristic articulation.

We emphasize PenPen because it directly quantifies a concrete geometric failure, body self-interpenetration, without requiring a potentially problematic reference motion or subjective judgment. It therefore provides an interpretable and reproducible measure of geometric plausibility. However, we agree that this property is only one component of motion quality, which motivates our complementary evaluations.

To our best knowledge, a broadly validated automatic metric for dynamic naturalness and characteristic articulation across different character morphologies remains lacking. Existing evaluations therefore use proxies: R2ET [1] reports reference-based joint-position errors, while STaR [2] additionally evaluates trajectory curvature. These metrics assess reference fidelity and temporal smoothness, respectively. However, reference motions can themselves contain penetration, so a necessary geometric correction may increase MSE. Likewise, lower acceleration does not necessarily imply more natural motion or better preservation of an expressive gesture. STaR’s supplementary material also explicitly acknowledges penetration in the reference motions and describes MSE as an auxiliary evaluation. We therefore report Curv and MSE as useful auxiliary measurements without treating either as a definitive measure of perceptual quality.

Our user study and semantic evaluation address complementary aspects of this gap. Within the evaluated clips, participants preferred ReFM-HumanIK over its HumanIK initialization for overall motion quality: the mean rank improved from 2.39 to 2.03, with a paired difference of −0.36 and a 95% confidence interval of [−0.67, −0.05]. This interval resamples participants with their complete ratings and is conditional on the evaluated clips. These results support relative perceptual improvement within the study, while allowing that individual outputs still contain artifacts.

The additional encoders provide a complementary, representation-based assessment of motion consistency. We rescored unchanged outputs using two encoders that played no role in ReFM training or candidate selection: one trained with a different random seed and network width, and another excluding every benchmark motion identity from pretraining across all characters. Their mean source–output cosine similarities were:

OutputIndependently trained encoderBenchmark-disjoint encoder
ReFM-Copy0.9990.997
ReFM-HumanIK0.9970.997

Both evaluators achieved held-out motion-pair ROC-AUC of 0.998. These results support clip-level motion consistency beyond the guidance encoder.

We would also like to emphasize that ReFM is source-mesh-agnostic and therefore operates with less source information than MeshRet [3] and STaR [2], both of which use source and target meshes. MeshRet models dense source-surface interactions, while STaR incorporates geometric features from both characters. ReFM instead receives only skeletal motion and the rest-pose skeleton from the source; the target mesh remains available for geometric correction. Consequently, source-surface contacts and proximity relationships cannot be observed directly and must be inferred through skeletal motion and learned semantic guidance. This distinction matters when interpreting comparisons: our goal is to support retargeting when source geometry is unavailable, while improving geometric plausibility and retaining motion semantics under this reduced-information setting.

We thank the reviewer for emphasizing this valuable open problem. ReFM addresses part of it through source-mesh-agnostic progressive refinement guided by a learned cross-character semantic representation. Its contribution includes obtaining geometric improvements and evidence of perceptual benefit without access to the source geometry used by skinned methods, along with the proposed semantic encoder, which is viewed as an initial step toward a scalable semantic evaluation metric. We hope the reviewer will consider these contributions within their intended input setting, together with the complementary evidence and acknowledged limitations.

Sincerely,

Authors

[1] Zhang et al. Skinned motion retargeting with residual perception of motion semantics & geometry. CVPR, 2023.

[2] Yang et al. STaR: Seamless Spatial-Temporal Aware Motion Retargeting with Penetration and Consistency Constraints. CVPR, 2025.

[3] Ye, et al. Skinned motion retargeting with dense geometric interaction perception. NeurIPS, 2024.

Reviewer_BEshOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ReFM, a source-mesh-agnostic framework for motion retargeting through energy-guided progressive refinement. The method combines global-heading canonicalization, a cross-character semantic encoder pretrained with contrastive learning, and a learned refinement field that iteratively corrects an initialized target motion. Refinement balances semantic consistency, self-penetration reduction, temporal coherence, and preservation of the initialization, accepting only energy-decreasing updates at inference. The framework supports both direct motion copying and HumanIK initialization. Validation includes quantitative comparisons on Mixamo, qualitative results on ScanRet, component ablations, semantic-encoder analysis, and a user study. The reported results demonstrate reduced self-penetration while retaining high encoder-based semantic similarity, although improvements do not extend uniformly to the auxiliary motion-error and trajectory-curvature metrics.

Critical strengths
  1. A well-motivated progressive refinement formulation. ReFM treats retargeting as iterative correction under explicit semantic, geometric, and temporal objectives. Its energy-decreasing update mechanism provides a way to balance these competing requirements. The controlled comparison against a one-step predictor supports the value of this formulation: progressive refinement reduces penetration under both initialization settings.

  2. Practical applicability with consistent geometric improvements. ReFM requires no source mesh and can refine both copied motions and outputs from the widely used HumanIK system. On Mixamo, it reduces the reported penetration metric by approximately 16–17% relative to these initializations while retaining high encoder-based semantic similarity, and also achieves lower penetration than the evaluated baselines.

Critical weaknesses
  1. The characterization of prior work is overly broad. The introduction states that existing methods “commonly rely on self-reconstruction and copied-motion consistency,” which understates the richer guidance explored in prior work. Beyond the visual-language guidance of Zhang et al. (2024a), MeshRet (Ye et al., 2024) preserves dense surface interaction relationships to guide retargeting, as the paper itself acknowledges in the related work. Such constraints should not be conflated with simply matching copied joint rotations. This framing weakens the motivation for the proposed semantic representation by overstating the limitations of existing approaches. The authors should distinguish ReFM’s learned, source-mesh-agnostic semantic guidance from prior correspondence-based guidance more precisely.

  2. The user study omits important competitive baselines. Although STaR and MeshRet are included in the quantitative and qualitative comparisons, neither appears in the user study, which compares ReFM only against naive copying, HumanIK, SAN, and R2ET. Restricting the study to source-mesh-agnostic alternatives is understandable, but the omission limits the evidence for perceptual benefits relative to the stronger geometry-aware methods considered elsewhere. This is particularly consequential because the semantic-similarity metric uses the same encoder employed to guide ReFM, making independent human evaluation especially valuable. Including STaR and MeshRet, or explicitly justifying their exclusion and narrowing the resulting claims, would strengthen confidence in the reported advantages.

Questions for authors
  1. Clarification of the positioning relative to prior work: How should the introduction’s characterization of existing methods as relying on self-reconstruction and copied-motion consistency be reconciled with MeshRet’s dense surface-interaction guidance? Please clarify what semantic information ReFM captures beyond such correspondence-based constraints, distinguishing the benefits of its source-mesh-agnostic setting from claims about limitations of prior guidance.

  2. Clarification of the user-study scope and supporting evidence: Why were STaR and MeshRet excluded from the user study despite being included in the other comparisons? Is there human-evaluation evidence comparing ReFM with these methods? Since the reported semantic-similarity metric shares the encoder used to guide ReFM, this comparison would materially affect my confidence in its perceptual advantages. If such evidence is unavailable, please clarify the scope of the conclusions supported by the current study.

1 comment
AuthorsOpenReview ↗

Response to Reviewer BEsh

We thank the reviewer for highlighting the need for more precise positioning relative to prior work and clearer justification of the scope and supporting evidence of our human evaluation. We have revised the manuscript accordingly to strengthen these aspects. Below, we address each comment and question point by point.

W1/Q1: Prior-work characterization; learned semantic guidance without source meshes or rendered images.

We have distinguished prior geometric and semantic guidance more precisely. We agree that the introduction characterizes prior supervision too broadly. MeshRet models dense contact and noncontact surface interactions, and Zhang et al. (2024a) obtain semantic guidance from rendered motion with a vision–language model. These forms of guidance go beyond copied-motion consistency. We have revised Sec. 1 to distinguish these approaches explicitly.

ReFM learns semantic guidance directly from source skeletal motion. The encoder takes canonicalized joint motion and the rest-pose skeleton, learns to associate the same motion across characters, and scores candidate refinements against the source embedding. It requires no source mesh, source skinning weights, rendered images, or vision–language features. The target mesh supplies geometry for shape conditioning and penetration evaluation.

The learned representation supports cross-character matching and geometric correction. It captures motion identity and articulation patterns across characters, complementing explicit geometric correspondence constraints. On held-out motion groups, the same motion on different characters has mean similarity 0.997, versus 0.086 for different motions, including distinct executions within the same action category (Table 3). Removing both semantic conditioning and its energy raises Pen from 0.117 to 0.129 under each initialization (Table 2). The contribution is therefore the availability and utility of learned skeletal guidance without source-surface information; it does not require claiming that prior methods lack meaningful interaction guidance. We have clarified this contribution in Sec. 1.

W2/Q2: Missing STaR/MeshRet human comparisons; shared-encoder metric, independent evidence, fairness, and study scope.

We limited the user study to five alternatives to keep the ranking task manageable. For each of the 15 motions, participants ranked all outputs on three criteria. Including STaR and MeshRet would have required additional comparisons for every motion and criterion. Because the ranks are relative to the alternatives shown, the existing ratings cannot establish how participants would rank ReFM against these two methods. A valid comparison would require presenting the expanded set under the same blinded protocol and collecting fresh rankings from participants. We cannot reliably complete that additional study within the rebuttal period. We hope the reviewer will consider our fair comparison with source-mesh-agnostic alternatives, together with the independent-encoder results (in following), as complementary evidence of ReFM’s effectiveness, and we have clarified the scope of the study and the absence of human comparisons with STaR and MeshRet in Appendix F.6.

Participants ranked ReFM-HumanIK ahead of its HumanIK initialization. Participants viewed the source motion as a reference and evaluated anonymized outputs in randomized order, with matched target characters, viewpoints, and playback settings. Refinement improved the mean rank from 2.39 to 2.03 for overall motion quality and from 3.30 to 2.02 for semantic preservation, with lower ranks indicating stronger preference (Table 9). These judgments provide evidence of perceptual improvement over HumanIK independent of the guidance encoder.

Independent encoders provide complementary evidence of clip-level semantic consistency. We rescored the same ReFM outputs with two encoders that played no role in ReFM training or candidate selection. The first used a different random seed and network width from the guidance encoder. The second excluded every benchmark motion identity from pretraining across all characters. Both achieved a held-out ROC-AUC of 0.998 for distinguishing same-motion from different-motion pairs and assigned high mean cosine similarity to the source and refined motions under both initializations, as included in Appendix F.7.

OutputIndependent encoderBenchmark-disjoint encoder
ReFM-Copy0.9990.997
ReFM-HumanIK0.9970.997
Reviewer_7DV7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses the challenge of skin-agnostic motion retargeting—transferring skeletal motion from a source character to a target character with different skeletal proportions and topology without requiring source mesh or skinning information. The paper aims to solve two main problems of existing methods: 1. Semantic Representation without Dense Paired Supervision: Sparse paired supervision forces existing methods to rely on self-reconstruction or copied-motion consistency, which entangles useful motion articulation with geometric misalignment and self-penetration artifacts. 2. Formulation of Underdetermined Retargeting: Existing neural methods predominantly predict target motion in a single forward pass. However, motion retargeting is inherently underdetermined, requiring a balance between semantic preservation, physical plausibility, and temporal coherence rather than regression toward a single ground-truth reference.

To solve these problems, the authors propose ReFM, an energy-guided progressive refinement framework composed of three main technical modules: 1. SO(3) Canonicalizer: A parameter-free module that estimates the horizontal facing direction (yaw angle) from root and shoulder joints and canonicalizes root rotations and translations, removing redundant global heading variations while preserving pitch, roll, and parent-relative articulation; 2. Cross-Character Semantic Encoder: A 3D Graph Transformer pre-trained via supervised contrastive learning on cross-character motion pairs to extract character-invariant, clip-level motion embeddings used for semantic guidance and evaluation. 3. Energy-Guided Refinement Flow: A spatio-temporal graph transformer conditioned on source semantics, skeleton features, and target shape feature. Instead of one-step regression, it iteratively updates an initialized motion along a learned descent direction approximating the gradient of a composite differentiable motion energy. The energy balances semantic discrepancy (EsemE_{sem}), initialization deviation (EinitE_{init}), target self-penetration (EpenE_{pen}), temporal smoothness (EsmoE_{smo}), and trajectory curvature (EcurvE_{curv}).

The authors claim that ReFM achieves a favorable balance between semantic preservation and physical plausibility. Specifically, ReFM reduces self-penetration rates by approximately 16%–17% over naive copying and HumanIK initializations on Mixamo, generalizes to unseen characters, achieves an 8.3×8.3\times speedup over test-time direct gradient optimization, and is rated superior in user studies across overall motion quality, penetration handling, and semantic preservation.

Critical strengths
  1. The proposed method achieves SOTA performance under penetration and curvature metrics.
  2. The idea of using flow matching for motion retargeting is novel and interesting.
  3. This paper is well-organized.
  4. The experiment part is comprehensive. It includes complete ablation study, user study and high-quality visualization.
Critical weaknesses
  1. Flow-matching-based methods may cost much more time than the one-pass feed-foward-based methods.

  2. As can be seen in Table 2, the involvement of Canonicalizer does not help achieve improvement under the metrics semsimsem_{sim}, curvature and MSE. The authors need to explain the necesarity of the introduction of canonicalizer.

  3. The primary quantitative metric for evaluating semantic consistency is computed using the exact same frozen cross-character semantic encoder (EsemE_{sem}) that defines the semantic energy term EsemE_{sem} used during model training and inference-time energy monitoring. Because the model directly optimizes energy derived from EsemE_{sem}, reporting high Semsim\text{Sem}_{\text{sim}} values (e.g., 0.9980.998 in Table 1) primarily measures whether training optimized its objective, rather than serving as a fully independent external benchmark. Although the authors explicitly acknowledge this limitation in Sec. 5.1 and Sec. 6, the evaluation would be substantially strengthened if the authors evaluated semantic preservation using an alternative, independently trained motion feature extractor or action recognition model (e.g., pretrained on BABEL or NTU RGB+D) that played no role during ReFM training or loss computation.

  4. ReFM is strictly a refinement model that modifies an initialized target motion rather than generating motion from scratch. It restricts updates to editable limb joints via a binary mask and enforces a maximum iteration budget K=4K=4 with candidate step sizes A={0,0.02,0.05,0.1}\mathcal{A} = \{0, 0.02, 0.05, 0.1\}, the final motion quality remains tightly bound to the quality of the initial candidate. If an initial motion contains extreme global translation errors or severe topological misalignment outside the limb mask, ReFM cannot recover. The authors should state these operational bounds clearly and evaluate ReFM under worse synthetic initializations (e.g., initializations with added Gaussian noise or mismatched limb proportions) to establish the breakdown limits of the refinement flow.

Questions for authors

Please check the weakness part.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 7DV7 - Part I

We thank the reviewer for the constructive feedback and provide clarifications and additional experiments addressing refinement efficiency, canonicalization, semantic evaluation, and initialization robustness. We have revised the manuscript accordingly to incorporate these clarifications and new results. Below, we provide point-by-point responses to the comments.

W1: Refinement formulation; computational overhead relative to one-pass prediction.

ReFM uses learned, energy-guided refinement. The model predicts correction directions, and each step evaluates candidate updates against the motion energy, retaining the current motion if no candidate improves it. To isolate the benefit of progressive refinement, we trained a one-pass regressor with the same backbone, data, initialization, and energy. Progressive refinement reduces penetration (Pen) from 0.137 to 0.117 with Copy initialization and from 0.136 to 0.117 with HumanIK (Table 2). The additional steps therefore yield a measurable improvement over one-pass prediction.

Learned correction directions substantially reduce refinement time. On one NVIDIA T4, ReFM processes 60-frame, 65-joint clips at an amortized 0.40 s per clip with batch size 8. In a separate matched timing experiment with Copy initialization, direct optimization differentiates the same energy at every step and takes 4.3 s per motion, whereas ReFM predicts the correction direction and takes 0.52 s, as shown in Table 6. These comparisons quantify the trade-off: progressive refinement improves penetration over one-pass prediction while running 8.3×8.3\times faster than direct energy optimization.

W2: Necessity of canonicalization; effects on semantic similarity, Curv, and MSE.

Canonicalization makes refinement equivariant to global yaw. For a fixed target character, rotating the source motion about the vertical axis should rotate the retargeted motion by the same angle while preserving its local articulation. The canonicalizer enforces this property by removing the initial heading before refinement and restoring it afterward. Two motions that differ only in global yaw, together with their corresponding initializations, therefore receive the same correction in canonical space. This preserves pitch and roll and removes the need for the model to learn heading consistency from data.

To examine whether the penetration gap in Table 2 persists across input headings, we sampled three yaw angles uniformly from [0,2π][0,2\pi] and applied each rotation to the source root. The table reports Pen for the model trained without canonicalization under both initializations (Appendix F.7, Table 10).

InitializationRandom yaw 1Random yaw 2Random yaw 3
Copy0.12980.12930.1294
HumanIK0.13100.13070.1305

Canonicalization improves physical plausibility and ensures consistent refinement across global headings. As shown in Table 2, canonicalization reduces self-penetration under both initializations, with an absolute change of only 0.001 in semantic similarity in each case. Both auxiliary metrics, Curv and MSE, increase. However, in practice, an animator may change a character’s facing direction to fit a scene while keeping the motion itself fixed. We hope the retargeted motion will follow that change in heading while preserving the same local articulation throughout the sequence. Yaw equivariance ensures this behavior by applying the same refinement in canonical space and restoring the desired heading afterward. We therefore retain the canonicalizer for its benefit in reducing penetration and the consistency it provides across global headings. We included this discussion in Appendix F.7.

AuthorsOpenReview ↗

Response to Reviewer 7DV7 - Part II

W3: Shared guidance/evaluation encoder; independent evaluators and external action-recognition models.

We build two additional evaluators to address the direct reuse of the guidance encoder. We rescored the unchanged ReFM outputs with two encoders that played no role in ReFM training or candidate selection. The first was trained independently with a different seed and network width. The second excluded every motion identity in the 222-pair benchmark from pretraining across all characters. Both achieved held-out ROC-AUC of 0.998 for distinguishing same-motion from different-motion pairs.

OutputIndependent encoderBenchmark-disjoint encoder
ReFM-Copy0.9990.997
ReFM-HumanIK0.9970.997

Motion-identity consistency extends beyond the guidance encoder. Both evaluators assign high similarity to the refined motions. These results address the concern of reusing the direct evaluator.

Motion matching requires distinguishing executions within an action category. Our evaluation treats Dancing(1) and Dancing(2), for example, as distinct motions. An action recognizer pretrained on BABEL or NTU RGB+D would require compatible skeleton conversion and validation that its features resolve such distinctions; action-label recognition alone does not establish this ability. The user study supplies complementary evidence independent of these encoders: participants ranked ReFM-HumanIK ahead of HumanIK for semantic preservation. Together, the independent rescoring and human judgments strengthen the evidence beyond the guidance-based metric. We have reported these results in Appendix F.

W4: Initialization dependence; editable-joint and iteration limits; robustness to noise and proportion changes.

ReFM operates within a defined correction range. It updates the major arm and leg joint rotations on a compatible target skeleton for at most K=4K=4 steps, selecting a step size from {0,0.02,0.05,0.1}\{0,0.02,0.05,0.1\} at each step. Root translation and rotations outside the editable set remain fixed. Consequently, arbitrary root errors, incompatible joint correspondences, and errors requiring changes outside this set are outside its correction scope.

A controlled noise test probes robustness to corrupted initialization. To stress-test the method within this setting, we perturbed every joint of the copied initialization with random rotations of standard deviation 10∘10^\circ, including joints outside the editable set.

InitializationPen before → after ↓Semantic similarity before → after ↑Curv before → after ↓
Clean Copy0.140 → 0.1170.999 → 0.9980.66 → 0.77
Copy with 10∘10^\circ noise0.144 → 0.1290.922 → 0.92611.85 → 11.25

Refinement partially corrects the tested corruption. Under this corruption, refinement still successfully reduces Pen and modestly improves semantic similarity and Curv. It demonstrates that our ReFM is robust for worse synthetic initializations. We have added the initialization-noise experiment to Appendix F.7.

Unseen-target results support robustness to unfamiliar body proportions. Table 8 provides a complementary evaluation on 49 pairs involving three unseen target characters. Pen decreases from 0.090 to 0.080 for Copy and from 0.091 to 0.080 for HumanIK. This supports refinement on unfamiliar target shapes under compatible skeletal correspondences.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.