- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
BFMTrack keeps the pretrained BFM frozen and optimizes a latent sequence for each reference using parallel simulator rollouts and policy gradients. Temporally correlated noise encourages smooth, coherent motions. This improves tracking over zero-shot BFM inference, reducing MPJPE, and also supports sparse keyframe tracking.
Critical strengths
- Improves motion-tracking performance by optimizing latent sequences at test time, without fine-tuning the pretrained BFM.
- Demonstrates strong performance on a real humanoid robot.
Critical weaknesses
BFM-Zero provides a reusable skill space for downstream tasks, with reward inference as a core capability. Motion tracking is a special case that maximizes discounted rewards for matching reference states. BFM-Zero also shows that the pretrained space can contain better behaviors than zero-shot inference finds.
If the goal is to find better latents, the evaluation should extend beyond tracking to tasks such as reward inference, and include deeper latent-space analysis and comparisons with RL- or MPC-based methods. If the focus is time-varying tasks, the authors should explain why methods such as ReLA and LoLA cannot be extended to optimize latent sequences.The paper also does not clarify whether the tracking gap comes from zero-shot inference itself or from the choice of tracking reward—for example, whether a delta-based reward is less suitable than the reward used by PHC. Without addressing these questions, I am not yet convinced that the current contribution is sufficiently broad or substantial.
Questions for authors
- GPC shows strong fall recovery, so why doesn’t the generalist tracker pretrained with GPC achieve similar recovery performance?
- The ER baseline’s EMD seems much higher than the value reported in BFM-Zero. Is this only due to a change in the motion dataset?
4 comments
Reply to Reviewer GQoy (2/3)
W2a. Scope, and extending ReLA and LoLA. Our focus is the second case the reviewer describes: objectives that vary over time, with motion tracking as the task. Within this scope, the paper compares against two RL-trained trackers (a generalist trained with the same reward and a per-motion specialist) and against the sampling-based update that BFM-Zero applies to a latent sequence "in the spirit of DIAL-MPC" (Sec. 5.3, App. F). On the full test split, this update reduces the MPJPE of ER (L = 5) from 4.7 cm to 3.4 cm, and BFMTrack reduces it to 2.5 cm at the same rollout budget (Table 16).
ReLA and LoLA both build on the BFM's value , the return of the stationary policy for a fixed task latent . Neither part carries over to tracking: a time-varying reward has no fixed , and the state-action value of a latent sequence would have to depend on time and on all remaining reference poses. ReLA relies on this quantity directly, so it is not applicable in our setting. LoLA has a purely Monte Carlo version that skips the state-action bootstrap at the end. As a new baseline we apply this Monte Carlo estimator to the whole latent sequence at once, crediting every latent with the episode return. On the hardest subset, it does give some gains over ER latents, but performs worse than BFMTrack, even when using pink noise (App. G.7, Table 10):
| Update of the latent sequence | EMD | MPJPE | MMPJPE |
|---|---|---|---|
| None (ER with L = 5) | 3.11 | 11.4 cm | 84.8 cm |
| Monte Carlo estimator of LoLA | 2.77 | 10.0 cm | 78.7 cm |
| Monte Carlo estimator of LoLA, with pink noise | 2.46 | 8.6 cm | 75.2 cm |
| BFMTrack | 2.07 | 6.8 cm | 63.4 cm |
We are happy to run this comparison on the full test split for the final version.
W2b. Zero-shot inference vs. reward gap. Both contribute, and each of the two rewards suits one of the two methods. The ideal per-step tracking reward is a delta function at the reference pose. Zero-shot inference treats this reward as stationary (W1), so the latent it finds is the one that reaches and holds the pose, which is not necessarily the one that tracks best. Within this limit, the delta reward suits inference, because it has the closed form z = B(g) (Sec. 3), which ER averages over the next L poses. The PHC reward has no closed form. Inferring from it by least squares over 500,000 states from rollouts of the frozen BFM is less accurate than ER (EMD 5.27 vs 3.11 on the hardest subset, App. G.5). We attribute this to insufficient coverage near the reference: for a reference pose, the best of these states reaches a PHC reward of 0.54 on average on the hardest subset (maximum 1) and 0.75 on the full test split. For the optimization, it is the reverse. A rollout almost surely never hits the reference pose exactly, so the delta reward gives no signal. Its projection onto the rewards that the BFM represents, the goal-reaching reward, can be optimized but is less accurate than the PHC reward, which suggests that the projection loses information. A plausible reason is that the finite-dimensional embedding favors smooth rewards (Sec. 5.3). Sikchi et al. find the same gap for stationary rewards (§4.3 of their paper). The table gives the four cases on the hardest subset (Table 8, Fig. 16):
| Latents | Reward | EMD | MPJPE | MMPJPE |
|---|---|---|---|---|
| Zero-shot, ER (L = 5) | Goal-reaching (delta) | 3.11 | 11.4 cm | 84.8 cm |
| Zero-shot, reward inference | PHC | 5.27 | 23.9 cm | 119.8 cm |
| BFMTrack | Goal-reaching (projected delta) | 2.57 | 8.6 cm | 73.4 cm |
| BFMTrack | PHC | 2.07 | 6.8 cm | 63.4 cm |
Of the gap between ER and BFMTrack, 0.54 EMD comes from optimizing the latents instead of inferring them (row 3 against row 1) and 0.50 from the reward (row 4 against row 3). The full test split shows the same for the two rewards (Fig. 6, Table 15). We ran the inference baseline on the hardest subset and are happy to add the full test split for the final version.
Reply to Reviewer GQoy (3/3)
Q1. Fall recovery of GPC. GPC reports fall recovery for its generative controller, which samples skill tokens from the character's state, without a reference, and is free to choose a get-up skill after a fall (GPC §8.3, Fig. 6, Table 9). Our generalist reproduces GPC's tracking controller, the stage that follows a reference and that GPC evaluates on tracking accuracy only. In the revised paper, we train it with fallen-state initialization and a 2 s termination grace period (Sec. 5, App. F). It then recovers much better: its MPJPE increase under strong pushes falls from 15.1 cm to 5.6 cm (BFMTrack: 2.4 cm), and it gets up again after the box impact of Fig. 5 (see also the new clips in the supplementary material). App. G.4 compares the generalist with and without these additions.
Q2. EMD of the ER baseline. BFM-Zero computes its EMD on the 29 joint angles of the Unitree G1, whereas ours is computed on stacked positions and orientations of points along the SMPL humanoid, as done in Tirinzoni et al. (2025). App. B now also states this difference. Closest to BFM-Zero's EMD for us is computing it over the 69 SMPL joint angles. Under this EMD, our ER baseline scores 0.85 and BFMTrack 0.62 (Table 6).
Response to rebuttal
I’m glad to see the additional comparisons. Although the scope is narrow, I think the problem formulation makes sense, particularly its focus on finding an optimal time-varying latent sequence. I will consider raising my score.
Reply to Reviewer GQoy (1/3)
We thank the reviewer for the review. All section, table and figure numbers refer to the revised PDF.
W1. Relation to BFM-Zero. We agree that prior work has shown that the latent space of a BFM contains better behaviors than zero-shot inference finds. We do not claim this observation and credit Sikchi et al. and BFM-Zero for it (Sec. 2, Sec. 5.1). We would like to push back on one point: motion tracking is not a special case of reward inference. Each latent specifies a stationary policy, and reward inference maps a single state-based reward to a single latent. An ideal tracking reward favors being at the reference pose gt at time t. This reward is non-stationary, and since the BFM's state excludes time and the reference, it has no projection onto the reward family of the BFM (Sec. 3). Neither a single latent nor the value of a stationary policy captures the objective. ER instead solves a stationary surrogate of the tracking problem at every step. With L = 1, its policy maximizes the discounted goal reward of the next pose. It reaches and holds that pose, with no incentive to arrive on time or to anticipate what follows. Consistent with this, L = 1 is not the most accurate choice: averaging over the next L = 5 poses lowers the MPJPE from 5.10 cm to 4.72 cm (App. G.1, Fig. 10). In both cases, the latents are inferred from the reference alone and are never corrected by the motion they produce. Dedicated trackers make the problem stationary by conditioning the policy on the reference. A BFM cannot do the same and keep its reward inference for stationary rewards, because these come without a reference and every policy would then require one as input. BFMTrack instead keeps the BFM frozen and optimizes the latent sequence against the tracking return measured in simulation (Sec. 4). Each latent is credited with the tracking reward from its own step onward, so it is optimized to reach its pose on time and to anticipate the next ones.
Prior work on adapting latents covers three of four cases:
BFM-Zero describes its experiments as "a preliminary investigation of test-time adaptation" and calls for "a more thorough understanding of fast adaptation". It evaluates its update of a latent sequence on one leaping motion under altered friction. We study the remaining case on 990 held-out clips and compare with the prior updates in W2a. The introduction now states this delimitation.