acceptodds
Under review as a conference paper at ICLR 2027

The Reset-to-Rollout Gap in Adaptive Sampling for Humanoid Motion Tracking

Abstract

Recent advances in humanoid motion tracking have enabled policies to imitate diverse human motions with high fidelity. Learning such repertoires requires allocating training samples across motions of varying difficulty. Probability-based adaptive sampling blends failure-based reset prioritization with uniform sampling. Weaker blending favors faster improvement in overall tracking quality, whereas stronger blending promotes difficult-motion acquisition but can still leave some motions slow to learn or unlearned. We find a reset-to-rollout gap: difficult regions can contribute a substantially smaller share of rollout samples than their reset probabilities suggest. Increasing reset priority alone therefore does not ensure sufficient local practice for difficult-motion acquisition. To address this gap while combining weaker blending's faster refinement with stronger blending's acquisition benefits, we introduce Adaptive Rehearsal Allocation (ARA). This plug-in reserves a bounded pool of parallel environments for repeated rehearsal of difficult regions, while the remaining environments retain ordinary adaptive sampling. Across the evaluated settings, ARA consistently acquires all target motions. In training from scratch, weaker adaptive sampling augmented with ARA acquires all target motions and achieves better tracking quality than stronger adaptive sampling alone under the same training budget.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, Humanoid Robot

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gK3gOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that failure-based adaptive reset sampling (as in BeyondMimic and SONIC) gives difficult temporal bins far less of the PPO rollout buffer than their reset probabilities suggest. Attempts that start in hard bins terminate early, while long episodes in easy bins keep their environments busy. The authors formalize this as qb≈∑ipi,E[ℓi→b]/∑ipi,E[Li]q_b \approx \sum_i p_i,\mathbb{E}[\ell_{i\to b}] / \sum_i p_i,\mathbb{E}[L_i]. They then propose ARA, which reserves up to CARAC_{\text{ARA}} environments for short-horizon rehearsal of high-failure bins once a competence gate opens. On 59 motions (19 hard AMASS clips plus LAFAN1), ARA reaches 59/59 acquired motions. It also lets weak blending (α=0.3\alpha=0.3) beat strong blending (α=0.9\alpha=0.9) on MPJPE at 8192 environments.

Critical strengths
  • The diagnosis is practical and clearly stated: reset priority ≠\neq sample share. Fig. 2 shows rollout-to-reset ratios as low as ~0.055 for persistently failing bins.
  • ARA is simple and plug-in. It separates hard-bin acquisition from broad refinement and so eases a real trade-off set by α\alpha.
  • The training details (observations, rewards, DR, PPO settings) are thorough enough to aid reproduction.
Critical weaknesses
  1. Weak and possibly unnecessary empirical gains. With 16384 environments, plain A.S. already acquires 59/59 motions, and ARA changes S.R. by only 0.1 (Table 1). The benefit therefore shows up only in a single compute-limited setting. On held-out BONES-SEED (Table 2), ARA gives no improvement and is slightly worse at α=0.3\alpha=0.3. The "better tracking than strong A.S." claim mostly reflects the effect of α\alpha, not ARA itself.
  2. No ablation of six new hyperparameters. CARA,Dmax⁡,τc,τfC_{\text{ARA}}, D_{\max}, \tau_c, \tau_f, EMA rate, HshortH_{\text{short}} are all fixed; Sec. 5 concedes sensitivity at fewer environments and an acquisition/forgetting oscillation, but neither is quantified. For a method whose only contribution is an allocation rule, this is the core missing experiment.
  3. Contribution is thin. Eq. 2 is a one-line renewal-reward identity, and the survival-bias effect it describes is already noted in GMT ("limited practice received by difficult portions of long motions") and analyzed in Rhythm (RSS 2026). Framing it as a discovered phenomenon oversells the analysis.
Questions for authors
  1. Does ARA degrade motions the policy has already mastered?
  2. Why not correct pbp_b by expected episode length directly from Eq. 2? How does that compare to ARA?
1 comment

Thank you for the thoughtful review.

Weakness 1. Weak and possibly unnecessary empirical gains.

ARA consistently acquired all 59 target motions across the environment counts and sampling strengths in Table 1. A.S. also acquired all 59 at 16,384 environments. At the same count and α=0.9α=0.9, expanding the training set by roughly 13,000 AMASS clips gave 47/59 for A.S. and 59/59 for A.S.+ARA.

Acquisition and tracking on the 59 target motions after training with additional AMASS motions

MethodS.R. (%)S.C.EposE_{pos} (m)EvelE_{vel} (m/s)EmpjpeE_{mpjpe} (rad)
A.S.70.4847/590.1370.2720.106
A.S.+ARA92.3859/590.1380.2700.107

Weaker blending contributes to the tracking advantage in Figure 3. At fixed weak blending, ARA recovered the missing motions at a small tracking cost (Table 1). Held-out BONES-SEED performance was broadly similar, with slightly higher tracking errors for A.S.+ARA under weak blending (Table 2).

Weakness 2. No ablation of six new hyperparameters.

We varied the per-bin cap DmaxD_{max} and total budget CARAC_{ARA}, which bound rehearsal capacity. At a cap of 300, ARA acquired all 59 motions at every budget tested. At a budget of 1,000, a cap of 100 left one motion unacquired. Caps of 300 and 500 gave full acquisition, with slightly higher tracking errors at 500.

Sensitivity to rehearsal budget and per-bin cap

EnvsααMethodCARAC_{ARA}DmaxD_{max}S.R.↑↑S.C.↑↑Epos↓E_{pos}↓Evel↓E_{vel}↓Empjpe↓E_{mpjpe}↓
81920.3A.S.——92.62560.1140.2460.088
81920.3A.S.+ARA100030097.23590.1190.2510.093
81920.3A.S.+ARA100010096.59580.1160.2490.090
81920.3A.S.+ARA100050097.47590.1220.2540.095
81920.3A.S.+ARA50030097.16590.1170.2490.091
81920.3A.S.+ARA200030097.26590.1200.2520.093

In our training experience, final performance is less sensitive to the other four parameters near their Table 7 defaults, held fixed here. The gate (τc=0.99τ_c=0.99) lets the background sampler learn most of the corpus before rehearsal. The failure threshold (τf=0.5τ_f=0.5) shifts practice away from improving bins. With fewer environments, lowering it can help retention by delaying release (Section 5). A minimum duration of 0.1 s prevents immediate recycling near a bin's end. An EMA rate of 0.1 smooths brief failure spikes while responding to sustained changes.

Weakness 3. Contribution is thin.

We agree that prior work recognized the need to practice difficult motions and will clarify this in the introduction. We quantify the gap between intended and realized practice under adaptive sampling: a high-priority region can receive a much smaller share of rollout steps than its reset probability suggests. Equation 2 explains this relationship, Figure 2 measures it, and ARA reserves environments for repeated local practice.

Question 1. Does ARA degrade motions the policy has already mastered?

In Figure 3(a), A.S.+ARA's acquired-motion count fell by one around 32,000 iterations and recovered at the next evaluation. A.S. alone showed several temporary drops. From about 52,000 through 90,000 iterations, all 59 motions exceeded 50% success at every A.S.+ARA evaluation. Individual success rates could still vary above that threshold.

Question 2. Why not correct by expected episode length directly from Eq. 2? How does that compare to ARA?

We interpret the proposal as dividing each bin's adaptive reset probability by the mean episode duration from that bin, then renormalizing. This favors both early failures and easy bins near a clip's end. Dividing by the mean fraction of available duration completed would address the latter. Available duration is the shorter of the remaining clip length and the episode horizon.

The reset-to-rollout gap can still persist: a higher reset probability does not guarantee a corresponding share of local rollout steps (Eq. 2). The correction may provide enough practice in some settings and too little in others, as ordinary adaptive sampling did in Table 1. ARA reserves environments for repeated practice in selected regions. We have not evaluated the proposed correction.

Reviewer_NTqTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses an interesting discrepancy named "Reset-to-Rollout Gap", in which difficult motion transitions may receive high reset priority, but early terminations lead to substantially fewer rollout step collection than suggested by the reset probability. To bridge the gap, the paper proposes Adaptive Researhal Allocation that explicitly isolates a bounded capacity of parallel environments for repeated local practice on selected difficult regions. The remaining environments take ordinary adaptive sampling for broad refinement.

Critical strengths

Identifying and formalizaing the discrepancy between reset distribution and rollout exposure is fresh, insightful, and persuasive. ARA does not require complex meta-learning or changes to the reward formulation. Instead, it operates as a plug-and-play, purely as an environment scheduling logic, which make it widely adopted in training pipelines. In both fine-tuning and traning-from-scratch setups, ARA reliably acquires all challenging motions.

Critical weaknesses

In the generalization evaluation with BONES-SEED dataset, the quantitative difference between A.S. and A.S+ARA is practically negligible across all metrics. Thus, it is unclear whether ARA provides meaningful benefit to the in-the-wild applications. From this result alone, I would not be highly tempted to adopt ARA into my research.

Provided demo video merely displays simulation and real-world tracking results without explanation. It was impossible to interpret the video.

I am currently on the fence about this work; while the theoretical motivation and problem formulation are compelling, the practical gains shown in the empirical results remain questionable.

Questions for authors

Regarding Table 2, it would benefit from providing confidence interval or standard deviation as the numerical improvements are marginal.

Adding explanations to the video material would help assess the paper more clearly.

Optionally, providing sensitivity analysis on hyperparameters would strengthen the clarity of the paper as well.

1 comment

Thank you for the thoughtful review.

How large and how certain are the gains on BONES-SEED?

We agree that the BONES-SEED differences are small. We included this evaluation to measure whether concentrating practice on difficult training motions affects tracking on unseen nominal motions. At 8,192 environments and α=0.9α=0.9, adding ARA increased acquisition from 57 to all 59 target motions (Table 1). For the same configuration, held-out success changed by −0.049-0.049 percentage points (95% CI [−0.262,+0.121][-0.262, +0.121]). The practical benefit in this setting was acquiring the remaining target motions while maintaining comparable performance on unseen nominal motions.

BONES-SEED generalization at 8,192 environments and α=0.9α=0.9

Paired bootstrap intervals over 1,000 held-out motions.

MetricA.S.A.S.+ARADifference95% CI for difference
Success rate (%)98.2498.20−0.049-0.049[−0.262,+0.121][-0.262, +0.121]
Root position error (m)0.1190.117−0.00131-0.00131[−0.00320,+0.00039][-0.00320, +0.00039]
Root velocity error (m/s)0.1920.192−0.00063-0.00063[−0.00136,+0.00011][-0.00136, +0.00011]
Joint-angle error (rad)0.1160.113−0.00302-0.00302[−0.00355,−0.00246][-0.00355, -0.00246]

What do the simulation and hardware clips show?

We updated the supplementary video with motion labels and side-by-side views of the reference and policy execution so viewers can compare them directly.

How sensitive is ARA to hyperparameter choices?

Of the two parameters we varied, the per-bin cap DmaxD_{max} had the clearer effect on acquisition. It limits how many environments can rehearse a single difficult region at once. As the table below shows, with the total rehearsal budget CARAC_{ARA} fixed at 1,000, ARA acquired all 59 motions at caps of 300 and 500, but missed one motion when we lowered the cap to 100. With Dmax=300D_{max}=300, ARA acquired all 59 motions at each total budget we tested (500, 1,000, and 2,000 environments).

Sensitivity to rehearsal budget and per-bin cap

EnvsααMethodCARAC_{ARA}DmaxD_{max}S.R.↑↑S.C.↑↑Epos↓E_{pos}↓Evel↓E_{vel}↓Empjpe↓E_{mpjpe}↓
81920.3A.S.——92.62560.1140.2460.088
81920.3A.S.+ARA100030097.23590.1190.2510.093
81920.3A.S.+ARA100010096.59580.1160.2490.090
81920.3A.S.+ARA100050097.47590.1220.2540.095
81920.3A.S.+ARA50030097.16590.1170.2490.091
81920.3A.S.+ARA200030097.26590.1200.2520.093

We chose the remaining parameters heuristically (Table 7). For training from scratch, we set the competence threshold τc=0.99τ_c=0.99 because difficult transitions occupy only a small fraction of the corpus. This lets the background sampler learn most bins before ARA focuses on persistent failures. We set the failure threshold τf=0.5τ_f=0.5 because, once the policy finds a way through a difficult transition, it generally needs less concentrated practice. With fewer parallel environments, lowering τfτ_f can help retain an acquired motion by keeping its difficult bins eligible for rehearsal longer (Section 5). The minimum duration HshortH_{short} prevents an immediate timeout when rehearsal starts near the end of a bin. We chose an EMA update rate of 0.1 to smooth short-term fluctuations while allowing rehearsal assignments to follow changes in difficulty.

Reviewer_zQGAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Adaptive sampling is widely used in humanoid motion tracking, but prioritizing resets in difficult regions does not guarantee sufficient rollout exposure to those regions. The author targeted a very important problem in this field that trying to improve adaptive sampling. They proposed the Adaptive Rehearsal Allocation (ARA) strategy to assign a certain ratio of environments that specifically learn the difficult regions.

Critical strengths

The problem itself is important, and the idea of ARA is reasonable, but I actually don't see substantial improvement through the experiment.

Critical weaknesses

The empirical gains from ARA are relatively modest and inconsistent across settings. With 16,384 environments and α=0.9, ARA improves the success rate only from 98.91% to 99.01%, while the number of successfully acquired motions remains 59/59. With 8,192 environments and α=0.9, the success rate increases from 96.08% to 97.63%, and the acquired-motion count improves from 57 to 59. However, under the weaker sampling setting α=0.3, although the success rate increases from 92.62% to 97.23%, several tracking errors become worse, such as E_pos increasing from 0.114 to 0.119 and E_vel from 0.246 to 0.251. The results from the paper cannot sufficiently support the advantage of ARA, it is possible to obtain such performance fluctuation with different random seeds.

Questions for authors

see weaknesses

1 comment

Thank you for the thoughtful review.

How does ARA affect acquisition and tracking across settings?

Across the settings in Table 1, ARA consistently acquired all 59 target motions. We observed the same pattern when we expanded the training set with roughly 13,000 filtered AMASS clips and fine-tuned with 16,384 environments at α=0.9α=0.9. On the original 59 motions, A.S.+ARA acquired all 59, while A.S. alone acquired 47.

The tracking results reflect how the two methods allocate practice within a fixed training budget. Under weak blending, adaptive sampling distributes practice more broadly. ARA reserves some environments for difficult transitions, and in Table 1 this improves acquisition at a small cost in tracking accuracy relative to weak sampling alone. Figure 3 shows how the two can work together: weak sampling with ARA acquired all 59 motions and achieved lower joint-angle error than strong adaptive sampling alone under the same training budget.

Acquisition and tracking on the 59 target motions after training with additional AMASS motions

MethodS.R. (%)S.C.EposE_{pos} (m)EvelE_{vel} (m/s)EmpjpeE_{mpjpe} (rad)
A.S.70.4847/590.1370.2720.106
A.S.+ARA92.3859/590.1380.2700.107

Does the acquisition gain hold with a different training seed?

We repeated the fine-tuning comparison with a second training seed, following the Table 1 protocol with 8,192 environments and α=0.9α=0.9. At 100,000 iterations, A.S.+ARA acquired all 59 motions and A.S. alone acquired 57, matching the counts for the original seed. We also evaluated both methods 20,000 iterations earlier. A.S.+ARA had already acquired all 59 motions at that checkpoint, while A.S. alone had acquired 55.

Second training seed: acquisition and tracking on the 59 training motions

MethodIterationS.R. (%)S.C.EposE_{pos} (m)EvelE_{vel} (m/s)EmpjpeE_{mpjpe} (rad)
A.S.80,00086.9955/590.1420.2740.108
A.S.+ARA80,00093.1759/590.1500.2780.112
A.S.100,00096.3157/590.1190.2560.099
A.S.+ARA100,00098.7759/590.1200.2580.100

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.