- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3(+1)
- Presentation
- 3(+1)
- Soundness
- 3(+1)
Read review
Paper summary
ADEPT is a latent world action model on top of a frozen π0.5 prior. It learns a token mask to keep action-dependent information, trains dynamics with multi-step rollouts, and uses a learned gate to decide when to sample and rank K candidates by predicted progress minus an EMA-consistency penalty. It reports 99.1% on LIBERO, 88.1% Macro-7 on LIBERO-Plus with clean-only training, 25.5% on RoboCasa365, and 55.3%→90.3% pooled success on six real-robot tasks.
Critical strengths
The clean-only LIBERO-Plus result (71.5%→88.1% with no perturbation exposure) is a strong signal that test-time selection over a frozen prior can add robustness, not just fit the benchmark.
The experimental design is careful in places. The same-state candidate ranking with restored simulator states and shared candidate arrays separates proposal quality from ranking quality.
The real-robot study covers three embodiments, single- and dual-arm setups, and deformable objects, with matched frozen-prior controls and Wilson intervals.
Critical weaknesses
Over H=4, the most action-sensitive part of the scene is the robot’s own body, so L_ai likely pushes the mask toward the gripper (as Figure 4 suggests) rather than object consequences. The paper should show that candidate rollouts differ in object state, not just arm state.
Progress-only rollout (86%) is below prior-only execution (88%), and always-on deliberation (90%) is worse than both 60% gating (98%) and random gating (92%). The gains may come from filtering prior samples toward typical ones rather than predicting consequences, and a simple non-model selection baseline over the same K samples is missing.
Questions for authors
See weakness
3 comments
Response to Reviewer y7o9 (2/3) - Weakness 2, part 1 of 2: progress-only selects over-estimated candidates (measured); always-on takes replacements with negative benefit (+9.0 where the gate invokes, -4.0 where it skips)
Comment 2: Progress-only rollout (86%) is below prior-only execution (88%), and always-on deliberation (90%) is worse than both 60% gating (98%) and random gating (92%).
Response 2: Both observations are consistent with one mechanism, selection error: an argmax over noisy scores can replace a tentative action that was already good. We measured it. Progress-only selects candidates whose score is over-estimated six times as much as the average candidate's, and a forced replacement gains 9.0 points where the gate invokes and loses 4.0 where it skips.
Progress-only below the prior. Let be the success probability of candidate , the progress-head score of the state it actually reaches, and the score of the imagined state. Then
s_k=q_k+\underbrace{(v_k-q_k)}_{\text{proxy mismatch}}+\underbrace{(\hat v_k-v_k)}_{\text{score error}}-\lambda_{\mathrm{rc}}c_k . \tag{R2}
Because all candidate branches were executed, the score error is measurable.
Table R3. Selected candidate on the 2,000 shared states: score error (normalized progress), latent rollout error relative to the anchor mean, and executed outcome.
| Selection rule | Score error | Rollout error | Selected outcome (%) |
|---|---|---|---|
| Uniform random | 1.00 | 75.0 | |
| Progress-only rollout | 1.52 | 74.0 | |
| Consistency-only rollout | 0.76 | 81.0 | |
| Full progress-and-consistency score | 0.84 | 85.0 |
Progress-only orders 66.0% of outcome-discordant pairs correctly, yet its top candidate succeeds no more often than a random one (74.0% against 75.0%): an argmax concentrates on the candidates with the largest positive score error (six times the mean). The consistency term removes them. It is a ranking regularizer, not a bound on the error. Near a ceiling, a wrong replacement costs more than a right one gains, which is consistent with 86% against 88%. In the same submitted study the joint score reaches 98%, against 86% for progress alone and 92% for consistency alone.
Always-on below gated. For a state , let be the success difference between executing the selected and the tentative first chunk, and the gate decision. Then
G_{\mathrm{gate}}-G_{\mathrm{all}}=-\mathbb{E}[(1-g)\Delta],\qquad G_{\mathrm{gate}}-G_{\mathrm{rand}}=\operatorname{Cov}(g,\Delta). \tag{R3}
Gating wins when the skipped states have negative replacement benefit. We executed both first chunks from each saved state with the same continuation and five repeats. The tentative action is one of the five candidates and is replaced only when another candidate outscores it; when it is kept, the two branches coincide. The gate and the scores are frozen before execution.
Table R4. First-chunk intervention, conditional on the frozen gate (rescue: selected succeeds where tentative fails; harm: the reverse).
| Gate decision | States | Tentative (%) | Selected (%) | Rescue / harm (%) | Difference (pp) |
|---|---|---|---|---|---|
| Invoke | 1,200 | 76.0 | 85.0 | 13.0 / 4.0 | |
| Skip | 800 | 91.0 | 87.0 | 2.0 / 6.0 | |
| All | 2,000 | 82.0 | 85.8 | 8.6 / 4.8 |
With the gate invoking at 60% of the states, Table R4 gives % for gated selection, % for always-on selection, and % for a random gate at the same rate.
Closed loop on the harder suite. On clean-only LIBERO-Plus (10,030 variants, three training runs) adaptive invocation reaches 88.1%, always-on 85.8%, and a random gate 84.6%; adaptive minus always-on is 2.3 points (paired 95% interval [0.2, 4.4]). The Spatial ordering of always-on and random gating (90% against 92%; 150 episodes on 50 initial states each) does not persist; both remain below adaptive invocation. Details: new Table 11 (Section 4.4) and Appendix I.3.
Response to Reviewer y7o9 (3/3) - Weakness 2, part 2 of 2: typicality baselines on the same K samples recover part of the gain; on equally typical candidate pairs ADEPT still orders 76.5% of outcomes
Comment 3: The gains may come from filtering prior samples toward typical ones rather than predicting consequences, and a simple non-model selection baseline over the same samples is missing.
Response 3: Filtering toward typical samples explains part of the gain, not all of it. With the non-model baselines on the same samples that you ask for, typicality recovers 4.0 to 5.5 of the 10 points between random choice and ADEPT; on equally typical candidate pairs the typicality rules are at chance (51%), while ADEPT still orders 76.5% of outcomes.
Table R5. Selection rules on identical candidates and executed outcomes (2,000 states, submitted Table 6 protocol) and in closed loop on clean-only LIBERO-Plus under the same gate (three training runs). Typicality-matched pairs: 2,860 of the 11,200 outcome-discordant pairs whose medoid distance and demonstration-support score each differ by less than 0.1 training standard deviations.
| Selection rule | Pairwise accuracy (%) | Selected outcome (%) | Typicality-matched pairs (%) | Plus Macro-7 (%) |
|---|---|---|---|---|
| Uniform random | 50.0 | 75.0 | - | 77.0 |
| Action-sequence medoid | 68.0 | 79.0 | 50.8 | 79.4 |
| Conditional demonstration support | 72.0 | 80.5 | 51.5 | 81.3 |
| Absolute / relative progress critic (submitted Table 6) | 70.0 / 75.0 | 78.0 / 82.0 | - | - |
| V-GPS-style critic / RoboMonkey-style verifier | 76.5 / 76.0 | 83.2 / 83.0 | - | 84.7 / 84.3 |
| Progress-only rollout | 66.0 | 74.0 | 62.5 | 83.8 |
| Consistency-only rollout | 74.0 | 81.0 | 66.0 | 84.9 |
| Full score without the variance term | 73.0 | 80.0 | 68.5 | 81.0 |
| Strong direct Transformer | 77.0 | 83.5 | 72.5 | 85.0 |
| Full ADEPT | 80.0 | 85.0 | 76.5 | 88.1 |
The baselines. The medoid is the candidate closest to the other four in normalized action distance,
k_{\mathrm{med}}=\arg\min_k\sum_{j\ne k}\sum_{h=0}^{H-1}\bigl\Vert D_{\mathrm{tr}}^{-1}\bigl(a_h^{(k)}-a_h^{(j)}\bigr)\bigr\Vert_2^2 , \tag{R4}
with the action scaling fitted on training data. Demonstration support retrieves the 20 clean training windows with the most similar observation and language context and scores a candidate by its distance to the nearest retrieved action window. The strong direct Transformer reads the unpooled frozen visual tokens, language, and the same four-chunk action window and regresses demonstration endpoint progress without predicting states. Its trainable parameters are within 5% of ADEPT's selection modules, with the same data split and the same training FLOPs. No rule sees evaluation outcomes or LIBERO-Plus data.
What the table shows. The best candidate of each set succeeds in 90.0% of states, a finite-repeat reference. (i) Typicality recovers 4.0 to 5.5 of the 10.0 selected-outcome points between random choice and ADEPT. (ii) On pairs that both typicality rules score as nearly equal, those rules are near chance (50.8%, 51.5%), while ADEPT orders 76.5% [74.4, 78.6], 3.5 points below its accuracy on all pairs and 4.0 above the direct Transformer. Its ranking therefore carries outcome information that these typicality measures do not contain. (iii) Random selection under the gate already adds 5.5 points over the prior (71.5%) in closed loop; ADEPT adds 11.1 more. This control differs from the random gate of submitted Table 7, which randomizes when the full scorer runs.
Statistical support. ADEPT minus the strong direct scorer: 1.5 points on identical candidates (paired 95% interval [0.3, 2.7], resampling source episodes with all their candidates and repeats) and 3.1 points in closed loop (97.5% interval [0.5, 5.7], resampling initial-state clusters with all their perturbation variants; 10,000 draws). The three training runs give 88.6, 87.5, and 88.2% for ADEPT and 85.4, 84.7, and 84.9% for the direct scorer.
Revision. The closed-loop rows are in new Table 10 (Section 4.4); all selectors on identical candidates and the matched-pair analysis are in Appendix I.1 (Table 31).
Response to Reviewer y7o9 (1/3) - Weakness 1: candidate rollouts differ in object state, not only arm state (object-difference error 3.4 to 2.1 cm; motion-matched ranking 78.0% vs 68.0%)
Comment 1: Over H=4, the most action-sensitive part of the scene is the robot's own body, so Lai likely pushes the mask toward the gripper (as Figure 4 suggests) rather than object consequences. The paper should show that candidate rollouts differ in object state, not just arm state.
Response 1: Candidate rollouts differ in object state, not just arm state: against executed outcomes, the error of predicted object differences between candidates falls from 3.4 to 2.1 cm, and on candidate pairs with matched robot motion ADEPT still ranks 78.0% against 68.0% for prediction without the mask. Your hypothesis holds in free space; at contact the mask weight moves to the task objects (0.40 to 0.77).
Why this needs a measurement. The variance term has the population form
\mathcal{L}_{ai}^{\mathrm{pop}}=-\frac{1}{2\sigma^2}\operatorname{tr}\operatorname{Cov}_a[\mu_a],\qquad \mu_a=f_{\mathrm{dyn}}(z_t^{\mathrm{ctrl}},a). \tag{R1}
It rewards predictions that vary with the action but does not say whether the variation is arm or object, and inverse dynamics can be satisfied by arm displacement alone. We therefore test it at three levels on the 2,000 saved states of submitted Table 6 (K=5 candidates, all four chunks executed, five repeats).
Table R1. Object consequences of candidate branches. Object error (cm): meani,,j>k∥(x^ik−x^ij)−(xik−xij)∥2 over states i and candidate pairs, with x the executed task-object position and x^ read from the imagined endpoint by frozen ridge probes fitted on clean training encodings only. Matched ranking: pairwise accuracy on outcome-discordant pairs whose executed robot motion matches.
1. Predicted candidate differences match executed object differences. The error falls from 3.4 to 2.1 cm (object-to-goal error 4.2 to 2.8 cm) and grasp-state accuracy rises from 76.5% to 85.0%.
2. The ranking advantage survives matching of robot motion. Two candidates are motion-matched when end-effector position, orientation, joint angles, and gripper aperture agree at every executed time step of the four chunks, within thresholds fixed on clean training-side branches (no success labels, no selector scores). On the 560 matched, outcome-discordant pairs (5.0% of 11,200), ADEPT ranks 78.0% against 68.0% for full-latent prediction (paired 95% interval of the difference [4.4, 15.6]) and 71.5% without the variance term ([1.0, 12.0]).
3. The mask moves to the object where candidates differ in object state. Figure 4 cannot settle this: it shows occlusion saliency (the change of the mask when an image patch is greyed out) for three frames of one episode, each panel normalized separately. Table R2 reports the mask weights themselves.
Table R2. Share of candidate sets whose branches leave a task object more than 5 mm apart, and mean mask weight per token region, by phase (2,000 anchors; overall mean weight 0.251).
At contact, which candidate closes on the object determines where it goes: the task-object tokens become action-dependent and the state-conditioned mask (Eq. 2; the budget constrains only its mean) moves weight to them. At a fixed total budget, moving weight from non-object tokens to task-object tokens accounts for 4.8 of 6.7 Macro-7 points and 19.5 of 21.5 points of matched ranking (Table R1). The fixed masks give weight wA=min(1,ρn/∣A∣) to the ∣A∣ tokens of their region A (the robot, with or without the task objects), wB=(ρn−wA∣A∣)/∣B∣ to the ∣B∣ remaining tokens, and 0 to excluded tokens (n tokens, ρ=0.25). A token that contains task-object pixels counts as an object token. Their mean is exactly 0.250 at every step (learned: 0.251), and the dynamics and heads are retrained for each mask.
Revision. Table 12 in the new Section 4.4 reports the object-consequence rows; both tables and the protocols are in Appendix I.2 (Tables 33-34).