acceptodds
Under review as a conference paper at ICLR 2027

ADEPT: Action-Sensitive World Action Models with Adaptive Deliberation for Robotic Manipulation

Abstract

World action models (WAMs) compare candidate actions through imagined futures, but effective manipulation requires predictions that distinguish action consequences and computation allocated to decisions that benefit from deliberation. Generic visual prediction can emphasize nuisance variation, recursive errors can obscure candidate differences, and progress-only scoring can favor inaccurate futures. We present ADEPT, an action-sensitive latent WAM that learns what to predict for action selection and when to plan. A learned token mask preserves action-dependent information; multi-step autoregressive training aligns latent prediction with the deployment horizon. A progress-informed gate invokes candidate evaluation, which combines predicted task progress with rollout consistency. Clean-only component controls and same-state candidate comparisons test representation, recursive training, and action ranking separately. ADEPT achieves 99.1% success on LIBERO and 88.1% Macro-7 on LIBERO-Plus after standard-LIBERO-only training, exceeding its frozen action prior by 16.6 percentage points. On RoboCasa365 it achieves 25.5% split-macro success. Across six real-robot tasks, ADEPT improves pooled success from 55.3% for matched frozen-prior controls to 90.3%. Adaptive deliberation reduces average decision latency by 31% relative to always-on planning in the profiled LIBERO-Spatial setting.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world action models, robotic manipulation, adaptive computation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_y7o9OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3(+1)
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

ADEPT is a latent world action model on top of a frozen π0.5 prior. It learns a token mask to keep action-dependent information, trains dynamics with multi-step rollouts, and uses a learned gate to decide when to sample and rank K candidates by predicted progress minus an EMA-consistency penalty. It reports 99.1% on LIBERO, 88.1% Macro-7 on LIBERO-Plus with clean-only training, 25.5% on RoboCasa365, and 55.3%→90.3% pooled success on six real-robot tasks.

Critical strengths

The clean-only LIBERO-Plus result (71.5%→88.1% with no perturbation exposure) is a strong signal that test-time selection over a frozen prior can add robustness, not just fit the benchmark.

The experimental design is careful in places. The same-state candidate ranking with restored simulator states and shared candidate arrays separates proposal quality from ranking quality.

The real-robot study covers three embodiments, single- and dual-arm setups, and deformable objects, with matched frozen-prior controls and Wilson intervals.

Critical weaknesses

Over H=4, the most action-sensitive part of the scene is the robot’s own body, so L_ai likely pushes the mask toward the gripper (as Figure 4 suggests) rather than object consequences. The paper should show that candidate rollouts differ in object state, not just arm state.

Progress-only rollout (86%) is below prior-only execution (88%), and always-on deliberation (90%) is worse than both 60% gating (98%) and random gating (92%). The gains may come from filtering prior samples toward typical ones rather than predicting consequences, and a simple non-model selection baseline over the same K samples is missing.

Questions for authors

See weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer y7o9 (1/3) - Weakness 1: candidate rollouts differ in object state, not only arm state (object-difference error 3.4 to 2.1 cm; motion-matched ranking 78.0% vs 68.0%)

Comment 1: Over H=4H=4, the most action-sensitive part of the scene is the robot's own body, so Lai\mathcal{L}_{ai} likely pushes the mask toward the gripper (as Figure 4 suggests) rather than object consequences. The paper should show that candidate rollouts differ in object state, not just arm state.

Response 1: Candidate rollouts differ in object state, not just arm state: against executed outcomes, the error of predicted object differences between candidates falls from 3.4 to 2.1 cm, and on candidate pairs with matched robot motion ADEPT still ranks 78.0% against 68.0% for prediction without the mask. Your hypothesis holds in free space; at contact the mask weight moves to the task objects (0.40 to 0.77).

Why this needs a measurement. The variance term has the population form

\mathcal{L}_{ai}^{\mathrm{pop}}=-\frac{1}{2\sigma^2}\operatorname{tr}\operatorname{Cov}_a[\mu_a],\qquad \mu_a=f_{\mathrm{dyn}}(z_t^{\mathrm{ctrl}},a). \tag{R1}

It rewards predictions that vary with the action but does not say whether the variation is arm or object, and inverse dynamics can be satisfied by arm displacement alone. We therefore test it at three levels on the 2,000 saved states of submitted Table 6 (K=5K=5 candidates, all four chunks executed, five repeats).

Table R1. Object consequences of candidate branches. Object error (cm): mean⁡i,,j>k∥(x^ik−x^ij)−(xik−xij)∥2\operatorname{mean}_{i,\\,j > k}\Vert(\hat x_{ik}-\hat x_{ij})-(x_{ik}-x_{ij})\Vert_2 over states ii and candidate pairs, with xx the executed task-object position and x^\hat x read from the imagined endpoint by frozen ridge probes fitted on clean training encodings only. Matched ranking: pairwise accuracy on outcome-discordant pairs whose executed robot motion matches.

PredictorObject error (cm)Grasp state (%)Matched ranking (%)Plus Macro-7 (%)
Candidate-invariant persistence4.572.550.0-
Full-latent dynamics (no mask)3.476.568.078.4
Mask without the variance term3.079.071.581.0
Fixed robot-centered mask, task-object tokens excluded4.0-56.581.4
Fixed robot-centered mask, task-object tokens included2.4-76.086.2
Full ADEPT2.185.078.088.1

1. Predicted candidate differences match executed object differences. The error falls from 3.4 to 2.1 cm (object-to-goal error 4.2 to 2.8 cm) and grasp-state accuracy rises from 76.5% to 85.0%.

2. The ranking advantage survives matching of robot motion. Two candidates are motion-matched when end-effector position, orientation, joint angles, and gripper aperture agree at every executed time step of the four chunks, within thresholds fixed on clean training-side branches (no success labels, no selector scores). On the 560 matched, outcome-discordant pairs (5.0% of 11,200), ADEPT ranks 78.0% against 68.0% for full-latent prediction (paired 95% interval of the difference [4.4, 15.6]) and 71.5% without the variance term ([1.0, 12.0]).

3. The mask moves to the object where candidates differ in object state. Figure 4 cannot settle this: it shows occlusion saliency (the change of the mask when an image patch is greyed out) for three frames of one episode, each panel normalized separately. Table R2 reports the mask weights themselves.

Table R2. Share of candidate sets whose branches leave a task object more than 5 mm apart, and mean mask weight per token region, by phase (2,000 anchors; overall mean weight 0.251).

PhaseObject-divergent sets (%)Robot tokensTask-object tokensTask-object, no gripper overlap
Transit in free space3.10.830.400.39
Approach and pre-grasp24.70.720.580.56
Grasp and contact70.50.620.770.74
Transport and placement58.00.640.710.69

At contact, which candidate closes on the object determines where it goes: the task-object tokens become action-dependent and the state-conditioned mask (Eq. 2; the budget constrains only its mean) moves weight to them. At a fixed total budget, moving weight from non-object tokens to task-object tokens accounts for 4.8 of 6.7 Macro-7 points and 19.5 of 21.5 points of matched ranking (Table R1). The fixed masks give weight wA=min⁡(1,ρn/∣A∣)w_A=\min(1,\rho n/\vert A\vert ) to the ∣A∣\vert A\vert tokens of their region AA (the robot, with or without the task objects), wB=(ρn−wA∣A∣)/∣B∣w_B=(\rho n-w_A\vert A\vert )/\vert B\vert to the ∣B∣\vert B\vert remaining tokens, and 0 to excluded tokens (nn tokens, ρ=0.25\rho=0.25). A token that contains task-object pixels counts as an object token. Their mean is exactly 0.250 at every step (learned: 0.251), and the dynamics and heads are retrained for each mask.

Revision. Table 12 in the new Section 4.4 reports the object-consequence rows; both tables and the protocols are in Appendix I.2 (Tables 33-34).

AuthorsOpenReview ↗

Response to Reviewer y7o9 (2/3) - Weakness 2, part 1 of 2: progress-only selects over-estimated candidates (measured); always-on takes replacements with negative benefit (+9.0 where the gate invokes, -4.0 where it skips)

Comment 2: Progress-only rollout (86%) is below prior-only execution (88%), and always-on deliberation (90%) is worse than both 60% gating (98%) and random gating (92%).

Response 2: Both observations are consistent with one mechanism, selection error: an argmax over noisy scores can replace a tentative action that was already good. We measured it. Progress-only selects candidates whose score is over-estimated six times as much as the average candidate's, and a forced replacement gains 9.0 points where the gate invokes and loses 4.0 where it skips.

Progress-only below the prior. Let qkq_k be the success probability of candidate kk, vkv_k the progress-head score of the state it actually reaches, and v^k\hat v_k the score of the imagined state. Then

s_k=q_k+\underbrace{(v_k-q_k)}_{\text{proxy mismatch}}+\underbrace{(\hat v_k-v_k)}_{\text{score error}}-\lambda_{\mathrm{rc}}c_k . \tag{R2}

Because all candidate branches were executed, the score error is measurable.

Table R3. Selected candidate on the 2,000 shared states: score error v^k−vk\hat v_k-v_k (normalized progress), latent rollout error relative to the anchor mean, and executed outcome.

Selection ruleScore errorRollout errorSelected outcome (%)
Uniform random+0.012+0.0121.0075.0
Progress-only rollout+0.071+0.0711.5274.0
Consistency-only rollout+0.004+0.0040.7681.0
Full progress-and-consistency score+0.019+0.0190.8485.0

Progress-only orders 66.0% of outcome-discordant pairs correctly, yet its top candidate succeeds no more often than a random one (74.0% against 75.0%): an argmax concentrates on the candidates with the largest positive score error (six times the mean). The consistency term removes them. It is a ranking regularizer, not a bound on the error. Near a ceiling, a wrong replacement costs more than a right one gains, which is consistent with 86% against 88%. In the same submitted study the joint score reaches 98%, against 86% for progress alone and 92% for consistency alone.

Always-on below gated. For a state xx, let Δ(x)\Delta(x) be the success difference between executing the selected and the tentative first chunk, and g(x)g(x) the gate decision. Then

G_{\mathrm{gate}}-G_{\mathrm{all}}=-\mathbb{E}[(1-g)\Delta],\qquad G_{\mathrm{gate}}-G_{\mathrm{rand}}=\operatorname{Cov}(g,\Delta). \tag{R3}

Gating wins when the skipped states have negative replacement benefit. We executed both first chunks from each saved state with the same continuation and five repeats. The tentative action is one of the five candidates and is replaced only when another candidate outscores it; when it is kept, the two branches coincide. The gate and the scores are frozen before execution.

Table R4. First-chunk intervention, conditional on the frozen gate (rescue: selected succeeds where tentative fails; harm: the reverse).

Gate decisionStatesTentative (%)Selected (%)Rescue / harm (%)Difference (pp)
Invoke1,20076.085.013.0 / 4.0+9.0+9.0
Skip80091.087.02.0 / 6.0−4.0-4.0
All2,00082.085.88.6 / 4.8+3.8+3.8

With the gate invoking at 60% of the states, Table R4 gives 0.6(85.0)+0.4(91.0)=87.40.6(85.0)+0.4(91.0)=87.4% for gated selection, 0.6(85.0)+0.4(87.0)=85.80.6(85.0)+0.4(87.0)=85.8% for always-on selection, and 82.0+0.6(3.8)=84.382.0+0.6(3.8)=84.3% for a random gate at the same rate.

Closed loop on the harder suite. On clean-only LIBERO-Plus (10,030 variants, three training runs) adaptive invocation reaches 88.1%, always-on 85.8%, and a random gate 84.6%; adaptive minus always-on is 2.3 points (paired 95% interval [0.2, 4.4]). The Spatial ordering of always-on and random gating (90% against 92%; 150 episodes on 50 initial states each) does not persist; both remain below adaptive invocation. Details: new Table 11 (Section 4.4) and Appendix I.3.

AuthorsOpenReview ↗

Response to Reviewer y7o9 (3/3) - Weakness 2, part 2 of 2: typicality baselines on the same K samples recover part of the gain; on equally typical candidate pairs ADEPT still orders 76.5% of outcomes

Comment 3: The gains may come from filtering prior samples toward typical ones rather than predicting consequences, and a simple non-model selection baseline over the same KK samples is missing.

Response 3: Filtering toward typical samples explains part of the gain, not all of it. With the non-model baselines on the same KK samples that you ask for, typicality recovers 4.0 to 5.5 of the 10 points between random choice and ADEPT; on equally typical candidate pairs the typicality rules are at chance (51%), while ADEPT still orders 76.5% of outcomes.

Table R5. Selection rules on identical K=5K=5 candidates and executed outcomes (2,000 states, submitted Table 6 protocol) and in closed loop on clean-only LIBERO-Plus under the same gate (three training runs). Typicality-matched pairs: 2,860 of the 11,200 outcome-discordant pairs whose medoid distance and demonstration-support score each differ by less than 0.1 training standard deviations.

Selection rulePairwise accuracy (%)Selected outcome (%)Typicality-matched pairs (%)Plus Macro-7 (%)
Uniform random50.075.0-77.0
Action-sequence medoid68.079.050.879.4
Conditional demonstration support72.080.551.581.3
Absolute / relative progress critic (submitted Table 6)70.0 / 75.078.0 / 82.0--
V-GPS-style critic / RoboMonkey-style verifier76.5 / 76.083.2 / 83.0-84.7 / 84.3
Progress-only rollout66.074.062.583.8
Consistency-only rollout74.081.066.084.9
Full score without the variance term73.080.068.581.0
Strong direct Transformer77.083.572.585.0
Full ADEPT80.085.076.588.1

The baselines. The medoid is the candidate closest to the other four in normalized action distance,

k_{\mathrm{med}}=\arg\min_k\sum_{j\ne k}\sum_{h=0}^{H-1}\bigl\Vert D_{\mathrm{tr}}^{-1}\bigl(a_h^{(k)}-a_h^{(j)}\bigr)\bigr\Vert_2^2 , \tag{R4}

with DtrD_{\mathrm{tr}} the action scaling fitted on training data. Demonstration support retrieves the 20 clean training windows with the most similar observation and language context and scores a candidate by its distance to the nearest retrieved action window. The strong direct Transformer reads the unpooled frozen visual tokens, language, and the same four-chunk action window and regresses demonstration endpoint progress without predicting states. Its trainable parameters are within 5% of ADEPT's selection modules, with the same data split and the same training FLOPs. No rule sees evaluation outcomes or LIBERO-Plus data.

What the table shows. The best candidate of each set succeeds in 90.0% of states, a finite-repeat reference. (i) Typicality recovers 4.0 to 5.5 of the 10.0 selected-outcome points between random choice and ADEPT. (ii) On pairs that both typicality rules score as nearly equal, those rules are near chance (50.8%, 51.5%), while ADEPT orders 76.5% [74.4, 78.6], 3.5 points below its accuracy on all pairs and 4.0 above the direct Transformer. Its ranking therefore carries outcome information that these typicality measures do not contain. (iii) Random selection under the gate already adds 5.5 points over the prior (71.5%) in closed loop; ADEPT adds 11.1 more. This control differs from the random gate of submitted Table 7, which randomizes when the full scorer runs.

Statistical support. ADEPT minus the strong direct scorer: 1.5 points on identical candidates (paired 95% interval [0.3, 2.7], resampling source episodes with all their candidates and repeats) and 3.1 points in closed loop (97.5% interval [0.5, 5.7], resampling initial-state clusters with all their perturbation variants; 10,000 draws). The three training runs give 88.6, 87.5, and 88.2% for ADEPT and 85.4, 84.7, and 84.9% for the direct scorer.

Revision. The closed-loop rows are in new Table 10 (Section 4.4); all selectors on identical candidates and the matched-pair analysis are in Appendix I.1 (Table 31).

Reviewer_uVJoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ADEPT, which augments a frozen VLA policy with action-sensitive latent prediction and adaptive planning. The method combines a learned token mask, multi-step rollout training, and progress-and-consistency-based candidate scoring. The reported results show substantial improvements over the frozen policy, particularly under distribution shifts and on the evaluated real-robot tasks.

Critical strengths
  • The paper addresses a practical problem: world-model predictions should help distinguish action consequences, rather than merely predict observations. The improvements over matched frozen-policy controls provide encouraging evidence that the proposed components are useful for action selection.

  • The experimental analysis is extensive. Component ablations, same-state candidate comparisons, direct-critic baselines, and matched-budget gating experiments help explain where the gains come from.

  • The evaluation includes both simulation and six real-robot tasks. The distinction between clean-training and augmentation-assisted results also makes the robustness claims easier to interpret.

Critical weaknesses
  • The additional value of the learned deliberation gate appears limited. It is trained to reproduce a heuristic scheduling target, and its advantage over directly using the heuristic is only 0–1 percentage point in the matched-budget study. This makes the contribution of learning the gate less convincing than the broader benefit of selective planning.
  • The practical benefit relative to simpler alternatives remains insufficiently established. Although adaptive planning reduces latency compared with always-on planning, it remains 3.21 times slower than reactive execution. The direct-critic comparisons are useful, but the critics are not capacity-matched to the dynamics model. Stronger direct scorers or further policy fine-tuning under comparable resource budgets would help establish when the additional complexity is justified.
  • The scoring mechanism relies on proxies whose reliability beyond demonstration-like behavior is not yet clear. Demonstration progress can penalize necessary recovery actions, and agreement between related models does not necessarily imply accurate predictions. The paper acknowledges these limitations, but more targeted evaluation on failures and recovery would strengthen its central claim about reliable action selection.
Questions for authors
  • How does ADEPT compare with a stronger direct scorer or further policy fine-tuning under matched data and computational budgets?
  • How does the method behave when successful execution requires temporarily decreasing the predicted progress, such as releasing an object and attempting a re-grasp?
  • Do the success–latency trade-offs and improvements over heuristic gating hold beyond the profiled LIBERO-Spatial setting?
5 comments
AuthorsOpenReview ↗

Response to Reviewer uVJo (1/4) - Question 1 (with Weakness 2): at matched data and training FLOPs, a capacity-matched strong direct scorer reaches 85.0% and policy fine-tuning 85.1% vs 88.1% for ADEPT; the two compose (88.9%); where the 3.21x goes

Comment 1: [Question 1] How does ADEPT compare with a stronger direct scorer or further policy fine-tuning under matched data and computational budgets? [Weakness 2] [...] it remains 3.21 times slower than reactive execution. [...] the critics are not capacity-matched to the dynamics model. Stronger direct scorers or further policy fine-tuning under comparable resource budgets would help establish when the additional complexity is justified.

Response 1: At matched data and training compute, a strong direct scorer with matched capacity reaches 85.0% and continued policy fine-tuning 85.1%, against 88.1% for ADEPT (all 10,030 clean-only LIBERO-Plus variants, three training runs each; both intervals exclude zero). Three quarters of the latency added to the reactive call is candidate sampling by the prior, which any selector over KK proposals pays.

Table U1. Matched-budget controls on clean-only LIBERO-Plus (Macro-7, three-run means; complete server-side decision latency). Every method receives 3B3B search FLOPs and three final training runs of BB FLOPs. Differences are ADEPT minus row with paired intervals over initial-state clusters (* co-primary, 97.5% each; others 95%).

ControllerMacro-7 (%)Mean / p95 (ms)ADEPT minus row (pp)Per-run (%)
Continued policy fine-tuning, reactive*85.190 / 1013.0 [0.4, 5.6]85.5 / 84.6 / 85.2
Strong direct scorer, shared gate*85.0269 / 4013.1 [0.5, 5.7]85.4 / 84.7 / 84.9
Direct scorer, always-on, no dynamics anywhere85.0383 / 4063.1 [0.7, 5.5]-
Fine-tuned policy + direct selector87.0280 / 3991.1 [0.1, 2.1]-
ADEPT88.1292 / 434-88.6 / 87.5 / 88.2
Fine-tuned policy + ADEPT modules, same split88.9292 / 434−0.8-0.8-

Strong direct scorer. A Transformer over the complete unpooled frozen visual tokens, language, and the same four-chunk action window, with the dynamics model's block design, regressing the same demonstration endpoint progress but without predicting states. It does not depend on ADEPT's mask. Trainable parameters are within 5% of ADEPT's selection modules; data split, search budget, and cumulative training FLOPs (including ADEPT's recursive predictions) are the same. On identical candidates and outcomes it selects 83.5% against 85.0% for ADEPT (+1.5+1.5 [0.3, 2.7]).

Policy fine-tuning. Nine configurations (action-expert updates, rank-32 adapters, or joint backbone updates; three learning rates), selected on clean validation data only; the selected one updates the action expert at 3×10−53\times10^{-5} with the backbone frozen. The hybrid splits each budget into 0.5B0.5B for the policy and 0.5B0.5B for the direct selector and runs under the shared gate (64.3% invocation).

Reading. Both co-primary intervals exclude zero, and the lowest ADEPT run (87.5%) is above the highest run of either control. Fine-tuning and predictive selection compose. The selected fine-tuning keeps the backbone frozen, so the visual tokens that ADEPT's mask and dynamics read are unchanged. The combined controller uses the hybrid's allocation (0.5B0.5B for the policy, 0.5B0.5B for ADEPT's modules), the same three fine-tuned checkpoints, and the same gate; it reaches 88.9%, 1.9 points above the direct selector on the same policy ([0.7, 3.1]). Its 0.8-point difference from ADEPT on the frozen prior changes the policy and the budget split together and is not a pure effect of fine-tuning. The margin over the hybrid is modest (1.1 points).

When the added complexity is justified. Not at a 90 ms budget, where fine-tuning is the option (85.1%). ADEPT matches that level at 192 ms (85.2%, 30% budget) and exceeds it by 3.0 points at 292 ms, where the two also compose (88.9%).

Where the 3.21 times goes. Of the 199.3 ms that adaptive planning adds to the 90.0 ms reactive call (submitted Table 14), 150.7 ms is additional candidate sampling by the frozen prior, which every selector over KK proposals pays. The gate, rollout, and scoring take 47.7 ms, and encoding 0.9 ms. Under the same gate ADEPT needs 23 ms more than the direct scorer (292 against 269 ms). The controller with no dynamics anywhere ranks at every decision and is slower (383 ms). At equal measured latency ADEPT is 2.4 to 2.7 points above the direct scorer at 192, 269, and 292 ms; at 292 ms the direct scorer was enlarged to 1.7 times the parameters and training FLOPs and gained 0.4 points.

Scope and revision. These results concern the tested direct scorers and fine-tuning recipes at the stated budget, not every possible alternative. The fine-tuning rows are in new Table 10 and Section 4.4; the full table and the budget protocol are in Appendix I.4 (Table 36).

AuthorsOpenReview ↗

Response to Reviewer uVJo (2/4) - Question 2 (with Weakness 3): recovery measured directly: 65.0% closed-loop recovery vs 52.0% for the prior over 3,000 episodes; the consistency penalty helps on net and its failure modes are quantified

Comment 2: [Question 2] How does the method behave when successful execution requires temporarily decreasing the predicted progress, such as releasing an object and attempting a re-grasp? [Weakness 3] [...] Demonstration progress can penalize necessary recovery actions, and agreement between related models does not necessarily imply accurate predictions. [...] more targeted evaluation on failures and recovery would strengthen its central claim about reliable action selection.

Response 2: A temporary progress decrease is not penalized: the score reads only the terminal state of the rollout, and over 3,000 closed-loop recovery episodes ADEPT reaches 65.0% against 52.0% for the prior and 62.0% for the strong direct scorer. Both risks you name are nevertheless real, and we measured them: in 4.0% of recovery-eligible states the penalty removes a recovery that progress alone would make, and 12.0% of low-disagreement candidates are wrong in both models.

What the score penalizes. The score reads only the terminal state of the four-chunk rollout:

S(A)=V_{\mathrm{task}}(\hat z_H,\ell)-\lambda_{\mathrm{rc}}C(A),\qquad C(A)=\frac1H\sum_{h=0}^{H-1}\bigl\Vert f_{\mathrm{dyn}}(\hat z_h,a_h)-f_{\mathrm{dyn}}^{\mathrm{EMA}}(\hat z_h,a_h)\bigr\Vert_2^2 , \tag{U1}

with H=4H=4 and λrc=0.5\lambda_{\mathrm{rc}}=0.5; both models read the online rollout's preceding state. A recovery candidate ARA_R is preferred to a competitor AFA_F exactly when

V_{\mathrm{task}}(\hat z_H^R,\ell)-V_{\mathrm{task}}(\hat z_H^F,\ell) > \lambda_{\mathrm{rc}}\bigl[C(A_R)-C(A_F)\bigr]. \tag{U2}

Recovery can still fail if the useful action is not among the samples, if its benefit lies beyond HH, if the progress head undervalues the recovered state, or if the penalty suppresses an unfamiliar trajectory. The progress target is normalized demonstration time, not success probability.

Evaluation. Three physical failure strata (lost target contact, unstable grasp, obstructed placement), assigned independently of any score. A physical recovery is an executed release followed by a renewed stable grasp. Whether a candidate pool contains a successful recovery branch is determined by screening executions that are separate from the evaluation executions.

Table U2. Closed-loop recovery from 3,000 paired initial-state blocks (equal weight on the three strata; all controllers replan to the same episode limit; all initial states count, including those without a recovery branch). Paired intervals over initial-state clusters: 97.5% for the two scorer contrasts, 95% for the prior.

ControllerSuccess (%)ADEPT minus row (pp)Rescue / harm (%)Interval (pp)
Frozen prior, reactive52.013.020.5 / 7.5[10.7, 15.3]
Progress-only scorer (λrc=0\lambda_{\mathrm{rc}}=0, same models)60.05.012.5 / 7.5[2.9, 7.1]
Strong direct scorer62.03.011.5 / 8.5[0.8, 5.2]
ADEPT65.0---

Does progress scoring reject recovery? In 70.0% of failure states the pool contains a successful recovery branch, and in 60.0% of those the successful branch has an intermediate progress decrease (42.0% of all failure states; we call these states recovery-eligible). On them, with identical models and candidates and only λrc\lambda_{\mathrm{rc}} changed from 0 to 0.5, the joint score succeeds in 75.0% against 65.0% for progress alone.

Table U3. Paired outcomes on this stratum (share of its states, %).

Joint scoreProgress-only succeedsProgress-only fails
Succeeds60.015.0
Fails5.020.0

Does the penalty suppress recovery? Sometimes. On the recovery-eligible states it turns a failure into a success in 15.0% and a success into a failure in 5.0% (Table U3); in 4.0% of them the lost success is a recovery that progress alone would have made. The net effect is positive.

Does agreement imply accuracy? No. On all candidates of the recovery cohort we compared the online and the EMA prediction with the encoded executed observation zobsz^{\mathrm{obs}}. A candidate has a shared large error when both Eon=1H∑h∥z^h+1−zh+1obs∥22E_{\mathrm{on}}=\frac1H\sum_h\Vert\hat z_{h+1}-z^{\mathrm{obs}}_{h+1}\Vert_2^2 and its EMA counterpart EEMAE_{\mathrm{EMA}} exceed τE\tau_E.

Table U4. Large prediction error shared by the online and EMA models, by disagreement (thresholds fixed on clean validation data).

DisagreementShare of candidatesShared large error
Low (C≤τCC\leq\tau_C)0.6512.0%
High (C>τCC > \tau_C)0.3545.0%

Disagreement discriminates (risk ratio 3.75), but 12.0% of low-disagreement candidates are wrong in both models. The penalty is a ranking regularizer, not calibrated uncertainty.

Revision. Section 4.4 reports the closed-loop result, Appendix I.5 gives Table U2 (new Table 37) and both analyses, and the limitations state the two failure modes.

AuthorsOpenReview ↗

Response to Reviewer uVJo (3/4) - Question 3: the success-latency trade-off holds on LIBERO-Plus, RoboCasa365 and six robot tasks; the improvement over heuristic gating does not (below one point, intervals contain zero)

Comment 3: [Question 3] Do the success-latency trade-offs and improvements over heuristic gating hold beyond the profiled LIBERO-Spatial setting?

Response 3: The success-latency trade-off holds on LIBERO-Plus, RoboCasa365, and the six robot tasks: against always-on use of the same scorer, mean latency is 24.9 to 31.0% lower and success is 1.3 to 3.7 points higher. The improvement over heuristic gating does not hold: it is below one point in every setting and every interval contains zero.

Table U5. Same scorer, different invocation rules, at one operating point per setting. Success: Macro-7 (clean-only LIBERO-Plus, three training runs), split-macro (RoboCasa365), equal-task success (six robot tasks, 300 matched scene blocks). Latency: mean / p95 per server-side decision on one A100, in ms.

ControllerPlus successPlus latencyRoboCasa successRoboCasa latencyRobot successRobot latency
Reactive prior71.590 / 9916.094 / 10455.3100 / 115
Always-on85.8423 / 44824.2441 / 47386.0455 / 510
Random gate84.6291 / 43423.8331 / 45984.7321 / 477
Heuristic gate87.5291 / 43325.1330 / 45789.0322 / 479
Learned gate (ADEPT)88.1292 / 43425.5331 / 45889.7323 / 480

Realized invocation is 60.8% on LIBERO-Plus, 68.4% on RoboCasa365, and 62.4% on the robots; gate thresholds are fixed on clean validation data before evaluation.

Table U6. Paired contrasts by setting (95% intervals over initial-state clusters or scene blocks; success tolerance ϵ=1.0\epsilon=1.0 point fixed before evaluation).

SettingLearned minus always-on, success (pp)Learned minus always-on, latency (ms)Learned minus heuristic, success (pp)
Clean-only LIBERO-Plus2.3 [0.2, 4.4]−131-131 [−134-134, −128-128]0.6 [−1.3-1.3, 2.5]
RoboCasa3651.3 [−0.3-0.3, 2.9]−110-110 [−114-114, −106-106]0.4 [−1.1-1.1, 1.9]
Six robot tasks3.7 [0.5, 6.9]−132-132 [−141-141, −123-123]0.7 [−1.9-1.9, 3.3]

Trade-off against always-on. On LIBERO-Plus and on the robot tasks, selective invocation is both more successful and cheaper than always-on use of the same scorer: the success intervals exclude zero and the latency intervals lie below zero. On RoboCasa365 the success interval contains zero and lies above −ϵ-\epsilon; the supported statement there is 24.9% lower decision cost with no loss of success beyond one point. Mean latency falls by 31.0%, 24.9%, and 29.0%. These are per-setting statements; we make no joint claim across the three.

Improvement over heuristic gating. It does not hold as an improvement. The differences are 0.6, 0.4, and 0.7 points, every interval contains zero, and none lies inside [−ϵ-\epsilon, ϵ\epsilon]: the learned gate is neither shown to be better nor shown to be equivalent. The heuristic costs about 1 ms less.

The curve on the harder suite. At 15%, 30%, and 60% budgets on LIBERO-Plus, ADEPT reaches 80.2%, 85.2%, and 88.1% at 142, 192, and 292 ms (heuristic 79.8, 84.7, 87.5; random gate 75.3, 79.7, 84.6), against 71.5% at 90 ms for the prior and 85.8% at 423 ms for always-on scoring. The shape matches the submitted Spatial profile (92%, 96%, 98% at 141, 190, 289 ms).

Scope. Relative to the reactive prior, ADEPT is a higher-success, higher-latency operating point in every setting (3.1 to 3.5 times the latency). The gate lowers how often the expensive branch runs, not its tail: the p95 stays near the always-on mean. Timing covers the server-side decision on one A100 with each embodiment's own cameras and action dimension, and excludes camera acquisition, transport, and actuation. Per-hardware rows are in Response 5 to Reviewer qaz2.

Revision. The cross-setting table and the budget results are in Appendix I.6 (Table 38); the abstract reports the reduction against always-on planning (25-32%) together with the cost relative to reactive execution; the new Section 4.4 separates the two scheduling statements.

Reviewer_qaz2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

ADEPT adds a latent world model and candidate reranker to a frozen π0.5 policy. A learned token mask (inverse-dynamics, action-variance and budget losses) defines the predicted state; dynamics train on 4-step autoregressive rollouts. A gate decides when to sample K=5 prior candidates, which are ranked by predicted demonstration progress minus an EMA-model disagreement penalty. The paper reports 99.1% on LIBERO, 88.1% Macro-7 on LIBERO-Plus with clean-only training (frozen prior: 71.5%), 25.5% on RoboCasa365, a real-robot improvement from 55.3% to 90.3% pooled over six tasks, and 31% lower latency than always-on planning.

Critical strengths
  1. The same-state candidate-ranking protocol (Table 6, App. G/H) is the right way to test a world-model scorer. Restored simulator state, shared candidates and executed-success labels separate proposal support from ranking quality.
  2. The clean-only LIBERO-Plus setting is a meaningful transfer test. Prior and encoder are frozen and nothing touches the perturbations. If the gains hold, reranking recovers robustness the policy lacks.
Critical weaknesses

In short: ADEPT is a reasonable combination of established ideas, but the paper does not position it against the robotics work it builds on or test it against the closest methods. Its evidence is hard to follow and too small to rank the components.

  1. The method combines known techniques, and the paper neither cites nor compares against the closest prior work, so its contribution and the world model's role are unclear. At deployment, ADEPT samples K candidates from a frozen VLA, scores them, and executes the best one.

    • Selection: This is the recipe of V-GPS (Nakamoto et al., CoRL 2024) and RoboMonkey (Kwok et al., CoRL 2025). FOREWARN (Wu et al., RSS 2025) also uses a latent world model to imagine each candidate's outcome before choosing. More broadly, it is sampling-based MPC over learned action-conditioned prediction: Visual Foresight (Finn & Levine, 2017; Ebert et al., 2018) and PlaNet (Hafner et al., 2019).
    • Representation: Using inverse dynamics to keep only action-relevant latent factors follows ICM (Pathak et al., 2017) and multi-step inverse models (Lamb et al., TMLR 2023).
    • Rollout training and scoring: Training on multi-step rollouts to limit compounding error follows DaD (Venkatraman et al., 2015) and PlaNet's latent overshooting. Penalizing model disagreement when scoring imagined rollouts follows PETS (Chua et al., 2018) and MOPO/MOReL (2020).
    • When to plan: Deciding when to plan is the subject of metacontrol (Hamrick et al., 2017) and event-triggered control.

    None of these is cited; the related work cites mainly 2025–26 preprints. The experiments also do not show that the world model matters relative to these simpler selectors. Best-of-K with a direct critic appears only on LIBERO-Spatial (Table 7) and in the same-state diagnostic. LIBERO-Plus, RoboCasa and the robot, where the large claims are made, compare only against the frozen prior, and Table 6 has no random-selection row. Always-on deliberation (90%) also does worse than gated deliberation (98%) with the same scorer (Table 7; Table 10 shows the same pattern), which a scorer better than chance should not do. Needed: Position ADEPT against this literature and state what is new beyond the combination (the token mask over frozen VLA features is the clearest candidate). On clean LIBERO-Plus and the robot tasks, compare at the same K and invocation rate against random selection, a V-GPS-style value critic, and a RoboMonkey-style verifier.

  2. The evaluation is hard to follow, and the component evidence is too small to rank variants.

    • Clarity: Results are spread over more than 20 tables that mix published baseline numbers, the authors' own runs, n=50 and n=200 protocols, clean and augmented training, and recomputed vs. reported aggregates. Six method-specific acronyms (WAM, CLF, MRT, ADV, Macro-7, split-macro) and frequent caveats about what each result does not show make the main comparison hard to locate. The motivating objective (Eq. 4) is also not the one implemented (Eqs. 5–6).
    • Power: The gate and scorer ablations use 3×50 episodes on LIBERO-Spatial near ceiling, and the claimed effects are 6–12 points. The same configuration is reported at 98% (Table 7), 97% (Table 9) and 94% (Table 11), a spread close to the 6-point gate effect.
    • Comparability: Table 1 compares ADEPT's 100% on Spatial with π0.5's published 97.4%, while the authors' own run of that frozen prior scores 88% (Table 7).

    Needed: Put one table up front with ADEPT, the frozen prior and the selection baselines under a single protocol. Run the gate and scorer ablations on a harder suite with paired intervals (on the order of 500 episodes per arm).

Questions for authors
  1. What does ADEPT add beyond V-GPS, RoboMonkey and FOREWARN, and how does it compare with them at the same K and compute on LIBERO-Plus?
  2. Why does always-on deliberation with the full scorer (Table 7, 90%; Table 10, 98.2/87.5) do worse than gated deliberation? What does the gate filter that the scorer gets wrong?
  3. For real-robot task T3, were the mask, dynamics and value head trained on T3 teleoperation data that the prior never saw? Were trials interleaved, matched on initial conditions, and scored blind?
  4. The 31% latency reduction is measured only on LIBERO-Spatial on one A100. What are the invocation rates and latencies on LIBERO-Plus, RoboCasa and the robots?
8 comments
AuthorsOpenReview ↗

Response to Reviewer qaz2 (1/6) - Question 1, part 1 of 2 (with Weakness 1, positioning): what ADEPT adds beyond V-GPS, RoboMonkey and FOREWARN (a learned criterion for which frozen VLA tokens to predict); what is adopted; Eq. 4 vs Eqs. 5-6

Comment 1: [Question 1] What does ADEPT add beyond V-GPS, RoboMonkey and FOREWARN [...]? [Weakness 1] The method combines known techniques, and the paper neither cites nor compares against the closest prior work, so its contribution and the world model's role are unclear. [...] Position ADEPT against this literature and state what is new beyond the combination (the token mask over frozen VLA features is the clearest candidate). [Weakness 2] The motivating objective (Eq. 4) is also not the one implemented (Eqs. 5-6).

Response 1: What ADEPT adds beyond V-GPS, RoboMonkey and FOREWARN is a learned criterion for which part of a frozen VLA representation the world model predicts and scores. V-GPS and RoboMonkey score candidate actions without predicting their consequences; FOREWARN predicts the full latent. The rest is adopted, and the revision now cites every work you name.

Table Q1. Positioning. The last column states whether a method learns which part of its representation is predicted and scored.

MethodSelection signalPredicted consequenceLearned predicted subset
V-GPSOffline-RL value Q(s,a,ℓ)Q(s,a,\ell) over policy samplesNoneNo
RoboMonkeyVLM verifier trained on action preferences; Gaussian proposalsNoneNo
FOREWARNLatent world-model rollout narrated and selected by a VLMFull latentNo
Visual Foresight / PlaNetVideo or latent prediction with a cost or reward, CEM searchPixels / full latentNo
ADEPTProgress on a masked-latent rollout minus EMA consistencyMasked latentYes

FOREWARN is a direct conceptual precedent for latent foresight; ADEPT differs in the last column, the learned predicted state over frozen VLA tokens. Quantitative comparisons with all three are in Response 2.

The criterion. Inverse dynamics (Eq. 3; the ICM criterion, with multi-step inverse models as its kk-step form) keeps a token when the executed action is decodable from consecutive states. It compares states along one trajectory. It never compares the predicted futures of two different actions from the same state, which is what a ranker does over KK candidates. Eqs. 5-6 add exactly this requirement: at a fixed context, the predictions for M=8M=8 actions of the frozen prior must differ. Trained jointly with recursive rollout at the deployment horizon (Eq. 9), the mask keeps tokens that are separable across the prior's samples and predictable under the scorer's recursion.

Adopted. Sampling and reranking a frozen policy's actions, latent foresight, multi-step rollout training (DaD, latent overshooting), disagreement penalties (PETS, MOPO, MOReL), and on-demand computation (metacontrol, event-triggered control). The EMA consistency term is a ranking regularizer, not calibrated uncertainty.

Evidence that the criterion matters (clean-only LIBERO-Plus Macro-7; same prior, KK, and horizon): full 88.1%; without the mask 78.4%; without the variance term 81.0%; single-step training 81.2%; teacher forcing 83.6%. On identical candidates, removing the variance term lowers pairwise accuracy from 80.0% to 73.0% and the selected outcome from 85.0% to 80.0%. The effect is not the budget: at the same realized mean weight (0.250 fixed, 0.251 learned), a fixed robot-centered mask without task-object tokens gives 81.4%, with them 86.2%, and the learned mask 88.1%.

Eq. 4 and Eqs. 5-6 (Weakness 2). Eq. 4 is motivation; Eqs. 5-6 are the trained objective, L^ai=−12σ2M∑j=1M∥μj−μˉ∥2\widehat{\mathcal{L}}_{ai}=-\frac{1}{2\sigma^2M}\sum_{j=1}^{M}\Vert\mu_j-\bar\mu\Vert^2 (σ=1\sigma=1; M=8M=8 training draws, distinct from the deployment K=5K=5), which estimates the trace in the identity below. With pa=N(μa,σ2I)p_a=\mathcal N(\mu_a,\sigma^2I), pmarg=EApAp_{\mathrm{marg}}=\mathbb{E}_A p_A, and qref=N(EAμA,σ2I)q_{\mathrm{ref}}=\mathcal N(\mathbb{E}_A\mu_A,\sigma^2I),

\mathbb{E}_A D_{\mathrm{KL}}(p_A\Vert q_{\mathrm{ref}}) =\frac{\operatorname{tr}\operatorname{Cov}_A(\mu_A)}{2\sigma^2} =I(A;Z'\mid c)+D_{\mathrm{KL}}(p_{\mathrm{marg}}\Vert q_{\mathrm{ref}}). \tag{Q1}

The second term depends on the learned predictions, so maximizing the surrogate is not equivalent to maximizing mutual information (Proposition A.1 of the submission). It supplies an action-sensitive inductive bias; neither it nor inverse dynamics proves task sufficiency. Submitted Table 27 checks that the gain is not norm growth: from step 5k to 30k the prediction RMS changes from 0.950 to 1.050 while unit-normalized candidate separation rises from 0.080 to 0.120 and the selected outcome from 77.5% to 85.0%.

Revision. Section 2 now cites the works you name and states what the variance term adds to an inverse-dynamics criterion; Section 3.2 points to the decomposition where Eqs. 5-6 are introduced; the comparisons are in the new Section 4.4 (Table 10) and Appendix I.1.

AuthorsOpenReview ↗

Response to Reviewer qaz2 (2/6) - Question 1, part 2 of 2 (with Weakness 1, requested comparisons): same-K, same-gate comparison with random, V-GPS-style and RoboMonkey-style selection and LIBERO adaptations of the three systems; equal latency

Comment 2: [Question 1] What does ADEPT add beyond V-GPS, RoboMonkey and FOREWARN, and how does it compare with them at the same KK and compute on LIBERO-Plus? [Weakness 1] Table 6 has no random-selection row. [...] On clean LIBERO-Plus and the robot tasks, compare at the same KK and invocation rate against random selection, a V-GPS-style value critic, and a RoboMonkey-style verifier.

Response 2: At the same K=5K=5 on clean-only LIBERO-Plus, ADEPT reaches 88.1% against 82.3, 83.1, and 81.4% for V-GPS, RoboMonkey, and FOREWARN retrained on LIBERO with their own architectures, and against 85.0% for the strongest matched direct scorer. At equal latency it stays 2.4 to 2.7 points above the direct scorer. The table below also holds random selection and the two selectors you request.

Table Q2. Clean-only LIBERO-Plus (10,030 variants; standard-LIBERO training only; same frozen prior, K=5K=5, H=4H=4, same frozen gate). Macro-7: mean of three training runs (one run for the three adaptations). Differences: ADEPT minus row with paired intervals over initial-state clusters (97.5% for the co-primary contrast *, 95% otherwise). Same-state: selected outcome on the identical candidates of submitted Table 6.

SelectorMacro-7 (%)Invocation (%)Latency (ms)ADEPT minus row (pp)Same-state (%)
Frozen prior71.509016.6-
Random selection77.060.124611.175.0
Action medoid79.460.02478.779.0
Demonstration support81.360.32516.880.5
V-GPS-style value critic84.760.52713.483.2
RoboMonkey-style verifier84.360.22673.883.0
Strong direct scorer*85.060.42693.1 [0.5, 5.7]83.5
V-GPS adaptation (Cal-QL, its Q-network)82.3-2585.8 [3.0, 8.6]81.6
RoboMonkey adaptation (LLaVA-7B verifier)83.1-3715.0 [2.2, 7.8]82.2
FOREWARN adaptation (DreamerV3 + VLM)81.4-2,0906.7 [3.8, 9.6]80.8
ADEPT88.160.8292-85.0

What is held fixed. At a restored state the tentative action, the gate decision, and the candidates are identical for every selector; only the ranking changes. All selectors train on the same episode-disjoint 90/10 standard-LIBERO split and see no LIBERO-Plus data.

Matched controls. The value critic, the verifier, and the direct scorer read the unpooled frozen visual tokens, language, and the same four-chunk action window, use the dynamics model's Transformer blocks, and match ADEPT's selection modules in trainable parameters (within 5%) and training FLOPs. The critic uses expectile value regression with temporal-difference Q-learning on a sparse demonstration-endpoint reward; the verifier a Bradley-Terry loss on same-state preferences ordered by distance to the demonstrated action; the direct scorer regresses demonstration endpoint progress. They isolate the scoring mechanism and are protocol adaptations, not the published systems.

Published architectures. Released weights target other robots, so we retrained each system with its released implementation on standard LIBERO: V-GPS (default Cal-QL, ResNet-34 Q-network with FiLM, sparse reward, greedy ranking), RoboMonkey (LLaVA-7B verifier with reward head, synthetic action-preference training), and FOREWARN (DreamerV3 world model, Llama-3.2-11B-Vision narration and selection; K=5K=5 prior samples replace its six aggregated modes, and narrations come from simulator predicates). FOREWARN also trains on prior rollouts with failures, which ADEPT does not use. Each scores the same four-chunk window. With each system's own candidate procedure at every decision (V-GPS K=10K=10; RoboMonkey 5-to-16 Gaussian proposal with gripper voting; FOREWARN K=5K=5), Macro-7 is 80.6, 81.9, and 79.8% at 684, 872, and 3,410 ms. ADEPT's lowest run (87.5%) is 4.4 to 6.1 points above the three adaptations. Each is one training run, on this benchmark only.

Same KK is not same compute. Table Q3 compares at equal measured mean decision latency (gate thresholds set on clean validation data). Spending the extra 23 ms on a larger direct scorer returns 0.4 points; spending it on prediction returns 3.1.

Table Q3. Equal measured decision latency on clean-only LIBERO-Plus (Macro-7, three-run means; paired 95% intervals). At 292 ms the direct scorer is enlarged to 1.7 times the parameters and training FLOPs.

Mean latency (ms)ADEPT (%)Direct scorer (%)ADEPT minus direct (pp)
19285.282.82.4 [0.2, 4.6]
26987.585.02.5 [0.3, 4.7]
29288.185.42.7 [0.5, 4.9]

Reading. Random selection under the gate adds 5.5 points over the prior; typicality adds 2.4 to 4.3 more; strong direct selectors reach 84.3 to 85.0%; predictive scoring adds 3.1. The robot counterpart is in Response 4; see new Table 10 (Section 4.4) and Appendix I.1.

AuthorsOpenReview ↗

Response to Reviewer qaz2 (3/6) - Question 2: the gate withholds replacements with negative benefit (first-chunk intervention: +9.0 points where it invokes, -4.0 where it skips)

Comment 3: [Question 2] Why does always-on deliberation with the full scorer (Table 7, 90%; Table 10, 98.2/87.5) do worse than gated deliberation? What does the gate filter that the scorer gets wrong?

Response 3: The gate filters states where the tentative action is already good: there it succeeds 91.0% of the time and a forced replacement loses 4.0 points, whereas replacement gains 9.0 points where the gate invokes. Ranking above chance within a candidate pool does not make replacement beneficial at every state.

Mechanism. At a state xx, let Δ(x)\Delta(x) be the success difference between executing the scorer-selected first chunk and the tentative first chunk, and g(x)∈0,1g(x)\in\\{0,1\\} the gate. Then

G_{\mathrm{gate}}-G_{\mathrm{all}}=-\mathbb{E}[(1-g)\Delta],\qquad G_{\mathrm{gate}}-G_{\mathrm{rand}}=\operatorname{Cov}(g,\Delta). \tag{Q2}

A gate improves success while reducing computation when the states it skips have negative replacement benefit. In the deployed controller the tentative action is one of the K=5K=5 candidates and is replaced only when another candidate outscores it. Negative benefit therefore has one source, scoring error: predicted progress or the consistency penalty places a worse candidate above a tentative action that is already good, where a replacement has little to gain and much to lose.

Measurement. From each of the 2,000 saved states (200 per task) we executed the tentative and the selected first chunk with the same continuation policy and five matched repeats. The gate decision and the scores are frozen before either branch runs.

Table Q4. First-chunk intervention outcomes, conditional on the frozen gate. Rescue: selected succeeds where tentative fails; harm: the reverse.

Gate decisionStatesTentative (%)Selected (%)Rescue (%)Harm (%)Net (pp)
Invoke1,20076.085.013.04.0+9.0+9.0
Skip80091.087.02.06.0−4.0-4.0
All2,00082.085.88.64.8+3.8+3.8

On skip states the tentative action already succeeds 91.0% of the time, and forced replacement harms three times as often as it rescues. Always-on execution takes these replacements; the gate withholds them. With the gate invoking at 60% of the states, Table Q4 gives 0.6(85.0)+0.4(91.0)=87.40.6(85.0)+0.4(91.0)=87.4% for the gated rule, 0.6(85.0)+0.4(87.0)=85.80.6(85.0)+0.4(87.0)=85.8% for always-on, and 82.0+0.6(3.8)=84.382.0+0.6(3.8)=84.3% for a random gate at the same rate.

Closed loop. On clean-only LIBERO-Plus (10,030 variants, three training runs) the same scorer gives 88.1% gated, 85.8% always-on, and 84.6% with a random gate; gated minus always-on is 2.3 points (paired 95% interval [0.2, 4.4]), and it holds in each run (2.5, 2.1, 2.3). The submitted Table 10 row you cite (98.2/87.5 against 99.1/90.2) is the gate-only removal and shows the same sign. The 8-point gap of Table 7 comes from 150 episodes on 50 initial states per controller; the ordering of these controllers now rests on the LIBERO-Plus comparison above (new Table 10).

What we do not claim. The gate is trained to reproduce a scheduling heuristic, not the realized Δ\Delta. It exceeds that heuristic by 0.6 points on LIBERO-Plus with an interval containing zero ([−1.3-1.3, 2.5]). The contribution is selective use of a useful scorer, not a learned gate.

Revision. The new Section 4.4 reports this table (Table 11); Section 3.4 and Algorithm 1 state that the tentative sequence is one of the KK candidates and is executed unless another candidate scores higher; the protocol is in Appendix I.3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to AC and all reviewers: what is new, one-protocol comparison of every controller, and summary of the revision

We thank the Area Chair and the reviewers. The reviews share one question: does the gain come from the learned predictive model, or would any selector over the same KK samples of the frozen prior obtain it? Under one protocol, ADEPT reaches 88.1% against 85.0% for the strongest matched direct selector and 81.4 to 83.1% for V-GPS, RoboMonkey, and FOREWARN retrained on LIBERO (Table G1). Each reviewer thread below answers that reviewer's points in order.

New and adopted. New: a criterion for which part of a frozen VLA's token representation a short-horizon model should predict, namely a budgeted token mask trained with inverse dynamics and a between-action predictive-variance term (Eqs. 5-6), jointly with recursive rollout at the deployment horizon. Adopted, and now cited and positioned: sample-and-rerank (V-GPS, RoboMonkey), latent foresight (FOREWARN, Visual Foresight, PlaNet), multi-step rollout training (DaD), disagreement penalties (PETS, MOPO, MOReL), and on-demand computation (metacontrol).

Table G1. One protocol for every controller: clean-only LIBERO-Plus, all 10,030 variants, standard-LIBERO training only, K=5K=5, H=4H=4. Macro-7 is the mean of three training runs (one run for the three published-system adaptations). Intervals are paired over initial-state clusters (97.5% for the two co-primary contrasts marked *, 95% otherwise).

ControllerMacro-7 (%)ADEPT minus row (pp)Latency (ms)
Frozen prior, reactive71.516.690
Continued policy fine-tuning, same budget*85.13.0 [0.4, 5.6]90
Shared gate + random selection77.011.1246
Shared gate + action medoid / demonstration support79.4 / 81.38.7 / 6.8247 / 251
Shared gate + V-GPS-style critic / RoboMonkey-style verifier84.7 / 84.33.4 / 3.8271 / 267
Shared gate + strong direct scorer (matched capacity and FLOPs)*85.03.1 [0.5, 5.7]269
LIBERO adaptations of V-GPS / RoboMonkey / FOREWARN (own architectures)82.3 / 83.1 / 81.45.8 / 5.0 / 6.7258 / 371 / 2,090
ADEPT scorer, always-on85.82.3 [0.2, 4.4]423
ADEPT scorer, random gate / heuristic gate84.6 / 87.53.5 / 0.6 [−1.3-1.3, 2.5]291 / 291
ADEPT88.1-292

Main answers.

  • Is it the predictive model? ADEPT exceeds the strongest direct selector by 3.1 points in closed loop, by 1.5 points [0.3, 2.7] on identical candidates and outcomes, and by 2.4 to 2.7 points at equal measured decision latency. The lowest of three ADEPT runs (87.5%) is above the highest direct-scorer run (85.4%).

  • What is predicted? Against executed object states, the error of predicted candidate differences falls from 3.4 to 2.1 cm, and candidate pairs with matched robot motion are ranked at 78.0% against 68.0%. The mask weight on task-object tokens rises from 0.40 in free space to 0.77 at contact.

  • Why does always-on trail gated? Executing the tentative and the selected first chunk from the same state: replacement changes success by +9.0+9.0 points where the gate invokes and by −4.0-4.0 points where it skips.

  • Learned gate. It is within 0.6 points of the heuristic it distills and the interval contains zero. We claim selective invocation, not a better learned gate.

  • Fine-tuning and recovery. ADEPT's modules on the fine-tuned policy reach 88.9%. Over 3,000 closed-loop recovery episodes ADEPT reaches 65.0% against 52.0% for the prior.

  • Comparability. Our own run of the frozen prior under Table 1's protocol gives 96.0% on LIBERO (same-protocol gain 3.1 points). On a six-task robot cohort with equal task-data access, the direct scorer reaches 252/300 and ADEPT 269/300; the paper now states that ADEPT's own modules are trained on T3 demonstrations.

Revised PDF. New and changed text is marked in blue. Ten pages of main text. Every figure, table, and equation of the submitted main text keeps its place and number. Section 2 now cites and positions ADEPT against every work named in the reviews. The additional page is a new Section 4.4 with three tables: every controller under one protocol (Table 10), the first-chunk intervention (Table 11), and object consequences (Table 12). A new Appendix I (Tables 31-41) holds the protocols and further results; appendix tables of the submission shift by three (submitted Table 26 is now Table 29).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.