acceptodds
Under review as a conference paper at ICLR 2027

Reasoning as a Distribution: Generative Plan Latents for Vision–Language–Action Models

Abstract

Vision–Language–Action (VLA) policies increasingly reason before acting, but existing reasoning models compress this intermediate decision into a single representation. When several plans are valid, this commitment hides alternatives that could otherwise be selected or revised. We introduce the Generative Plan Latent (GPL), which represents the reasoning state as a low-dimensional conditional distribution over plans, such as which object, grasp, or route to use. This makes the reasoning state addressable: without retraining the VLA, plans can be sampled (coverage), selected on request (steering), and reweighted by deployment outcomes (outcome adaptation). We test all three on SimplerEnv manipulation with , and extend the analysis to RoboCasa, LIBERO, and a maze domain. Steering the plan raises the fraction of executions that follow the request from 0.48 to 0.94, while the same guidance on the native action distribution shows no reliable gain. Finally, we show that an explicit plan is most useful when it carries decision information not already determined by the observation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Latent Reasoning, Generative Planning, Robot Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yQDmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that the intermediate reasoning state of a Vision-Language-Action (VLA) policy should be represented not as a single committed vector but as a conditional distribution over task-level plans. The authors introduce the Generative Plan Latent (GPL): a small rectified-flow head that samples a low-dimensional plan from a frozen VLA's conditioning representation, projects it into soft tokens, and conditions the action expert on it, while the backbone stays frozen and only LoRA plus a projector are trained. This makes the reasoning state addressable along three axes: coverage, steering, and outcome adaptation. The paper further proposes a redundancy principle: an explicit plan helps only when it carries decision information the observation does not already determine, measured by a redundancy score. Experiments span SimplerEnv, RoboCasa, LIBERO, and a maze domain, with both hand-designed and behavior-learned plan coordinates.

Critical strengths
  • The paper targets a real and under-examined blind spot: existing VLAs may already be generative at the action level, but their upstream decision is still collapsed into one point per observation. The authors turns the intermediate decision into a distribution that can be sampled, steered, and reweighted, rather than merely a decided output, which is appealing.
  • The related-work positioning is complete. Beyond the conceptual comparison table of latent-reasoning VLAs, the paper implements matched baselines that isolate the effect of where multimodality lives.
  • The proposed method is lightweight and easy to graft onto a pretrained VLA, and the evaluation is broad.
Critical weaknesses
  • The paper’s contribution is closer to an incremental repositioning. Each component, i.e., rectified flow, classifier-guided sampling, Boltzmann reranking, latent skills, soft-token prefix conditioning, LoRA, is well established. The novelty lies mainly in re-situating a known idea at the VLA decision bottleneck and packaging it into a unified interface. That is a legitimate contribution, but it reads as a recombination of existing ideas rather than a fundamentally new method.
  • The evaluation is not fully convincing for the paper's broader claim. Most multimodal settings are constructed by the authors precisely. The headline metrics are command-mode rate and steer range rather than task success rate, and the reported absolute success is low (GPL grasps at roughly 0.33 versus 0.50 for the language baseline), with the plan-conditioning interface itself incurring a non-trivial competence penalty. A policy that follows commands more often but grasps less often may look favorable on steer range without being better in practice.
Questions for authors
  • Given the reported competence penalty of the plan-conditioning interface and the low grasp rate of the plan arm, how do you weigh the added controllability against the loss in raw execution?
  • Could you report the end-to-end task success rate on a setting that is not purpose-built?
2 comments
AuthorsOpenReview ↗

Response to Reviewer yQDm

We thank the reviewer for their constructive evaluation and positive remarks regarding our framing and evaluation.

W1: Core contribution and component isolation. While individual building blocks like rectified flows and LoRA adapters are established, their synthesis at the VLA decision bottleneck is novel. These parts do not automatically yield our observed properties when combined; our experiments identify exactly where they fail to do so:

  • Guidance placement matters. Classifier guidance on π0's generative action chunk (using a matched classifier) suppresses grasping entirely (requested completions dropped from 8 to 0 out of 96). Applying the same guidance to the plan increases completions (14 to 18) and nearly eliminates wrong-cube grasps (14 down to 1).
  • Reranking efficiency. Reranking π0's own action samples (with the same scenes, capacity, and critic) does not reduce execution cost because action noise primarily alters a single committed strategy. Reranking plans does.
  • Read-out behavior. A winner-take-all read-out avoids averaging modes (invalid draws: 0.03) but fails to cover or steer anything (Table 1).
  • Demonstration dependency. In LIBERO, the plan interface on π0.5 shows a weak plan-selection effect (+0.19 grasped episodes) until the demonstrations are changed (+0.65).

Our core contribution is an understanding of where a distribution over decisions must live and when a reasoning plan is genuinely needed. Prior mechanisms (e.g., value-based selection of latent skills, classifier-steered latent diffusion reasoning) do not test these specific questions.

Q1: Steer range vs. practical success. We tested this with an end-to-end metric, commanded success: the requested object is grasped and lifted.

Table A. Commanded success in our two-cube task. Grasping the other cube or failing to grasp counts as failure; 95% bootstrapped CIs over 8 held-out layouts.

ExecutorCommand acts onSuccessGPL − other armGrasps taking requested cube
Shared research executor (240 per arm)Language26%—51% (62 of 121)
GPL plan33%+7 [+1, +15]100% (79 of 79)
π0 (96 per arm)Action chunk0%—no grasp
GPL plan19%+19 [+13, +24]95% (18 of 19)

The 0.50 vs. 0.33 grasp rate cited by the reviewer counts grasps of either cube: the language arm grasps more often, but only 51% of its grasps take the requested cube (chance level), against 100% for the plan arm. Net following (commanded success minus wrong-cube completions) is +33 points for the plan code vs. +1 for language. On π0, the plan arm outperforms the action chunk at every tested guidance strength (ww = 1, 2, 4, 8) and also beats the action arm's best setting (no guidance) by +10 to +14 points.

Weighing the trade-off. The 0.274 grasp-rate cost in LIBERO follows the adapter seed, not the plan: it ranges from 0.00 to 0.61 across six seeds and does not track steering strength (r=0.18r = 0.18). On the real robot (below), the added control costs no detectable raw success and raises it under deployment shifts.

Q2: SimplerEnv tasks and the redundancy principle. On four standard SimplerEnv tasks, the plan interface shows no detectable difference in task success (400 paired episodes).

Table B. Task success (100 paired episodes per task).

TaskStock π0π0 + GPL planDifference [95% CI]
Spoon on towel0.570.56−0.01 [−0.10, +0.08]
Carrot on plate0.500.42−0.08 [−0.19, +0.03]
Eggplant in basket0.900.87−0.03 [−0.10, +0.04]
Stack cube0.640.61−0.03 [−0.12, +0.06]
Pooled0.6530.615−0.04 [−0.09, +0.01]

In these fully observed tasks the plan carries no information the image lacks, so the redundancy principle predicts no gain, and we detect none. When an occlusion removes the decision from the observation, the plan helps: in an occluded "spoon on towel" task (spoon hidden from step 5), a carried plan from GPL's flow head raises task success from 0.12 (stock π0) to 0.38 (+0.26 [+0.12, +0.40]; n=50n = 50 per arm, two-sided Fisher p=0.005p = 0.005).

Real robot: task success, not only steer range. On a real SO-101 arm (general response; video in the supplementary material), the plan-free π0.5 completes the task in 17/17 runs but chooses the strategy itself, so a request is met only by chance (50%). With GPL, the robot executes the requested strategy and completes the task in 43/52 runs (83%; exact binomial p<10−5p < 10^{-5}), with raw success of 45/52 (not detectably lower; one-sided Fisher p=0.12p = 0.12). When a deployment shift disables one strategy, reweighting GPL's plans from each run's outcome, with no gradient step, raises task success from 45% to 81% (raised target) and from 45% to 64% (roll on its side). Following commands translates into higher success in practice.

Thanks for the response. My concern about the incremental contribution has been resolved. However, in Table B, GPL leads to a slight performance drop. While I accept the explanation of "no gain" based on the "redundancy principle", the slight performance drop may need more clarification.

Overall, the rebuttal has significantly strengthened my view of the paper. I am raising my score accordingly.

Reviewer_8TNSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary. The paper argues that reasoning-augmented VLAs collapse their intermediate decision into a single deterministic state. When several plans are valid, that state either averages them into an invalid plan or commits to one and hides the rest. The authors propose the Generative Plan Latent (GPL). A small rectified-flow head, conditioned on frozen π0 features, models a distribution p(z|c) over low-dimensional task-level plans (e.g., which object, grasp orientation, route). Sampled plans are injected into the action expert as 8 soft tokens via a projector and LoRA, trained only with π0’s native flow-matching loss and plan dropout.

Critical strengths

Strength 1: Controlled isolation of the plan vs. action locus. The read-out ablation (same WTA weights, MAP vs. component-sampled) and the shared-executor comparisons separate addressability from executor competence and model capacity. These controls are what make the central claim credible, and they are the main way the paper adds something beyond prior latent-plan work.

Strength 2: A falsifiable redundancy principle with causal evidence. The LIBERO counterfactual intervention changes only the demonstrations, and steer range rises from +0.19 to +0.65 with a monotone dose response. This turns the method into a predictive claim about when plans help, and that claim also explains the paper's own negative results.

Critical weaknesses

Weakness 1: Low-competence, constructed settings limit external validity. Most evidence comes from toy multimodal tasks the authors built, such as two identical cubes, run on weak executors: grasp rate is about 0.33, stock π0 succeeds on 5/48 two-cube grasps, and the plan interface itself costs 0.274 in grasp rate. With base competence this low, it is unclear whether addressability would matter, or survive, in realistic tasks where VLAs actually succeed. The paper's practical significance hinges on that question.

Weakness 2: The headline comparisons are tilted toward GPL. In the action-level steering baseline, the target is decodable from the action chunk at near-chance accuracy (balanced accuracy of about 0.57), so that arm's guidance classifier is weak from the start. Outcome adaptation is reported as an expected value over the full candidate pool, which requires knowing the outcome of every candidate rather than deploying a single draw. Both choices inflate the key "plan beats action" and "reranking matches retraining" claims, which are exactly the claims that would distinguish this work from prior latent-plan methods.

Questions for authors

See the weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8TNS

We thank the reviewer for their careful reading and insightful comments.

W1. Addressability where the VLA already succeeds. Trained on 64 pick and 60 push demonstrations under one instruction that does not choose the strategy, the base π0.5 is competent: it succeeds in 17/17 runs. But it chooses the strategy itself, so a request is met only by chance (50%), and it exposes no choice that a user or an outcome could change at test time. Both happen after deployment: a user wants a specific strategy, or a scene change disables the preferred one. With GPL, the same policy gains three properties:

  • It keeps both strategies. Either can be requested: the robot executes the requested pick and completes the task in 24/27 runs, and the requested push in 19/25.
  • It follows the command. Over the 52 requested runs, it executes the requested strategy and completes the task in 43/52 (83% vs. 50%; exact binomial p<10−5p < 10^{-5}), with raw success of 45/52 vs. 17/17 (difference not detectable; one-sided Fisher p=0.12p = 0.12).
  • It adapts to shifts at test time. An unseen 3 cm platform under the target makes every push fail; with the roll on its side, pinch grasps fail (0/16) while most pushes still work (31/41). In both cases a strategy the policy already knows still works, so the fix is a different choice, not a new skill. Reweighting GPL's plans from each run's outcome, with no gradient step, raises success from 45% to 81% on the raised target (22/27, vs. 5/11 for the same plan head without updates, which keeps its training-time mix of strategies) and from 45% to 64% with the roll on its side.

The interface also survives where no decision is open: on four standard SimplerEnv tasks, where stock π0 succeeds in 50–90% of episodes, task success changes by −0.04 [−0.09, +0.01] (400 paired episodes). When an occlusion hides the target from step 5 of the spoon task, the carried plan raises success from 0.12 to 0.38 (n=50n = 50 per arm; two-sided Fisher p=0.005p = 0.005).

The numbers cited come from three settings. 0.33 is the plan arm's grasp rate on a frozen ECoT-OpenVLA-7B research executor in our two-cube diagnostic, and every one of those grasps took the requested cube (vs. 51% for language). 5/48 is stock π0 in the same diagnostic. 0.274 is π0.5's grasp-rate cost on LIBERO, a mean over six adapter seeds: it ranges from 0.00 to 0.61, the same seed gives nearly the same cost on both demonstration sets (r=0.997r = 0.997), and it does not track steering strength (r=0.18r = 0.18), so it comes from adapter optimization rather than the plan interface.

W2a. The two classifiers are symmetric. Neither classifier receives cc; both are MLPs of matched capacity (131,673 vs. 131,841 parameters), trained and normalized the same way. The action classifier's low held-out accuracy is what the comparison measures: the requested cube is an explicit coordinate of the plan but must be inferred from the action chunk. On the approach metric, the unguided action arm starts higher (0.573 vs. 0.479), so the plan's approach advantage under guidance does not come from a better starting point. On commanded success, guiding the plan completes the requested grasp more often than guiding π0's action chunk at every guided strength we tested (ww = 1, 2, 4, 8): +0.15 to +0.21 (ahead in 6 to 8 of 8 held-out layouts; matched adapters, same demonstrations; unguided, +0.06, not detectable).

W2b. What the learner sees, what is executed, and what is reported are separate.

  • Update. The critic or reward table is fit only on binary outcomes of executed plans; the true reliabilities (0.9, 0.25) stay hidden. In Fig. 4B, each round executes K = 16 plans per training scene, drawn from the frozen prior (reranking arms) or the current retrained flow (retraining baselines).
  • Selection. Deployment draws one plan from the reweighted pool (Eq. 7).
  • Reporting. Fig. 4B scores that draw on a fresh, never-executed pool from the frozen prior: ∑kρkr(k)\sum_k \rho_k r^{(k)}, with r(k)r^{(k)} the simulator's true success probability of candidate kk, is the exact expected success of that single draw. Only the evaluator uses it; the retraining baselines are scored the same way.

On the real SO-101 (W1), each run executes the draw itself: one plan from a pool of 8, the per-strategy count updated from that run alone, and the reported rates are executed runs (runs 2–8; one raised-target run excluded: the operator touched the roll). Single-plan selection also matches the Boltzmann value in simulation:

  • Two-spoon π0 (closed loop): the critic's top-ranked plan reaches the allowed spoon in 18/20 held-out scenes, vs. 0.38 for a random draw.
  • Contact sketch (scripted executor; fit and scored on disjoint layouts): executing only the critic's top-scoring plan gives the same gains as its Boltzmann value (+0.44 to +0.51).

Thank you for the detailed and thoughtful rebuttal. The clarifications and additional results address my main concerns, and I appreciate the effort the authors put into the response. I maintain my positive assessment of the paper and keep my score.

Reviewer_K6zYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Generative Plan Latent (GPL), a framework that models intermediate task-level decisions in Vision-Language-Action (VLA) models as conditional distributions rather than deterministic representations. Its main contribution is to make alternative plans explicitly addressable at inference time, enabling multimodal decision coverage, user-directed steering, and outcome-based adaptation without retraining the underlying VLA. The authors further propose a redundancy principle to characterize when explicit plans provide useful decision information beyond the observation, and validate the framework through extensive experiments across manipulation and navigation tasks.

Critical strengths
  1. An interesting insight: multimodality should also exist at the decision level, not only at the action level. the authors also did experiments to explain why "plan" distribution should be isolated from action distribution

  2. GPL provides a principled way to preserve and access alternative decisions through coverage, steering, and outcome adaptation. Notably, these capabilities can be achieved without retraining the underlying plan generator during inference-time steering or outcome adaptation, making the framework potentially useful for flexible deployment and adaptation.

  3. The paper goes beyond demonstrating overall task performance and systematically investigates when and why explicit plans are useful. In particular, the comparisons between plan-level and action-level guidance, the controlled experiments on the redundancy principle, and the extensions to learned plan representations and different environments provide valuable evidence supporting the central claims.

Critical weaknesses
  1. The definition and construction of plans remain underexplored. Although GPL provides a compelling framework for modeling distributions over plans, the paper does not sufficiently clarify what constitutes a meaningful task-level plan and how such representations should be constructed in general.

  2. The conditioning information used by the action-level guidance classifier is unclear. While the plan-level classifier is explicitly conditioned on the VLA representation c=fϕ(o,l)c=f_\phi(o,l), Appendix E.2 only describes the action-level classifier as operating on the noisy action chunk xtx_t, without clarifying whether it receives the same visual-language conditioning. This detail is important for ensuring a fair comparison between the two guidance approaches.

  3. Validation remains mostly simulation-based, and most multimodal decision scenarios are author-constructed rather than drawn from real robotic tasks. The authors also report a non-negligible grasp-rate penalty from plan conditioning, even after reducing plan–observation redundancy, Whether the framework scales to richer decision spaces, and whether its gains justify the added conditioning and execution cost remains unclear.

Questions for authors
  1. how can meaningful and transferable plan representations be discovered from demonstrations without extensive task-specific engineering?

2 .Can authors provide more comparsion details in W2?

  1. Could the authors evaluate GPL on more complex or long-horizon manipulation tasks to further demonstrate the effectiveness and practical benefits of plan-level representations?
2 comments
AuthorsOpenReview ↗

Response to Reviewer K6zY

We thank the reviewer for their valuable feedback and thoughtful comments.

W1 / Q1: Defining and finding a plan. We define a plan as the high-level decision left open before motion begins: target object, strategy (e.g., pick vs. push), contact point, or route. Words describe such geometric decisions poorly, so GPL represents the decision as a low-dimensional continuous variable that the executor reads directly, and keeps a distribution over it. For example, in our two-cube task, passing the request through a frozen reasoner as a plan code rather than as language eliminates grasps of the unrequested cube (0 of 79 grasps, vs. 59 of 121 with language).

In our experiments, plans are beneficial when they:

  • separate valid alternatives in the demonstrations;
  • are not already fixed by the executor's conditioning cc (redundancy score R<1R < 1, computed from cached features before any adapter training);
  • can be read directly by the executor.

Plans also need not be engineered per task. An unsupervised 2-D latent, learned via an autoencoder on future end-effector motion, achieves a 0.78 approach rate under guidance (vs. stock π0's 0.50), performing comparably to a hand-designed anchor (paired +0.03). Furthermore, a single object-relative contact-sketch schema serves four task families and acts as the plan on our real robot. We have added this definition to the method section.

W2 / Q2: Classifier conditioning. In the π0 comparison, neither classifier receives the scene conditioning cc: plan guidance uses q(y∣zt,t)q(y \mid z_t, t) and action guidance uses q(y∣xt,t)q(y \mid x_t, t). Both are MLPs of matched capacity (131,673 vs. 131,841 parameters), trained the same way, with the same guidance strength normalized by the Euler step. Scene information enters only through the generators: the plan flow vθ(zt,t,c)v_\theta(z_t, t, c) and π0's action expert (images, instruction and state). Equation 2 shows the scene-conditioned form, q(y∣zt,t,c)q(y \mid z_t, t, c), used by the guided plan arm in LIBERO. We have added these details to Appendix E.2.

W3: Real robot, constructed scenarios, and cost.

  • Real robot. On a real SO-101 arm (pick or push), the robot executes the requested strategy and completes the task in 43/52 runs (83%, vs. 50% by chance without the plan), with raw success of 45/52 (vs. 17/17; difference not detectable, one-sided Fisher p=0.12p = 0.12).
  • Adaptability. The difficulty comes from physics, not from rules we wrote: the soft, light roll slides when upright but rolls on its side. Under two unseen shifts (a raised target; the roll on its side), reweighting plans from each run's outcome, with no gradient step, raises success from 45% to 73% (general response).
  • Constructed scenarios. Standard VLA benchmarks reduce multimodal alternatives to a single ground-truth trajectory: for each LIBERO-10 task, all 50 demonstrations follow the same execution order, leaving no decision for a plan to resolve, so they cannot measure decision coverage. On four standard SimplerEnv tasks, where the scene fixes the decision, GPL keeps baseline performance (−0.04 [−0.09, +0.01] in task success, 400 paired episodes), as the redundancy principle predicts.
  • Grasp-rate cost. The 0.274 cost (LIBERO, π0.5) is a mean over six adapter training seeds, and the seed sets it: the cost ranges from 0.00 to 0.61 across seeds, the same seed gives nearly the same cost on both demonstration sets (r=0.997r = 0.997), and it does not track steering strength (r=0.18r = 0.18). The cost therefore comes from adapter optimization rather than from the plan interface; at least one seed shows no cost at all.
  • Latency. The plan interface adds little overhead. On the robot server, π0.5 with the guided plan takes 129 ms per action chunk (71 ms compiled), vs. 107 ms (59 ms compiled) without the plan (same GPU). With a frozen ECoT-7B backbone, an action through the plan takes 56 ms, whereas decoding a text reasoning chain takes 4.5 s. The interface is trained once; adapting to outcomes only updates a success count per strategy.
  • Richer decision spaces. The same interface already handles plan spaces from a 2-D learned latent to a 54-D contact sketch, which suggests it extends to richer ones without architectural changes.

Q3: Long-horizon tasks. Integrating GPL into a larger hierarchical framework to address complex, long-horizon, multi-step tasks is indeed an exciting and valuable direction for future research. In this work, our primary focus is on formalizing and addressing decision-level multimodality at the subtask/System 1 execution level. Nevertheless, our paper provides evidence of plan-level reasoning over extended horizons, including order optimization in multi-object manipulation (Fig. 4A), maintaining a plan through sustained visual occlusions from step 5 to the end of a 120-step episode, and navigating complex 16-move routes in the maze domain.

thanks for your work, I would like to maintain my socre

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their constructive feedback and appreciate that all three found the question of where multimodality should reside in a reasoning VLA to be valuable. Given the broad scope of settings covered in this work, we briefly restate our main contributions.

A different way to reason for physical tasks. By a plan we mean the reasoning strategy a task leaves open before any motion: which object, pick or push, where to make contact. Existing reasoning VLAs commit this decision to one output per observation, often text, which has two limits. First, once the decision is committed, the alternatives are gone: a user cannot request another, and an outcome cannot shift the strategy. Second, in physical tasks the decision is often geometric, which text describes only indirectly. GPL represents the decision as a small continuous plan that the executor reads directly, and keeps a distribution over it, so the alternatives can be sampled, requested, and reweighted from outcomes.

Contributions.

  • Method: GPL, a distribution over task-level plans, added to a pretrained VLA.
  • Experiments: we show why reasoning as a distribution is appealing for VLAs.
    • It learns all available plans. GPL covers every valid plan in all held-out layouts, where a regression read-out averages the plans into an invalid one and a winner-take-all read-out keeps only one.
    • It can be guided at inference time, and the plan provides a stronger leverage point than language instructions or direct action control. On π0, guiding the plan completes the requested grasp more often than guiding π0's action chunk, by +0.15 to +0.21 at every guidance strength.
    • It adapts from outcomes. When a deployment shift makes one plan fail, reweighting plans by their outcomes, without updating the policy or the plan head, raises success by +0.44 to +0.51 in four simulated task families (learned critic) and by +0.28 on the real robot (per-strategy success counts; 0.45 to 0.73). Thus, if the plan that was best during training stops working at deployment, GPL lets the policy adapt its strategy without retraining.
  • The redundancy principle: a plan helps only when the observation does not already fix the decision, and we test this causally: in LIBERO, changing only the demonstrations raises the plan-selection effect from +0.19 to +0.65 (grasped episodes).

Real-world robot evaluation. To address reviewer concerns regarding real-world applicability, we evaluated GPL on a physical SO-101 arm tasked with relocating a toilet-paper roll onto a target square via picking or pushing. The training set has 124 demonstrations (64 pick, 60 push) under a single instruction that does not say which strategy to use ("pick and push toilet paper"). π0.5 is fine-tuned in full, jointly with a projector that maps the plan to 8 soft tokens; GPL's plan head, a small rectified-flow network trained on the demonstrations' contact sketches, generates a plan from each run's observation. No weights change during deployment.

  • Commanding the strategy. Without the plan, the base policy succeeds in 17/17 runs but chooses the strategy itself, so it meets a requested strategy only at chance (50%: with 27 pick and 25 push requests, a policy that ignores the request matches it in 48–52% of runs, whatever strategy it prefers). With GPL's plans, the robot executes the requested strategy and completes the task in 43/52 runs (83%), with raw success of 45/52.
  • Adapting to environmental shifts. We demonstrate test-time strategy adaptation under physical distribution shifts without parameter updates. Introducing an unseen 3 cm platform beneath the target causes pushing to fail completely; however, dynamically reweighting candidate plans based on outcome history recovers performance to 81% success, compared to 45% for the same plan head without outcome updates (+36 points). In a second scenario where the roll is placed on its side (a pose absent from training), pinch-grasps fail completely (0/16 success), while pushing causes the roll to roll rather than slide (31/41 executed pushes succeed). Here, adaptive reweighting achieves 64% success (runs 2–8) versus 45% without outcome adaptation (+19 points). Combined across both shifts, outcome adaptation improves overall success from 45% to 73% (+28 points).

The toilet-paper roll serves as a challenging benchmark object due to its orientation-dependent dynamics. Shifting from an upright training pose to a horizontal test orientation significantly alters pushing and rolling mechanics. Furthermore, the soft compliance of the object introduces complex post-roll motion profiles that can propel the roll beyond the reach of the manipulator, accounting for the lower absolute success rate in the horizontal configuration.

The revised PDF highlights all changes in blue and adds the robot experiments to the main body (Section 6.3); a video of the robot runs is in the supplementary material.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.