acceptodds
Under review as a conference paper at ICLR 2027

When Best Prediction Fails: Adjacent Set Action Reconstruction for Latent World Model Planning

Abstract

Controllers based on sampling and latent world models rank candidate action sequences by predicted terminal cost, execute the first action block, and replan. Residual prediction error can give an infeasible sequence an anomalously low cost, even when the terminal objective reflects physical task progress. A larger proposal pool creates more opportunities for such errors to outrank feasible alternatives. We call this conditional failure proposal overgeneration. In Cube candidate execution audits, increasing the proposal budget from 72 to 288 reduces minimum latent cost selection feasibility from .375 to .062 for position targets and from .344 to .031 for position and yaw targets, although every larger pool contains a feasible sequence. We introduce Adjacent Set Action Reconstruction (ASAR). Among low-cost proposals, ASAR identifies an adjacent set using standardized early action prefixes, reconstructs a locally weighted action sequence with a light minimum-cost anchor, and executes its first block. On 75 Carry and Release queries, Kernel ASAR improves event completion success over minimum cost selection by 28.0, 24.0, and 18.7 percentage points under latent cost and by 18.7, 20.0, and 17.3 points under a trajectory reachability cost at 72, 144, and 288 proposals. Finite-pool analysis characterizes lower-tail selection risk, radius support under an explicit probability gap, and containment of the executed action block under a local physical condition.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, action selection, sampling-based planning, robot control, proposal overgeneration, action reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_UM3dOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper deals with sample-based planning in continuous control with latent world models. The paper first demonstrates the planners are subject to the phenomenon of proposal overgeneration, where generating candidate actions for the planner results in an increased likelihood of selecting infeasible action sequences. The paper then introduces ASAR, a method for constructing candidate actions outside the candidate set. ASAR generates actions by mixing two groups of actions: i) the action with the lowest cost, and ii) a weighted mix of local neighborhoods of actions. Then ASAR is evaluated in several continuous control domains.

Critical strengths
  1. I found the idea of measuring proposal overgeneration interesting. I think taking this into consideration can lead to new work regarding candidate generation for sample based planners.
Critical weaknesses

The critical weakness is the readability of the paper. I took several passes through the paper and I only got a bare understanding of what is going on in the paper; in fact, I may not have understood it correctly because it was so difficult to read. This has definitely hindered my ability to properly assess the soundness of ASAR and evaluate the claims made in the paper.

The two parts of the paper --- proposal overestimation and action construction --- appear to be disjoint. I do not understand how one part relates to the other or how understanding proposal overestimation influences the functioning of ASAR.

I also question the comparison of ASAR against the baseline CEM. I think the idea behind the experiments were to show that augmenting the action set of CEM using ASAR improves its performance, if I understood the setup correctly. However, it just seems like ASAR gets some extra computation because it uses these computations to generate actions. Would CEM perform better if given the same amount of total computation? Again, I may have misunderstood the setup since the explanation in the paper is lacking.

Table 1 is hard to understand based on the description in the corresponding section. The description reads as if Top-20 exposure is the the fraction of pools with feasible actions in the set of low cost actions. First off the budget reduces this value. Second, the paper states that reachability improves this fraction. But that's only at higher budgets --- at lower budgets, it reduces this value. I am also not sure how these pools are made. Equation 6 is an incorrect application of the probability chain rule.

Table 3 also does not make sense --- I do not understand the confidence intervals as written.

The equations and theory are not properly defined. I cannot assess their correctness.

Questions for authors

No questions to ask.

4 comments
AuthorsOpenReview ↗

Response to Concerns on the ASAR Mechanism, Computational Fairness, and Technical Clarifications

We thank the reviewer for their time and comments. We would like to address the concerns and clarify several important points regarding our method and experimental results.

  1. Connection Between Overgeneration and ASAR We would like to clarify that overgeneration and ASAR are not two independent components. Rather, overgeneration serves as a diagnosis of the underlying failure mechanism, while ASAR is designed to address this problem.

Overgeneration illustrates that sampling more candidate actions during planning can expand the range of possible actions, but also increases the chance of selecting actions associated with extreme model prediction errors. In our work, overgeneration therefore serves as a mechanism-level diagnosis, suggesting that execution failures may arise during the mapping from the candidate action set to the action ultimately executed.

ASAR takes this observation one step further. While retaining the original world model and CEM cost function, we introduce local support and action reconstruction into this mapping to address the overgeneration problem.

Our core intuition can be illustrated with two candidate actions, A and B. Suppose A has many similar neighboring actions that the model also predicts can accomplish the task, whereas even a small modification to B causes the model to predict failure. Given the continuity of physical dynamics, we hypothesize that A is more likely to generalize to nearby actions and to be genuinely feasible in the physical environment.

To examine this hypothesis, we conduct a controlled evaluation matching states, predicted costs, and sampling origins. We observe a 10.03% improvement in empirical physical feasibility associated with local support.

This provides empirical evidence for our central intuition: local support contains information about physical feasibility that is not fully captured by an individual candidate's predicted cost.

  1. Computational Fairness Compared with CEM

To directly address the concern regarding computational fairness, we have conducted an additional experiment under identical hardware configurations and fixed computation budgets.

Specifically, we run vanilla CEM and ASAR on the same NVIDIA RTX 6000 Ada GPU and dual-socket AMD EPYC 7763 system, with CPU execution restricted to two threads. Both methods are allocated the same 0.5-second computation budget. We evaluate both methods on the Hard Cube task over 128 randomly initialized trials under this matched computational setting.

Vanilla CEM: 22/128 successful trials (17.19%).

ASAR: 32/128 successful trials (25.00%).

Under the same computational budget, ASAR corresponds to a 7.81% improvement in success rate. And this result demonstrates that ASAR outperforms vanilla CEM under the same computation-time budget in this controlled experiment.

  1. Definition of Top-20 Exposure

We would like to clarify the definition of Top-20 exposure in Table 1. Top-20 exposure measures the proportion of action selections in which at least one feasible action appears among the 20 candidates with the lowest predicted costs.

Therefore, the existence of feasible actions in the full candidate pool is not inconsistent with a decrease in Top-20 exposure. This is because newly sampled infeasible actions may receive lower predicted costs and consequently enter the top 20, displacing feasible actions from the lowest-cost candidates.

In summary, the existence of feasible actions measures sampling coverage alone, whereas Top-20 exposure jointly reflects sampling coverage and the quality of candidate ranking by predicted cost.

  1. Interpretation of the Confidence Intervals in Table 3

We would like to clarify that the confidence intervals in Table 3 represent the 95% confidence intervals of the success-rate differences between ASAR and the corresponding baseline in each row, rather than the confidence intervals of ASAR's own success rate.

For example, +0.039 [0.016, 0.065] indicates that ASAR improves the success rate by 3.9 percentage points relative to the baseline, with a 95% confidence interval for this difference ranging from 1.6 to 6.5 percentage points.

To avoid ambiguity, we will explicitly clarify the comparison direction, statistical unit, and seed-grouped bootstrap procedure in the table header and caption.

  1. Application of the Probability Chain Rule in Equation (6)

Equation (6) is a direct application of the probability chain rule and does not require an independence assumption.

Let:

P denote the event that at least one feasible candidate exists in the entire candidate pool. T denote the event that at least one feasible candidate exists among the top-k candidates ranked by predicted cost.

Whenever $ T $ occurs, $ P $ must also occur. Therefore, $ T $ is a subset of $ P $. For $ \Pr(P) > 0 $, we have:

Pr⁡(T)=Pr⁡(T∩P)=Pr⁡(P)Pr⁡(T∣P).\Pr(T) = \Pr(T \cap P) = \Pr(P)\Pr(T \mid P).

This equality follows directly from the definition of conditional probability and does not assume independence between P and T.

The derivation is already provided in Appendix D.1. We will also add the explicit event definitions and intermediate steps to the main text to make the argument easier to verify.

We hope these clarifications address the reviewer's concerns regarding the motivation, computational fairness, and technical correctness of our approach.

I would like to thank the authors for taking their time answer my questions.

Re. 1: I understand that in the paper, this was the intended message. However, given the paper as it is, this is not apparent in the paper. It reads as if there is no flow between the two.

Re. 3, 4: I am glad that these are clarified.

Re. 2: I am happy that the authors ran these experiments as I believe they are a critical evaluation against the baseline.

Re. 5: This explanation makes sense now. I think that a line explicitly stating that T⊂PT \subset P needs to be included before equation 6.

I again thank the authors for their rebuttal. However, I still maintain my score. While the authors mention they will add proper definitons and formalizations, I was not able to assess the paper on its correctness given what I read.

AuthorsOpenReview ↗

Revised Manuscript and Corrections to the Candidate-Budget Audit

We thank the reviewer for the follow-up and for acknowledging our clarifications and additional experiments.

We understand that your remaining concern is whether the technical correctness of our method can be properly assessed from the manuscript itself, rather than from our rebuttal explanations alone.

To address this concern, we have now uploaded a revised PDF with improvements to the writing and presentation, particularly focusing on making the connection between overgeneration and ASAR clearer and improving the presentation of the technical arguments and experimental results.

During our re-examination of the experiments, we also found that some of the "feasibility" labels in the previous candidate-budget audit were generated based on relative rankings among candidates and therefore could not be directly interpreted as indicating whether a candidate satisfied a fixed physical success criterion.

Specifically, this issue affected the following parts of the manuscript:

Page 4, §4: The 32-candidate-pool budget audit and the corresponding Top-1 description.

Page 5, Table 1 and Figure 2: The corresponding Position / Position+yaw results.

Appendix G.2, Table 9: The corresponding two rows.

Appendix G.3, Table 11: The results reported in this table.

We have corrected this issue, rerun the affected experiments, and updated the corresponding results and descriptions in the revised manuscript.

We appreciate your concern that the correctness of the work should be assessable directly from the manuscript. Since the revised PDF is now available, we would be grateful if you could take another look and reconsider your assessment in light of these changes.

Reviewer_3fX2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper finds that in latent world model planning, increasing the number of candidate action sequences can reduce planning reliability because residual prediction errors cause infeasible trajectories to receive overly optimistic costs. Based on this, the authors propose Adjacent Set Action Reconstruction (ASAR), which identifies a locally dense group of low-cost action sequences based on their early action prefixes and reconstructs a new action sequence through weighted averaging, rather than selecting the single lowest-cost candidate. The experiments are build upon the existing world model and CEM planner and simulated Cube manipulation and shows improvements in task success.

Critical strengths

I like the simplicity of the proposed method. ASAR leverages local structure among low-cost action sequences without requiring additional world model training or changes to the existing CEM planner.

The paper achieves substantial gains on Cube Carry and Release tasks, and also provides theoretical insights into lower-tail selection risk and conditions under which locally supported action reconstruction can remain physically admissible.

Critical weaknesses

My major concern is that ASAR only achieves improvements on Cube Carry and Release, but gains are much smaller on general Cube tasks and negligible or negative on Reacher and Push-T. This questions about its broader applicability.

Also, the proposed method performs worse on tasks requiring multiple stages, so I believe local prefix similarity may fail to capture the sequential structure necessary for complex planning.

The method is under the assumption that locally dense and low-cost action sequences are more reliable than isolated candidates. However, density may simply reflect the CEM sampling distribution rather than physical feasibility. Dense clusters can also contain systematically incorrect predictions.

Questions for authors

See weaknesses

3 comments
AuthorsOpenReview ↗

Response to Concerns on Feasibility, Long-Horizon Performance, and Generalizability

We appreciate the reviewer's constructive feedback, particularly regarding the density mechanism, long-horizon performance, and generalizability across environments. To address these concerns, we have designed three targeted, controlled evaluations.

  1. Density and Physical Feasibility. We would first like to clarify an important distinction: we do not claim that the overall sampling density of CEM directly implies physical feasibility. Rather, our hypothesis is that local support among goal-relevant, low-cost action prefixes provides information about physical feasibility beyond what is captured by the predicted cost of an individual candidate. To test this hypothesis, we evaluate independently sampled states with fixed candidate pools, controlling for predicted cost and sampling origin. Under these matched conditions, the neighborhoods selected by our method exhibit a 10.03% improvement in empirical physical feasibility over matched controls.
  2. Additional Cube Settings and Long-Horizon Tasks. We introduce a multi-stage evaluation in which vanilla CEM and ASAR are provided with identical predefined semantic subgoals, initial conditions, and computational budgets. In this put in drawer task, robotic arm need to unlock drawer, pull open, approach/lower/grasp block, lift and move inward, lower/release, move away and close drawer. Under this controlled setting, full task success improves from 0.0% to 31.67% in 300 trials.
  3. Generalizability Across Environments. We thank the reviewer for highlighting the importance of evaluating generalizability across different tasks and environments. To provide a more comprehensive evaluation of cross task generalizability, we are extending our experiments to additional tasks and environments. These evaluations are now nearly complete. To facilitate a timely and constructive discussion, we would like to first address the concerns for which we already have sufficient evidence, while finalizing the additional cross task evaluations. Currently, we are verifying the results and consolidating the quantitative comparisons to ensure accurate and rigorous reporting. We will provide the complete results and corresponding analysis in a follow up response within the next two days.
AuthorsOpenReview ↗

Follow-up: Additional Multi-Stage Task Evaluations

We thank the reviewer for their patience. As promised in our previous response, we would like to provide additional experimental results addressing the generalizability of ASAR beyond the original Cube tasks.

We conducted two additional multi-stage manipulation evaluations in the Scene environment. Both tasks involve multiple operations with sequential dependencies, but emphasize different manipulation requirements. For each task, the corresponding Scene world model was trained for 50,000 steps. We used finer-grained expert-image subgoals shared by CEM and ASAR, decomposing the operations into stages such as approaching, aligning, pressing buttons, and operating handles.

Task 1: Closing and Locking the Drawer and Window

  • Initial state: The drawer and window are both open and locked, with a cube on the table.
  • Goal: Close and lock both the drawer and window while keeping the cube near its original position.
  • Procedure: Unlock the drawer by pressing its button, push it closed, and press the button again to lock it. Then unlock the window, slide it closed, and lock it again.

This task primarily evaluates button pressing, handle manipulation, lock-state transitions, and sequential dependencies.

Over 300 trials, CEM achieves 15.33% success, while ASAR achieves 27.67%**, corresponding to a **+12.34 percentage-point improvement.

Task 2: Placing a Cube in the Drawer and Opening the Window

  • Initial state: The drawer and window are both closed and locked, with a cube on the table.
  • Goal: Place the cube inside the drawer, close and lock the drawer, and open and lock the window.
  • Procedure: Unlock and open the drawer, approach and grasp the cube, lift it, place it inside the drawer, and release it. The robot then withdraws, closes and locks the drawer, unlocks and slides open the window, and locks it again.

This task additionally involves grasping, object transportation, placement, and coordination between multiple manipulation stages.

Over 300 trials, CEM achieves 0% success, while ASAR achieves 12.00%**, corresponding to a **+12.00 percentage-point improvement.

Summary of Additional Results

TaskCEMASARImprovement
Closing and Locking Drawer & Window15.33%27.67%+12.34
Placing Cube in Drawer & Opening Window0%12.00%+12.00

These results provide additional evidence that ASAR's improvements are not restricted to the original Cube Carry and Release setting. In particular, the two tasks cover different manipulation requirements, including lock-state transitions, articulated-object operations, and object transportation. We think the consistent improvements over CEM in both tasks support the applicability of local action reconstruction to more complex multi-stage manipulation.

AuthorsOpenReview ↗

Revised Manuscript and Experimental Corrections

We have uploaded a revised PDF with improvements to the writing and presentation.

During our re-examination of the experiments, we found that some of the "feasibility" labels in the previous candidate-budget audit were generated based on relative rankings among candidates and therefore could not be directly interpreted as indicating whether a candidate satisfied a fixed physical success criterion.

Specifically, this issue affected the following parts of the manuscript:

Page 4, §4: The 32-candidate-pool budget audit and the corresponding Top-1 description.

Page 5, Table 1 and Figure 2: The corresponding Position / Position+yaw results.

Appendix G.2, Table 9: The corresponding two rows.

Appendix G.3, Table 11: The results reported in this table.

We have corrected this issue, rerun the affected experiments, and updated the corresponding results and descriptions in the revised manuscript.

We hope that the revised manuscript and the additional clarifications address your concerns, particularly regarding the writing and presentation. We would greatly appreciate it if you could reconsider your assessment in light of these revisions.

Reviewer_RGtwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a failure mode in latent world model planning: a larger candidate pool can contain feasible action sequences while making minimum-cost selection less reliable because of prediction errors. The proposed method finds a local group of similar early action prefixes among low-cost candidates and constructs a weighted action sequence from this group. It then executes the first action block and replans. Experiments mainly use simulated Cube manipulation, including candidate execution audits, comparisons under latent and reachability costs, and replication with independently trained world model weights. The paper also provides conditional analyses of local support and action containment.

Critical strengths
  1. The candidate execution audits provide a useful diagnosis of planning failures. Independently executing candidates separates the presence of feasible solutions from the planner’s ability to select them. The observed decline in selection feasibility despite feasible candidates being present makes the problem concrete and gives useful evidence beyond closed-loop success rates alone.

  2. The Cube results include several meaningful controls. The paper compares reconstruction with minimum-cost selection, least-isolated existing-candidate selection, and ordinary elite averaging. It also tests two terminal costs and independently trained model weights. These comparisons provide credible evidence that reconstruction can improve control in the tested setting without retraining the world model. I also appreciate the explicit reporting of negative results and limitations.

Critical weaknesses
  1. The explanation of why local prefix similarity helps could be strengthened. Table 27 provides mixed evidence for the relationship between prefix density and feasibility. The more favorable comparison selects the least-isolated candidate among candidates already known to be feasible, so it may not directly establish the reliability of ASAR’s chosen neighborhood. The theoretical results also rely on conditions whose presence in practice remains somewhat unclear. The empirical gains appear to be better supported than the proposed explanation for them.
  2. The demonstrated gains have a relatively limited scope. The largest improvements are on Carry and Release queries that begin with an already grasped cube. Full pick-and-place does not show the same benefit. The transfer experiments evaluate Portfolio ASAR rather than the main Kernel variant and show no clear improvement. The paper acknowledges these limits, but the evidence for benefits beyond the tested local manipulation setting seems limited.
Questions for authors
  1. Could the authors clarify how we might interpret the mixed results of the prefix-density audit? Which findings best support using local prefix similarity to guide reconstruction?
  2. Could the authors clarify what the weaker results on full pick-and-place and transfer tasks imply for the scope of Kernel ASAR? In particular, how should we interpret the transfer results, which were obtained with Portfolio ASAR?
2 comments
AuthorsOpenReview ↗

Response to Questions on Density Analysis, Full-Task Performance, and Portfolio Transfer

We thank the reviewer for recognizing the significance of our diagnosis of planning failures and our comparisons on Cube. We also appreciate the opportunity to clarify the reviewer's questions.

Q1. How Should We Interpret the Mixed Results of the Density Audit?

Table 27 actually presents several different comparisons. Although all of them compare the isolation of action prefixes between a candidate that fails in actual execution and one that succeeds, the successful candidates are selected using different criteria.

When we select the successful candidate with the lowest predicted cost, it is less isolated than the failed candidate in 17 out of 31 comparisons (54.8%). However, when we retrospectively select the least isolated candidate among the successful candidates, the selected successful action is less isolated than the failed action in 24 out of 31 comparisons.

These experiments are intended to demonstrate that feasible action candidates can form locally supported groups, while traditional action selection based on predicted cost alone cannot exploit such support structures.

The most direct evidence supporting the use of local prefix similarity to guide reconstruction comes from our comparison of the actual reconstruction methods. In the Cube evaluation (Table 21), we compare Kernel ASAR against several action selection and reconstruction alternatives, including the existing candidate with the lowest predicted cost, the least isolated existing candidate, and simple elite averaging. Kernel ASAR achieves a success rate 20% higher than these alternatives. This strongly supports the rationale for using local prefix similarity to guide action reconstruction.

Q2. How Do Full-Task Performance and Portfolio Transfer Results Qualify Our Conclusions?

We position Kernel ASAR as a local action reconstruction method applied at each replanning step, rather than a method for solving long-horizon task decomposition.

Its benefits on Cube are not limited to a particular selectively chosen subset. In the overall Cube evaluation, the success rate increases from 72.3% to 76.2%. When using an independently trained model, the success rate increases from 70.3% to 73.8%.

The weaker results on the complete pick-and-place task indicate that local improvements do not necessarily translate directly into end-to-end performance gains. However, we believe this issue can be largely avoided by adopting a subgoal-based JEPA-style approach.

For example, we introduce a multi-stage Put in Drawer task, requiring the robotic arm to unlock and open a drawer, grasp and place a block inside, release it, and finally close the drawer. We compare vanilla CEM and ASAR under identical predefined semantic subgoals, initial conditions, and computational budgets. Under this controlled setting, full-task success increases from 0.0% with vanilla CEM to 31.67% with ASAR.

This suggests that, when applying ASAR to long-horizon tasks, it is preferable to decompose the task into multiple subgoals and execute them sequentially, so that ASAR's advantages in local action reconstruction can be utilized more effectively.

Regarding Portfolio, it is a variant of ASAR Kernel that we have explored. When applying Portfolio across different tasks, we do not modify its parameters. Through this exploration, we aim to lay the groundwork for developing more general methods in future work.

We believe that the exploration of Portfolio should not be considered a substitute for cross-task evaluation of the Kernel method itself.

AuthorsOpenReview ↗

Revised Manuscript and Experimental Corrections

We have uploaded a revised PDF with improvements to the writing and presentation.

During our re-examination of the experiments, we found that some of the "feasibility" labels in the previous candidate-budget audit were generated based on relative rankings among candidates and therefore could not be directly interpreted as indicating whether a candidate satisfied a fixed physical success criterion.

Specifically, this issue affected the following parts of the manuscript:

Page 4, §4: The 32-candidate-pool budget audit and the corresponding Top-1 description.

Page 5, Table 1 and Figure 2: The corresponding Position / Position+yaw results.

Appendix G.2, Table 9: The corresponding two rows.

Appendix G.3, Table 11: The results reported in this table.

We have corrected this issue, rerun the affected experiments, and updated the corresponding results and descriptions in the revised manuscript.

We hope that the revised manuscript and the additional clarifications address your concerns, particularly regarding the writing and presentation. We would greatly appreciate it if you could reconsider your assessment in light of these revisions.

Reviewer_AAZ2OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a new method for planning with latent world models using CEM. In conventional CEM, random action candidates are simulated using the world model, and their terminal states are evaluated with a cost function measuring the discrepancy from the goal state. However, the world model is not an oracle simulator of the environment. Its rollouts contain modeling errors, so some low-cost candidates are inevitably infeasible in the real environment. The paper calls this problem overgeneration: as the CEM pool size grows, these infeasible trajectories increasingly dominate the elite samples. The paper suggests a simple heuristic that averages a local cluster of low-cost samples. The proposed method shows marginal gains in specific environments and setups.

Critical strengths
  • Motivation. The proposed overgeneration problem is clearly explained, and it is worth exploring and of interest to the community. It hinders test-time scaling of latent world models and their application to complex real-world embodiment problems.
Critical weaknesses
  • Writing. The prose is hard to follow. Key terms such as “feasibility” and “reachability” are used from the abstract onward, but feasibility is only defined formally in Section 3, and reachability is defined only by citation. The figures also do little to demonstrate the core idea.

  • Method. ASAR does not appear to resolve overgeneration. It still relies on the same cost ranking, and success does not improve with budget. The method amounts to averaging a local cluster of low-cost samples, using a fairly ad-hoc rule. It beats plain averaging of the five lowest-cost sequences marginally, which does not seem to justify a contribution.

  • Experiments. Positive evidence is limited to Cube, with neutral results on Reacher and neutral or negative results on Push-T. The large gains come mainly from Carry and Release, a simple task in which the robot already holds the object.

Questions for authors
  • The proposed method should be validated on a more diverse set of environments. The current experiments are too limited.
  • The connection between the proposed overgeneration problem and the proposed method is not very clear to the reader. The assumption that feasible candidates have stronger local support than infeasible trajectory has weak evidence, and no principled theory is supporting this (at least in main).
  • The improvement from the proposed method is marginal or even negative, which makes the reader question the effectiveness of the approach.
3 comments
AuthorsOpenReview ↗

Response to Concerns on Environmental Diversity, Methodological Motivation, and Effectiveness

We thank the reviewer for the feedback regarding environmental diversity, the connection between the identified problem and our proposed method, and the empirical effectiveness of ASAR. We appreciate the opportunity to clarify the underlying motivation and provide additional experimental evidence.

  1. Environmental Diversity and Long-Horizon Evaluation. To address this concern, we have extended our evaluation beyond the original settings. In particular, we introduce a multi-stage Put in Drawer task, requiring the robotic arm to unlock and open a drawer, grasp and place a block inside, release it, and finally close the drawer.

We compare vanilla CEM and ASAR under identical predefined semantic subgoals, initial conditions, and computational budgets. Under this controlled setting, full-task success increases from 0.0% with vanilla CEM to 31.67% with ASAR. This result provides encouraging evidence that improvements in local action selection can translate into better completion rates in a more complex, multi-stage manipulation task.

In addition, we are extending our evaluations with matched world models. These experiments are now nearly complete, and we are currently verifying the results and consolidating the quantitative comparisons.To facilitate a timely and constructive discussion, we would like to share the available evidence first. We will provide the remaining results and corresponding analysis in a follow-up response within the next two days.

  1. Connection Between Overgeneration, ASAR, and the Underlying Mechanism. We appreciate the reviewer's question, as it provides an opportunity to clarify that overgeneration and ASAR are not two independent components, but rather the problem diagnosis and the proposed remedy within the same planning framework.

Overgeneration illustrates that sampling more candidate actions during planning can expand the range of possible actions, but also increases the chance of selecting actions associated with extreme model prediction errors. In our work, overgeneration therefore serves as a mechanism-level diagnosis, suggesting that execution failures may arise during the mapping from the candidate action set to the action ultimately executed.

ASAR takes this observation one step further. While retaining the original world model and CEM cost function, we introduce local support and action reconstruction into this mapping to address the overgeneration problem.

Our core intuition can be illustrated with two candidate actions, A and B. Suppose A has many similar neighboring actions that the model also predicts can accomplish the task, whereas even a small modification to B causes the model to predict failure. Given the continuity of physical dynamics, we hypothesize that A is more likely to generalize to nearby actions and to be genuinely feasible in the physical environment.

To examine this hypothesis, we conduct a controlled evaluation matching states, predicted costs, and sampling origins. We observe a 10.03% improvement in empirical physical feasibility associated with local support.

This provides empirical evidence for our central intuition: local support contains information about physical feasibility that is not fully captured by an individual candidate's predicted cost.

AuthorsOpenReview ↗

Follow-up: Additional Results with Matched Scene World Models

We thank the reviewer for their patience. As promised in our previous response, we would like to provide additional experimental results from our Scene evaluations.

We conducted two additional multi-stage manipulation evaluations in the Scene environment. Both tasks involve multiple operations with sequential dependencies, but emphasize different manipulation requirements. For each task, the corresponding Scene world model was trained for 50,000 steps. We used finer-grained expert-image subgoals shared by CEM and ASAR, decomposing the operations into stages such as approaching, aligning, pressing buttons, and operating handles.

Task 1: Closing and Locking the Drawer and Window

Initial state: The drawer and window are both open and locked, with a cube on the table.

Goal: Close and lock both the drawer and window while keeping the cube near its original position.

Procedure: Unlock the drawer by pressing its button, push it closed, and press the button again to lock it. Then unlock the window, slide it closed, and lock it again.

This task primarily evaluates button pressing, handle manipulation, lock-state transitions, and sequential dependencies.

Over 300 trials, CEM achieves 15.33% success, while ASAR achieves 27.67%, corresponding to a +12.34 percentage-point improvement.

Task 2: Placing a Cube in the Drawer and Opening the Window

Initial state: The drawer and window are both closed and locked, with a cube on the table.

Goal: Place the cube inside the drawer, close and lock the drawer, and open and lock the window.

Procedure: Unlock and open the drawer, approach and grasp the cube, lift it, place it inside the drawer, and release it. The robot then withdraws, closes and locks the drawer, unlocks and slides open the window, and locks it again.

This task additionally involves grasping, object transportation, placement, and coordination between multiple manipulation stages.

Over 300 trials, CEM achieves 0% success, while ASAR achieves 12.00%, corresponding to a +12.00 percentage-point improvement.

Summary of Additional Results

TaskCEMASARImprovement
Closing and Locking Drawer & Window15.33%27.67%+12.34 pp
Placing Cube in Drawer & Opening Window0%12.00%+12.00 pp

These experiments complement our previously reported multi-stage Put in Drawer evaluation, where the success rate improved from 0.0% with CEM to 31.67% with ASAR. The two new tasks involve additional window operations, lock-state transitions, and sequential dependencies.

Importantly, CEM and ASAR use the same finer-grained expert-image subgoals within each task. Therefore, the observed performance differences are evaluated under the same subgoal decomposition.

These results provide additional evidence that ASAR's improvements are not restricted to the original Cube Carry and Release setting. In particular, the two tasks cover different manipulation requirements, including lock-state transitions, articulated-object operations, and object transportation. We think the consistent improvements over CEM in both tasks further support the applicability of local action reconstruction to more complex multi-stage manipulation.

AuthorsOpenReview ↗

Revised Manuscript and Experimental Corrections

We have uploaded a revised PDF with improvements to the writing and presentation.

During our re-examination of the experiments, we found that some of the "feasibility" labels in the previous candidate-budget audit were generated based on relative rankings among candidates and therefore could not be directly interpreted as indicating whether a candidate satisfied a fixed physical success criterion.

Specifically, this issue affected the following parts of the manuscript:

Page 4, §4: The 32-candidate-pool budget audit and the corresponding Top-1 description.

Page 5, Table 1 and Figure 2: The corresponding Position / Position+yaw results.

Appendix G.2, Table 9: The corresponding two rows.

Appendix G.3, Table 11: The results reported in this table.

We have corrected this issue, rerun the affected experiments, and updated the corresponding results and descriptions in the revised manuscript.

We hope that the revised manuscript and the additional clarifications address your concerns, particularly regarding the writing and presentation. We would greatly appreciate it if you could reconsider your assessment in light of these revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.