acceptodds
Under review as a conference paper at ICLR 2027

A World Model for Assembly Sequence Planning

Abstract

This paper proposes a world model for planning assembly sequences for LEGO structures. The model predicts placement feasibility by looking ahead in a learned latent space, without invoking a physical simulator inside the look-ahead search. The primary technical challenge lies in judging whether committing to a placement leaves the remaining structure buildable, which requires reasoning over future steps. To address this, we represent the assembly with a multi-granularity heterogeneous graph, in which a master node aggregates the overall geometry of the assembled structure as a set of anchor tokens, while a brick-level sub-graph carries the interlocking topology and the static attributes of each brick. A latent dynamics model advances the anchor tokens locally after each placement, which lets the world model roll the geometry forward over several steps and anticipate whether a candidate placement obstructs the assembly of the remaining bricks. We validate the framework on held-out test structures from the StableText2Lego benchmark, comparing it against ablations and recent assembly-sequence methods. Our planner achieves near-oracle solutions while issuing the fewest exact-oracle calls per brick and avoiding the later-infeasible placements incurred by the baselines.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Modeling, Sequence Planning, LEGO Assembly

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_r52JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses a learned world model to plan LEGO assembly sequences through look-ahead, with exact checks before committing actions. It reports higher success rates and fewer oracle calls than the evaluated baselines.

Critical strengths

The paper tackles a practical planning problem where a feasible action can lead to a dead end.

Combining learned look-ahead with exact checks before committing actions is a sensible design.

The ablations provide evidence that look-ahead and broader geometric context improve performance.

Critical weaknesses

The problem is interesting, and combining learned look-ahead with exact checks before committing actions makes sense. However, I have several concerns:

  1. Timeouts are excluded from the success rate. The reported 95.4% is based on 925 of the 1,200 test structures. Excluding harder cases could make the results overly optimistic. Please report results on the full test set, counting timeouts as failures.

  2. The benefit of learned dynamics is unclear. Removing a brick gives a known next state. Why predict its latent representation instead of updating the structure and re-encoding the affected anchors? This is a key missing baseline.

  3. Fewer oracle calls do not mean faster planning. In Table 1, depth 3 improves success from 80.5% to 83.9%, but increases median runtime from 43 to 137 seconds. Please compare success rates under the same time budget.

  4. The local search restriction needs justification. Removing a brick can affect distant collisions or stability. Why does look-ahead only expand neighboring bricks? Are other feasible actions still considered?

  5. The feasibility checks need a shared configuration. As written, press, reach, and collision can each pass under different gripper configurations. A valid action needs one configuration satisfying all three. Does the implementation enforce this?

  6. The baseline comparison needs clarification. ASAP uses pretrained weights from another setting, and its near-zero success rates deserve a failure analysis. Does the WorkBenchMark implementation preserve its full search procedure?

  7. Relevant search-based work is missing. State-Based Disassembly Planning (Lei et al., AAAI 2025) reduces redundant simulation through intermediate-state reuse and directional blocking graphs. Although the task assumptions differ, please discuss this work and explain what learned dynamics adds beyond improved search and state reuse. An adapted baseline would help separate these effects.

Overall, the approach looks promising, but the current experiments do not yet clearly establish the added value of the learned world model.

Questions for authors

See Weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer r52J (1/2)

We thank the reviewer for the insightful comments and feedback. Below we address the questions and concerns:

Q1: Timeouts are excluded from the success rate. The reported 95.4% is based on 925 of the 1,200 test structures. Excluding harder cases could make the results overly optimistic. Please report results on the full test set, counting timeouts as failures.

We report the full test set of 4,785 structures.

armsuccess rate on all non-timeouts structuressuccess, timeouts as failures
wm/rollout d3 (ours)0.93950.9049
oracle/none d0 (exact oracle as judge)0.92240.8566
reach/rollout d30.83640.8251
reach/none d00.81410.7889
seq/none (no judge)0.81410.7889

With timeouts counted as failures, our method remains the strongest.

Q2: The benefit of learned dynamics is unclear. Removing a brick gives a known next state. Why predict its latent representation instead of updating the structure and re-encoding the affected anchors? This is a key missing baseline.

The necessity of latent dynamics model is that it avoids re-deriving the global quantities in latent rollouts. An incremental encoder must still maintain the occupancy grid, the exposed-face set, and the per-brick contact counts for the next state, while the latent dynamics model infers the change to latent features from the given brick alone and reconstructs nothing else. Updating the latent through the dynamics model takes 0.34 milliseconds, against 1.56 milliseconds to produce the same update by re-encoding. The substitution is lossless: against a control that re-encodes the exact next structure, the two agree under a paired test with p equal to 1, at 0.0175 against 0.0178 misplacements per brick.

Q3: Fewer oracle calls do not mean faster planning. In Table 1, depth 3 improves success from 80.5% to 83.9%, but increases median runtime from 43 to 137 seconds. Please compare success rates under the same time budget.

The main table is already produced under a fixed wall-clock budget of 1800 seconds per structure. The depth-3 and depth-0 numbers are therefore already a same-budget comparison.

AuthorsOpenReview ↗

Response to Reviewer r52J (2/2)

Q4: The local search restriction needs justification. Removing a brick can affect distant collisions or stability. Why does look-ahead only expand neighboring bricks? Are other feasible actions still considered?

We want to clarify a possible misreading, since these are two separate things. Collision and stability are always checked globally: every feasibility check reads the full set of anchor tokens, not a neighborhood, so when removing a brick, our method can detect distant collisions or instability. The neighborhood restriction applies only to which future candidates a rollout expands when it looks ahead, which is a pruning of the search tree. The set of actions that can actually be committed is always fully enumerated. We select one candidate from that full set and commit a single step, so there is no requirement that the next step be near the previous one.

Q5: The feasibility checks need a shared configuration. As written, press, reach, and collision can each pass under different gripper configurations. A valid action needs one configuration satisfying all three. Does the implementation enforce this?

Yes. A valid action enforces a single gripper configuration that satisfies press, reach, stability, and collision together, and this is shown in equation 1 as a conjunction of the five predicates.

Q6: The baseline comparison needs clarification. ASAP uses pretrained weights from another setting, and its near-zero success rates deserve a failure analysis. Does the WorkBenchMark implementation preserve its full search procedure?

On ASAP, we retrained ASAP on our own data with our exact LP stability labels, and we report a ladder on the 75 small structures under the same exact ruler: success rises from 2 of 75 with the original weights, to 6 of 75 when those weights are paired with our LP, to 11 of 75 when ASAP is retrained on our data, to 26 of 75 when it is both retrained and scored with our LP. For reference, our planner solves 49 of 75 at k=1k=1 and 75 of 75 at k≤5k \le 5 on the same structures. Retraining changes the order of the search, but it still cannot pass the collision verification, which indicates the contribution of the world model and collision head.

On WorkBenchMark, yes, our implementation preserves its full search procedure.

Q7: Relevant search-based work is missing. State-Based Disassembly Planning (Lei et al., AAAI 2025) reduces redundant simulation through intermediate-state reuse and directional blocking graphs. Although the task assumptions differ, please discuss this work and explain what learned dynamics adds beyond improved search and state reuse. An adapted baseline would help separate these effects.

We will add and discuss this work. We note first that intermediate-state reuse is not a point of difference, since we also memorize repeated states, with a hit rate of 64.3 percent inside the look-ahead. Scoped to the latent dynamics model, it predicts states that are never materialized, whereas both state reuse and directional blocking graphs accelerate evaluation on states that must first be constructed. Moreover, it is trained once and amortized across all held-out structures, whereas a directional blocking graph is built per assembly.

Reviewer_7H7WOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a learned “world model” for assembly sequence planning of LEGO structures with a dual-arm robot cell. The planner works by disassembly, removing bricks (or rigid groups of up to 5) from the finished structure one at a time and reversing the order. A step is feasible if it passes five predicates: two deterministic gates (gripper press point and IK reachability), and learned collision and stability predicates. Collision is predicted from “anchor token” latents of the built structure. Stability is predicted by a graph network over the brick adjacency graph. A latent dynamics model updates the anchor tokens after each imagined removal, so the planner can run a depth-3 look-ahead without calling the exact collision oracle. Each committed step is still re-checked by an exact collision oracle and an exact LP stability solver, so every returned sequence is verified.

Critical strengths
  1. A pragmatic and sound system design. The learned components never decide correctness. Every committed step is verified by exact solvers, so a returned sequence is valid by construction, and the learned parts only guide search. This is a sensible way to use learning in a safety-relevant pipeline.

  2. Careful empirical hygiene on the questions the paper does ask. The look-ahead depth study uses paired tests and reports the cost of deeper rollouts and the null results. The paper also states explicitly that timeouts change the evaluable subsets across tables.


I've read the authors' rebuttal and most of my concerns have been addressed. I've raised by my rating.

Critical weaknesses

W1. The need for a learned model is not established. The natural alternatives, e.g., optimized exact search and combinatorial or constraint solvers, are never tested. The paper's motivation is that exact look-ahead is too expensive, and it supports this with a single naive baseline: rolling the exact oracle forward, which times out on 26 of 75 structures. It does not show that the cost is inherent rather than an implementation artifact. There is no optimized exact search (caching or memoizing collision and stability checks, incremental evaluation, pruning), and no assembly-by-disassembly formulation solved by a constraint or OR solver. Assembly sequence planning has a long combinatorial-optimization literature that the related work does not discuss. Since the exact oracle is still run at every commit, the learned model works as a search heuristic. Without evidence that the learned component buys a speed-quality tradeoff that exact methods cannot match, the paper does not show that a world model is needed.

Jiménez (2013), “Survey on assembly sequencing: a combinatorial and geometrical perspective”, J. Intelligent Manufacturing 24(2):235–250. Liu & Zeng (2009), “A survey of assembly planning based on intelligent optimization algorithms”.

W2. The "world model" framing and the headline claims go beyond what the evidence shows. Most of the "dynamics" being modeled is the removal of a brick from a known occupancy grid, which is deterministic and exactly known. The latent dynamics model only substitutes for re-encoding the true next structure, and the main text gives no evidence that this is necessary or beneficial.

Questions for authors
  1. How does the method compare against an optimized exact search (memoized or incremental oracle calls, with the same pruning) and against a constraint/OR formulation, at the same wall-clock budget? Which component dominates the exact baseline’s cost (IK, collision, or the LP), and what are per-call costs?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 7H7W

We thank the reviewer for the insightful comments and feedback. Below we address the questions and concerns:

Q1: How does the method compare against an optimized exact search (memoized or incremental oracle calls, with the same pruning) and against a constraint/OR formulation, at the same wall-clock budget? Which component dominates the exact baseline’s cost (IK, collision, or the LP), and what are per-call costs?

On the question of optimized exact search, the exact search baseline in our main experiments already uses memoization, pruning, and incremental evaluation on collision(Table 2, the "Exact-oracle + look" row).

On the per-call cost of the exact oracle, we measured it and found that stability is the factor that grows with structure size, increasing from 0.006s to 1.677s across the test set, and becomes the dominant cost component at around 130 bricks. Inverse kinematics (26 ms to 58 ms) and collision scan (0.51 s to 0.82 s) do not dominate at large structures. Moreover, the stability LP is not incremental, as removing a brick changes the global force balance non-monotonically.

On the question of a constrained solver, we implemented one for assembly-by-disassembly, following the same blocking-graph formulation as Culbertson et al. (IROS 2019), using a static blocking set with CP-SAT and a stability lazy no-good. The model decides a partition of the bricks into rigidly-coupled groups of size at most five, together with an order over those groups. For a group gg placed at step s[g]s[g], let t[i]t[i] be the step of brick ii. The constraints are the following, where x[g]x[g] selects a group, y[g,o]y[g,o] selects one minimal blocking set BoB_o as the geometric condition, and p[i,j]p[i,j] is the ground-support parent pointer:

AllDifferent(s)\text{AllDifferent}(s)

∑g∋ix[g]=1\sum_{g \ni i} x[g] = 1 for all ii

x[g]=1⇒t[i]=s[g]x[g]=1 \Rightarrow t[i]=s[g] for all i∈gi \in g

∑oy[g,o]=x[g]\sum_{o} y[g,o] = x[g]

x[g]=1⇒t[j]<s[g]x[g]=1 \Rightarrow t[j] < s[g] for all j∈(⋃i∈gabove(i))∖gj \in \left(\bigcup_{i \in g}\text{above}(i)\right)\setminus g

y[g,o]=1⇒t[j]<s[g]y[g,o]=1 \Rightarrow t[j] < s[g] for all j∈Boj \in B_o

∑j∈N(i)p[i,j]+p[i,⊥]=1\sum_{j \in N(i)} p[i,j] + p[i,\perp] = 1

p[i,j]=1⇒t[j]≥t[i]p[i,j]=1 \Rightarrow t[j] \ge t[i]

The objective is feasibility, that is, finding any assignment satisfying all constraints within the budget.

A complete solver must enumerate every connected subset of size at most five, and the number of such groups grows as Θ(ndˉ k−1)\Theta(n \bar{d}^{\,k-1}), where nn is the number of bricks in the structure, k≤5k \le 5 is the maximum group size, and dˉ\bar{d} is the mean degree of the vertical contact graph. This is the tradeoff our world-model rollout avoids, since the rollout scores candidates as they arise rather than enumerating the whole grouped space in advance.

We compared our rollout world model with the exact solver on 120 benchmark structures stratified by brick count, under the same 1800-second per-structure budget as our main table and with every method scored by the same exact ruler. Counting timeouts as failures, the OR solver succeeds on 40 of 120 structures (33.333.3 percent), while our world-model rollout succeeds on 110 of 120 (91.791.7 percent). The gap widens with size: above 100 bricks, the OR solver succeeds on only 10 percent of structures.

Q2: The "world model" framing and the headline claims go beyond what the evidence shows. Most of the "dynamics" being modeled is the removal of a brick from a known occupancy grid, which is deterministic and exactly known. The latent dynamics model only substitutes for re-encoding the true next structure, and the main text gives no evidence that this is necessary or beneficial.

The necessity of the latent dynamics model is that it avoids re-deriving the global quantities during latent rollouts. An incremental encoder must still maintain the occupancy grid, the exposed-face set, and the per-brick contact counts for the next state, whereas the latent dynamics model infers the change to the latent features from the given brick alone and reconstructs nothing else. Updating the latent through the dynamics model takes 0.340.34 milliseconds, against 1.561.56 milliseconds to produce the same update by re-encoding. The substitution is lossless: against a control that re-encodes the exact next structure, the two agree under a paired test with pp equal to 11, at 0.01750.0175 against 0.01780.0178 misplacements per brick.

Reviewer_aza8OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors tackle a key problem in assembly sequence planning: looking ahead in the sequence of assembly steps to observe physical feasibility since a feasible step immediately ahead may lead to an infeasible assembly later down the line. This work builds upon ASAP by adding a world model which predicts whether a placement is feasible as more bricks are removed. This model represents the assembly as a multi-granularity heterogeneous graph which allows for reasoning over the global geometry (collisions) and local connectivity (stability). An encoder converts the geometry of the assembly into latent vectors, a dynamics model advances the latents after each rollout, and a planner rolls the world model forward to score each candidate part to remove before committing to a part. On the StableText2Lego benchmark, the world model achieves greater success rates compared to greedy ordering and single-step exact oracle predicate.

Critical strengths
  • The manipulation-level world model is well thought out: encoder mapping anchor neighborhoods to latent representation, collision predicate, deterministic gates, stability predicate for before and after a group is detached, and having a dynamics model renew only latents of anchors near the placement.
  • The paper’s multi-granularity graph is a reasonable representation that allows the world model to both reason about global collision geometry and local interlocking connectivity. This is an improvement over generic latent rollouts which fail to preserve the physical validity of the assembly.
  • Gripper collision geometry is considered which makes the method more feasible for sim-to-real.
Critical weaknesses
  • This approach beats purely planner-based approach like ASAP and WorkBenchMark, but no baselines compared to other world-model approaches were run such as “H-WM: Hierarchical World Models for Robotic Planning (2026)” or “PhysWorld: Robot Learning from a Physical World Model (2025)”. Evaluating the improvements over these benchmarks is crucial in my rating.
  • No limitations section was provided by the authors.
Questions for authors
  • Only one dataset (StableText2Lego) was tested on - but this dataset does not include non-brick mechanical parts (joints, pegs, axles, gears) and directional assembly constraints (e.g. cross-axis insertions). Would this method accommodate more diverse assembly datasets and what are the main challenges for these other kinds of tasks?
  • What are some reasons why increasing the depth past 5 does not see any gain over depth 3?
3 comments
AuthorsOpenReview ↗

Response to Reviewer aza8 (1/2)

We thank the reviewer for the insightful comments and feedback. Below we address the questions and concerns:

Q1: This approach beats purely planner-based approach like ASAP and WorkBenchMark, but no baselines compared to other world-model approaches were run, such as “H-WM: Hierarchical World Models for Robotic Planning (2026)” or “PhysWorld: Robot Learning from a Physical World Model (2025)”. Evaluating the improvements over these benchmarks is crucial in my rating.

We thank the reviewer for pointing to these works.

H-WM has no public code release, and its setting includes an RGB observation stream and a downstream VLA, whereas our task operates on a brick set defined on an integer stud grid without vision. We therefore re-implemented its logic-only ablation (Section 5.4), following its logical world-model design exactly as described in its Section 4.1, and trained it with our own data.

For the training setup, we follow H-WM directly, which adopts the training formulation of Self-CriTeach, arXiv:2509.21543. We therefore reuse that pipeline: a full fine-tune of Qwen3-4B, learning rate 1e−51\mathrm{e}{-5}, global batch size 16, cosine schedule, AdamW, trained for up to 10k steps.

Because the H-WM logical world model is an LLM with 4 billion parameters and takes about 20 s per step, running it over the full test set is not feasible within our rebuttal window. We therefore evaluate it on 120 benchmark structures stratified by brick count: the logical world model solves 101 of 120 structures, while ours solves 110 of 120.

PhysWorld (arXiv:2511.07416) has a different task setting from ours, as it aims to enable robot learning from video generation through physical world modeling. We re-implemented their method on our task by building the physical scene from our structures and training the only trainable component in their released code, a residual PPO policy, on our data, while leaving out their video generation, monocular reconstruction, texturing, and pose estimation, which are designed for their tasks.

Because this PPO policy is a small MLP ($\sim$367k parameters) that infers in milliseconds, we were able to evaluate it on the same 1,2001{,}200-structure test split as in Table 2 in the paper within the rebuttal window. The PhysWorld residual-PPO arm reaches a success rate of 0.8250.825 (confidence interval [0.802,0.846][0.802, 0.846]), versus 0.9540.954 for our method. The gap is significant under a paired McNemar test: among the structures the two methods disagree on, PhysWorld-PPO succeeds on 17 while our method succeeds on 128 (p=3.0e−22p = 3.0\mathrm{e}{-22}). This baseline learns to drive the number of exact feasibility queries down to our level, 0.5390.539 exact-oracle calls per brick, essentially on par with our 0.5110.511, but it does not learn physical feasibility: its misplacement rate is 0.1040.104 per brick, 5.4×5.4\times ours (0.0190.019). It achieves this by aggressive grouping, disassembling an average of 2.442.44 bricks per step, while ours is 1.141.14 bricks per step, rather than by better feasibility judgment.

The two baselines show that generic latent rollouts do not, by themselves, preserve physical validity, which is exactly the gap our learned manipulation-level collision and stability predicates are designed to close.

Q2: No limitations section was provided by the authors.

We thank the reviewer for pointing this out. We summarize the limitations below and will report them in the camera-ready version.

The pipeline is validated in simulation with analytic verifiers, leaving real-robot execution under sensing noise and calibration for future work. We model static equilibrium but not the dynamic contact forces that arise during insertion and pressing. Closing these gaps is a natural next step toward physical deployment.

AuthorsOpenReview ↗

Response to Reviewer aza8 (2/2)

Q3: Only one dataset (StableText2Lego) was tested on - but this dataset does not include non-brick mechanical parts (joints, pegs, axles, gears) and directional assembly constraints (e.g. cross-axis insertions). Would this method accommodate more diverse assembly datasets and what are the main challenges for these other kinds of tasks?

Our method does not currently handle non-brick mechanical parts, as we use an end-of-arm tool specifically designed for brick assembly tasks. Brick manipulation requires sub-millimeter accuracy under tight tolerances, and the cluttered, evolving structure causes traditional two-finger grippers to suffer from occlusion when assembling or disassembling against existing bricks. These task-intrinsic demands are what motivate a specialized tool that seats bricks through stud interlock, rather than a general-purpose gripper that grasps bricks on the side. Other assembly tasks may likewise face embodiment requirements that are specific to the task.

Directional assembly constraints are already present in our formulation. A placement is executed by one of 112 gripper configurations, defined by which arm acts, which of the four sides of the brick the tool presses from, which of seven tool offsets along that side is used, and which of two directions the tool approaches from. The last three choices are directional, and their feasibility depends on the surrounding structure.

Q4: What are some reasons why increasing the depth past 5 does not see any gain over depth 3?

We further measured how far ahead a decision actually needs to see in order to avoid a dead end. Across 1200 tested structures the required look-ahead span has a 90th percentile of 3 steps, so a depth of 3 already covers roughly 90 percent of the cases where looking ahead matters at all. The remaining cases form a long tail whose span reaches 9 steps. Depth 5 does not reach into that tail either, so raising the depth from 3 to 5 adds cost without covering the cases that would actually benefit.

Thank you for the responses to my questions. I appreciate the addition of more baselines and including a limitations section.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Summary

Dear AC and the reviewers,

Thank you for your time and insightful feedback. We appreciate the reviewer's recognition of the strengths of our work, including the well-thought-out manipulation-level world model, the pragmatic and sound system design, the suitability of the multi-granularity graph representation, the incorporation of gripper-level geometry to facilitate sim-to-real transfer, and the integration of learned look-ahead with exact checks before committing to actions.

We have addressed each reviewer's questions and concerns individually, including comparisons with H-WM and PhysWorld, the lack of a limitations section, the motivation for using latent dynamics and learned models, and the exclusion of timed-out examples from metric computation.

We have also uploaded revised versions of the paper and supplementary material. A summary of the revisions (marked in blue in pdf) is provided below.

  1. Main table (Table2, left) now reporting timesouts as failures. Recreated Table 2 (left) to report success on the full held-out test split (4,785 structures) with timeouts counted as failures, and buildable-assembly success is now 90.5% on the full split (greedy 78.9%, single-step exact-oracle 85.7%).
  2. Three new baselines were added to the comparison. Two world-model baselines (H-WM, PhysWorld), re-implemented from their papers/code and retrained on our StableLego split. Full implementation details are described in supp. An OR-solver baseline is re-implemented from its paper, full formulation in supp. All three are new rows in Table 2 (right), each annotated with its evaluation subset, plus a new discussion paragraph in §4.4.
  3. ASAP result revised. Change ASAP result to a retrained version with our LP stability verifier. (original 2/75 →retrained 26/75; ours 49/75 at k=1, 75/75 at k≤5).
  4. Clarified the latent dynamics head. Made explicit that it reads only the current latent and the placed-brick featurization, and not next-state occupancy, exposed-face set, or per-brick contact counts. Added a comparison in supplementary showing that the latent update is lossless (paired p=1; 0.0175 vs. 0.0178 mis-placement) at ~0.34 ms vs. 1.56 ms re-encoding.
  5. Limitations added.
  6. State-based disassembly planning discussion (Lei et al., AAAI 2025) added to Related Work.
  7. Editorial/correctness. Added four new references with verified metadata.

Thanks,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.