acceptodds
Under review as a conference paper at ICLR 2027

OSTRA: Learning Multi-Stage Robot Manipulation through Object-State Transition Supervision

Abstract

Sparse supervision is a key challenge in multi-stage manipulation, with the lack of supervision over the past-to-future evolution of objects required by the policy being particularly critical. To address this issue, we introduce the Object-State Transition Map (**OST-Map**), a unified object-centric representation that organizes task-relevant objects as identity-consistent graphs and represents historical and future object states in a shared structured space. Building on OST-Map, we propose **OSTRA**, a diffusion-based Object-State Transition Reasoning Architecture that hierarchically predicts future object states, target TCP poses, and continuous action trajectories. OSTRA combines Transition-Guided Attention, which prioritizes historical object-state transitions during future-state prediction, with Staged Denoising, which progressively translates predicted object changes into robot subgoals and executable actions. Experiments on two simulation benchmarks and five diverse real-world tasks demonstrate consistent improvements over representative imitation learning baselines. Further ablations validate the contributions of OST-Map and the proposed architectural designs.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Control Policy, Embodied Intelligence

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YJVoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents OSTRA for multi-stage robot manipulation. The main idea is to keep track of how task-relevant objects have changed and use predictions of their future states to guide robot actions. Objects are represented using identity-consistent structural graphs, and the policy predicts future object positions, end-effector subgoals, and continuous actions in sequence. Experiments on three simulation benchmarks and seven real-world tasks report improvements over several imitation learning baselines.

Critical strengths
  1. The paper addresses a meaningful problem: a robot performing a multi-stage task needs to understand what has already happened to the objects, not just their current appearance. Using identity-consistent object histories and future targets is a sensible way to make this information easier to learn.

  2. The evaluation covers several simulation benchmarks and seven real-world tasks. The architectural ablations and perception-error analysis provide useful evidence that the proposed representation and prediction hierarchy contribute to the reported performance.

Critical weaknesses
  1. The training-data pipeline is insufficiently specified for reproducibility. Although the architecture and losses are described, the paper does not clearly explain how the structural estimator is trained, how historical and future keyframes are selected, or how future object-position and TCP targets are obtained and aligned. These are central components of the method, so the omissions make it difficult to reproduce the results and assess the additional supervision required.

  2. The dependence on task-specific preparation is not sufficiently evaluated. Appendix B.4 specifies task-dependent structural templates, but the effort needed to construct them and their reuse across objects or tasks remain unclear. The reported evaluations do not isolate transfer to unseen objects with an unchanged template. Consequently, it is difficult to assess how much of the practical benefit depends on task-specific knowledge and preparation.

Questions for authors
  1. Data preparation: Which templates, affordance annotations, geometric labels, and keyframes are manually specified or additionally supervised? How is the structural estimator trained, and how are future object and TCP targets obtained and aligned? Please clarify the preparation cost for a new task and whether baselines receive comparable information.

  2. Generalization: Can the method handle unseen object instances or sizes without changing templates or retraining specialized regressors? Such an evaluation would help establish its transferability. Please also clarify whether LIBERO uses per-task or shared policies, and how language instructions or task identities enter the model.

  3. Future-state targets: Since future supervision contains node positions but not orientations, how are rotations handled for single-node objects? A position-only versus position-and-orientation comparison on an alignment-sensitive task would help assess this limitation.

  4. Method clarity: Could the authors provide stage-wise input/output equations with tensor dimensions and a training algorithm? Please explicitly distinguish direct object regression from TCP/action denoising, identify frozen/trainable modules, and specify action representations, noise schedules, and loss normalization.

  5. Deployment: Does the reported policy latency include the full reverse-diffusion process, and with how many steps? How are perception, replanning, and action execution scheduled? Please report full-system peak GPU memory and latency variability; a success–latency comparison at different update frequencies would help assess practical responsiveness.

6 comments
AuthorsOpenReview ↗

Responses to Question 1

(Q1) Data preparation and target alignment

Q1. Which templates, affordance annotations, geometric labels, and keyframes are manually specified or additionally supervised? How is the structural estimator trained, and how are future object and TCP targets obtained and aligned? Please clarify the preparation cost for a new task and whether baselines receive comparable information.

We address the five parts of the question in order:

  1. Templates, annotations, geometric labels, and keyframes. We define eight structural primitives and compose them into reusable object templates before task selection. Primitive definitions provide structural and interaction-region affordance samples; object templates specify part decomposition, node order, geometric parameter slots, and within-object relations. GPT-5.6 selects relevant templates from the first RGB frame and task description, and the selected object names or descriptions serve as SAM 3 prompts. Cross-object pairs receive Free relations; instance masks, poses, and geometric parameters are estimated from RGB-D observations. No task-specific template, instance-level geometric label, or affordance annotation is manually supplied for each task whose object structures are covered by the library. Keyframes are selected by gripper-state changes, zero TCP velocity, and episode termination, rather than manually annotated.
  2. Structural estimator training. The structural estimator is trained jointly with the policy under the stated objective, without a separate estimator dataset or pretraining stage (Appendix A.1).
  3. Future object and TCP targets. For training at frame tt, we select the next HkH_k keyframes strictly after tt, repeating the last available frame near the end of a trajectory. Future object-node positions and TCP targets use the same indices, while the action chunk starts at tt. Node positions come from processing future demonstration frames with FoundationPose and the OST-Map constructor, rather than reading simulator state; TCP targets come from the recorded robot trajectory. Future frames supply training targets only. At inference, historical keyframes are recorded online from past TCP velocities and gripper states. This alignment is described in the revised paper (Appendix A.1).
  4. Preparation cost for a new task. A task whose object structures are covered by the library reuses existing templates without manual task-specific construction. Composing an additional object template from existing primitives typically takes under 10 minutes; a genuinely new topology requires extending the template library. The vocabulary and construction procedure are detailed in the revised paper (Appendices B.2 and B.4).
  5. Comparable baseline information.
    1. Structural knowledge. Several baselines also use object-level representations that provide meaningful points of comparison: PPI predicts object pointflow and gripper keyposes, SPOT predicts object pose trajectories, and Im2Flow2Act uses object flow. Their representations differ from OST-Map, but they likewise exploit structured object information. OST-Map's explicit structural representation is itself a contribution and advantage of OSTRA. Its reusable object templates are selected from the first observation and task description, rather than supplying ground-truth task objects or states. We compare the complete methods and use OST-Map ablations to assess the contribution of this representation.
    2. Observations. OSTRA and the baselines use the same camera observations and robot proprioceptive information under the corresponding benchmark protocols. The comparison therefore holds the available sensory inputs and evaluation conditions fixed while allowing each method to use its own representation.
    3. Matched structured-input experiment. We also give PPI and SAM2Act+ the same historical OST-Map tokens used by OSTRA on RLBench2, and separately add the same future-node prediction target as auxiliary supervision. With OST-Map input and future-node supervision, PPI reaches 85.0% success and SAM2Act+ reaches 86.9%, compared with 89.8% for OSTRA under the same evaluation protocol. The controlled results, including each baseline's original and OST-Map-only variants, are reported in the revision (Appendix A.6).
AuthorsOpenReview ↗

Responses to Question 2, 3

(Q2) Unseen instances and LIBERO conditioning

Q2. Can the method handle unseen object instances or sizes without changing templates or retraining specialized regressors? Such an evaluation would help establish its transferability. Please also clarify whether LIBERO uses per-task or shared policies, and how language instructions or task identities enter the model.

  1. Transfer scope. For a new instance, size, or arrangement within a covered task family, the existing template, estimator parameter slots, and trained policy can be reused when the part topology and policy node vocabulary remain unchanged; perception updates instance dimensions and poses. A genuinely new part topology requires an additional object template. If it adds parameter slots or nodes, the estimator heads or policy vocabulary must be extended accordingly.
  2. LIBERO policies and language.
    1. Language conditioning on LIBERO. We encode the task instruction with a frozen CLIP ViT-B/32 text encoder, project its embedding into a language token, and concatenate that token directly with the Semantic Field scene tokens before policy cross-attention. We use the same input during training and evaluation (Appendix A.3). This simple adaptation lets OSTRA follow LIBERO's language-conditioned evaluation protocol for a fair comparison; we do not introduce a specialized language-processing architecture.
    2. Shared multi-task policy. Following MemoryVLA's setup, we train one shared policy for each of the Spatial, Object, and Goal suites, each containing 10 tasks. Long-10 and Long-90 jointly train one Long policy, and we report its Long-10 evaluation. We use the filtered LIBERO RLDS demonstrations released for MemoryVLA. Before training each policy, the union of objects selected across its training tasks defines a fixed vocabulary and node order. At inference, GPT-5.6 selects active templates from the first RGB frame and instruction; slots for absent objects are zero-filled. This procedure is specified in the revised paper (Appendix A.3; Algorithm 2 in Appendix B.4).

(Q3) Future-state targets and rotation

Q3. Since future supervision contains node positions but not orientations, how are rotations handled for single-node objects? A position-only versus position-and-orientation comparison on an alignment-sensitive task would help assess this limitation.

  1. Why future supervision focuses on position. The future-state target is designed to give the policy a simple, memory-like account of long-horizon object-state transitions. It therefore supervises changes in node positions rather than future node rotations. Rotation changes needed for precise alignment are more closely tied to short-horizon action generation, which the TCP and action stages handle using current observations and historical OST-Maps that include observed orientations. A single-node object's future orientation is not directly predicted, but the policy can still generate rotational actions.
  2. Effect of adding rotation supervision. We compare four variants on all seven RLBench2 tasks: no future prediction, future position, future rotation, and future position plus rotation. They use the same demonstrations, OST-Map templates, perception pipeline, training budget, and evaluation seeds; the prediction variants use Hk=4H_k=4 future keyframes. We report mean success over three checkpoints with 100 rollouts per task. Adding rotation to position improves mean success by just 0.1 percentage point (89.8% to 89.9%), with a gain of at most 0.3 points on any task. The negligible gain across the evaluated tasks supports our choice to use position for future-state supervision and leave precise rotational control to the policy. Full experimental details are provided in the revision (Appendix A.6).
RLBench2 taskNo future prediction (%)Position (%)Rotation (%)Position + rotation (%)
Box95.098.396.098.3
Ball88.093.088.093.0
Drawer80.087.082.087.3
Laptop75.082.776.083.0
Dustpan95.097.796.097.7
Tray87.092.088.092.0
Handover Easy71.577.773.278.0
Overall84.589.885.689.9
AuthorsOpenReview ↗

Responses to Question 4

(Q4) Prediction stages and training details

Q4. Could the authors provide stage-wise input/output equations with tensor dimensions and a training algorithm? Please explicitly distinguish direct object regression from TCP/action denoising, identify frozen/trainable modules, and specify action representations, noise schedules, and loss normalization.

The revised paper lists the stage-wise inputs, outputs, supervised targets, and tensor dimensions in the new tensor-layout table (Table 7 in Appendix A.1). The same appendix gives the joint minibatch training procedure; we summarize the key distinctions below.

  1. Stage targets and action representation. The object stage directly regresses normalized future node positions [B,Hk,N,3][B,H_k,N,3]; its target is not diffused. The TCP stage predicts noise [B,R,Hk,DE][B,R,H_k,D_E]. For 7D actions, the action stage predicts pose noise [B,R,Ha,7][B,R,H_a,7] for [x,y,z,qw,qx,qy,qz][x,y,z,q_w,q_x,q_y,q_z] and directly predicts gripper openness with a separate [B,R,Ha,1][B,R,H_a,1] head. For 4D actions [x,y,z,g][x,y,z,g], only XYZ is diffused; the fourth output directly supervises gripper openness. A 4D TCP target [x,y,z,gkeyframe][x,y,z,g_{\mathrm{keyframe}}] differs: all four channels are diffused. The reference horizons are To=Hk=4T_o=H_k=4 and Ha=16H_a=16.
  2. Noise schedules. Each training item independently samples a timestep from {0,…,999}\{0,\ldots,999\} and Gaussian noise for TCP and action targets. Both schedulers use epsilon prediction without sample clipping: position uses a scaled-linear beta schedule, and quaternion channels use a squared-cosine schedule. Inference uses eight DDIM reverse steps; each step reruns the object, TCP, and action stages, while the object prediction itself receives no scheduler update.
  3. Loss normalization and training algorithm. The object, TCP-noise, action-noise, and gripper losses are separate mean L1L_1 terms over their valid elements; a node mask, when present, restricts the object loss to valid XYZ scalars. Their reference weights have equal ratios relative to the action loss. For each minibatch, we normalize targets, sample timesteps and noise, build noisy TCP/action inputs, execute object →\rightarrow TCP →\rightarrow action, compute the four losses, and jointly update the trainable parameters. Stop-gradient is applied to the latent plans passed between stages, while shared context encoders still receive gradients.
  4. Frozen and trainable modules. The pretrained visual backbone and LIBERO CLIP ViT-B/32 text encoder are frozen; SAM 3 and FoundationPose supply observation-derived inputs outside the gradient graph. The structural estimator, Semantic Field and OST-Map encoders, language-token projection, prediction backbone, and output heads are trainable. AdamW updates the trainable parameters, and an EMA copy is updated after optimizer steps. Staged inference is specified separately (Algorithm 1).
Reviewer_oXJ8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OSTRA, an imitation-learning framework that uses explicit object-state transitions as intermediate supervision for multi-stage manipulation. Its Object-State Transition Map (OST-Map) represents task-relevant objects through persistent structural nodes, geometric and affordance attributes, and relations. Historical maps provide temporal context, while future structural-node positions serve as prediction targets. The policy first predicts future object states, then TCP subgoals, and finally continuous actions. Transition-Guided Attention processes map tokens before dense visual features, and stop-gradient boundaries separate the prediction stages. The evalua

Critical strengths
  1. The motivation is clear, explicitly supervising how objects change provides a useful intermediate representation between visual observations and robot actions.
  2. The experiment results are strong and strengthen the paper.
  3. The ablations prove meaningful design choices.
Critical weaknesses
  1. Appendix B.4 states that each task has a structural template specifying relevant objects, primitive decompositions, node ordering, and geometric parameters. Known relations can also be specified through templates, and structural estimation uses task-specific regressors. This is a substantial source of prior information. The paper should clarify how these templates, affordance regions, and relations are constructed, how much manual effort they require, and what must change when introducing a new object or task. The current experiments do not establish how well the approach transfers beyond these predefined structures.
  2. Please provide more details for the experiments: Although the appendix provides considerable architectural detail, it does not clearly specify how historical and future keyframes are selected, how the structural estimators are trained, or how many real-world demonstrations are collected per task.
  3. OSTRA receives task-specific object decompositions, relations, affordance information, and object-state supervision. The internal ablations demonstrate that these components help, but the comparison with external methods would be more informative if a strong baseline were given the same structured inputs and auxiliary targets.
  4. The reported future-state target consists of structural-node positions, while observed maps include orientations and richer relations. It would be useful to measure future-state prediction accuracy, particularly for rotational or articulated transitions, and relate prediction errors to downstream failures.
Questions for authors
  1. What manual annotations and task-specific engineering are required to construct a new OST-Map template, including affordances and structural relations? Can an existing policy handle unseen instances or compositions without new regressors or templates?

  2. How are historical and future keyframes selected? Is historical keyframe selection performed entirely online, using only information available at execution time?

  3. What data and labels train the structural estimator, and how many demonstrations are used for each real-world task? Are future-state targets obtained from perception or simulator state?

4 comments
AuthorsOpenReview ↗

Responses to Questions

Thank you for the thoughtful review. We clarify template construction, keyframe and target preparation, and the real-world demonstration count. We also report matched-input baseline comparisons and future-state prediction analyses.


(Q1) Templates, affordances, and transfer

Q1. What manual annotations and task-specific engineering are required to construct a new OST-Map template, including affordances and structural relations? Can an existing policy handle unseen instances or compositions without new regressors or templates?

  1. Manual annotation and task-specific engineering. For a new task whose object structures are covered by the library, no manual annotation or task-specific engineering is required to construct its OST-Map. The main manual cost lies in building primitive templates, but these templates are reusable across tasks. We define eight geometric primitives whose templates provide geometric samples, interaction-region affordance samples, and reference anchors. We compose them into reusable object templates specifying part decomposition, node order, geometric parameter slots, and within-object relations; composing an additional object template from existing primitives typically takes under 10 minutes. Given the first RGB frame and task description, GPT-5.6 selects relevant object templates and combines them into the task schema. Cross-object pairs receive Free relations, while instance masks, poses, and dimensions are estimated from observations. The revision details the primitive definitions (Appendix B.2) and construction algorithm (Appendix B.4).

  2. Unseen instances and compositions. For a scene with a different drawer instance or size, the existing template, estimator parameter slots, and trained policy can be reused when the drawer's part topology and the policy's node vocabulary remain unchanged; perception updates the instance dimensions and pose. A substantially different task, such as moving from drawer opening to pouring water, requires a policy trained for that task family.

    Within a covered task family, existing object templates can be selected and reused in new scenes, while perception updates instance dimensions and poses. Continuous geometric differences are represented by template parameters, and an imperfect local shape can be approximated with an existing primitive when the part topology remains covered. Unspecified cross-object relations receive the unconstrained Free type. A genuinely new topology requires an additional object template; if it adds nodes or parameter slots, the policy vocabulary or estimator heads must also be extended. Deformable objects fall outside the current fixed-primitive representation.


(Q2) Historical and future keyframes

Q2. How are historical and future keyframes selected? Is historical keyframe selection performed entirely online, using only information available at execution time?

Historical keyframe selection is performed entirely online, using only information available at execution time. We record a historical keyframe when the gripper state changes or the observed TCP velocity reaches zero, and include the episode's final frame at termination; in the RLBench2 bimanual case, either gripper change or a stop by both arms triggers a keyframe. For training at frame tt, we select the next HkH_k keyframes strictly after tt. If fewer than HkH_k remain, we repeat the last selected keyframe; if none remain, we repeat the current frame. Future object-node positions and TCP targets use the same indices; the action chunk begins at tt. Future observations are used only to build training targets. The revision describes this alignment (Appendix A.1).


(Q3) Structural estimator, future targets, and demonstrations

Q3. What data and labels train the structural estimator, and how many demonstrations are used for each real-world task? Are future-state targets obtained from perception or simulator state?

  1. Structural estimator training. The structural estimator learns from masked RGB-D point clouds in the same demonstrations, requires no additional estimator-specific training data or labels, and is trained end-to-end with the rest of OSTRA under the joint objective (Section 3.3), with implementation details in the appendix (Appendix A.1).
  2. Real-world demonstrations. We use 100 real-robot demonstrations to train each of the seven task policies, 700 demonstrations in total (Appendix A.5).
  3. Future-state targets. They come from perception: we apply the same perception and OST-Map construction pipeline to future demonstration frames, using FoundationPose for object poses and the structural estimator for primitive parameters, then obtain future node positions from the resulting maps; we do not read simulator state to construct these targets (Appendix A.1).
AuthorsOpenReview ↗

Response to Weakness 1, 2

(W1) Structural priors and transfer

W1. Appendix B.4 states that each task has a structural template specifying relevant objects, primitive decompositions, node ordering, and geometric parameters. Known relations can also be specified through templates, and structural estimation uses task-specific regressors. This is a substantial source of prior information. The paper should clarify how these templates, affordance regions, and relations are constructed, how much manual effort they require, and what must change when introducing a new object or task. The current experiments do not establish how well the approach transfers beyond these predefined structures.

Thank you for raising these concerns. We clarify two points:

  1. Template construction and prior information. The task-specific OST-Map template is not manually constructed for each task. The reusable primitive templates provide geometric samples, affordance samples, and reference anchors; object templates compose these primitives and specify within-object relations. Composing an additional object template from existing primitives typically takes under 10 minutes. GPT-5.6 selects the relevant object templates from the first RGB frame and task description, while perception estimates instance masks, poses, and geometric parameters. The revision details the primitive vocabulary (Appendix B.2) and template selection and construction (Appendix B.4).
  2. Transfer scope. A new task can reuse existing templates, estimator heads, and a policy for the same task family when its object structures and parameter slots are covered by the library and its node vocabulary remains unchanged. A genuinely new topology requires an additional object template; new parameter slots require a corresponding estimator output head, and new nodes require extending the policy's fixed vocabulary and training it for the enlarged structure (Appendix A.1).

(W2) Experimental details

W2. Please provide more details for the experiments: Although the appendix provides considerable architectural detail, it does not clearly specify how historical and future keyframes are selected, how the structural estimators are trained, or how many real-world demonstrations are collected per task.

We clarify the three requested experimental details:

  1. Keyframes. As detailed in Q2, historical keyframes are recorded online at gripper-state changes and motion stops; future targets use the next HkH_k keyframes after the current training frame, repeating the last selected keyframe if needed or the current frame if none remain (Appendix A.1).
  2. Structural estimator. It uses masked RGB-D object point clouds and is trained end-to-end with the policy under the joint objective (Section 3.3), without a separate estimator dataset or pretraining stage (Appendix A.1).
  3. Real-world demonstrations. We use 100 demonstrations for each of seven real-world tasks, 700 in total (Appendix A.5).
AuthorsOpenReview ↗

Response to Weakness 3

(W3) Baseline comparison

W3. OSTRA receives task-specific object decompositions, relations, affordance information, and object-state supervision. The internal ablations demonstrate that these components help, but the comparison with external methods would be more informative if a strong baseline were given the same structured inputs and auxiliary targets.

We clarify what the comparisons establish and report a matched-input experiment:

  1. Related structured baselines. Several baselines also use object-level representations: PPI predicts object pointflow and gripper keyposes, SPOT predicts object pose trajectories, and Im2Flow2Act uses object flow. GROOT uses object-centric 3D representations, while UWM jointly predicts future video and actions. These representations and targets differ from OST-Map, but provide relevant complete-system comparisons under common benchmark protocols.
  2. Matched-input comparison. We compare PPI and SAM2Act+ in their original form, with OSTRA's historical OST-Map inputs, and with both those inputs and the same future node-position targets. All variants use the same RLBench2 demonstrations and evaluation protocol. OST-Map input raises mean success from 80.8% to 82.4% for PPI and from 84.5% to 85.6% for SAM2Act+; adding future-node supervision raises them to 85.0% and 86.9%, respectively. OSTRA reaches 89.8%. Thus, the structured input and auxiliary target improve both external policies, while OSTRA retains the highest mean success in this comparison (Appendix A.6).

Matched structured-input baselines on RLBench2 (task success, %; higher is better).

MethodMeanBoxBallDrawerLaptopDustpanTrayHandover Easy
PPI80.896.789.379.746.398.792.062.7
PPI + OST-Map input82.497.090.381.750.398.993.065.6
PPI + OST-Map input + future-node target85.097.591.584.756.199.395.070.9
SAM2Act+84.595.288.081.571.897.089.568.5
SAM2Act+ + OST-Map input85.695.488.682.474.097.190.571.2
SAM2Act+ + OST-Map input + future-node target86.995.689.383.676.697.391.774.2
OSTRA89.898.393.087.082.797.792.077.7
AuthorsOpenReview ↗

Response to Weakness 4

(W4) Future-state prediction accuracy

W4. The reported future-state target consists of structural-node positions, while observed maps include orientations and richer relations. It would be useful to measure future-state prediction accuracy, particularly for rotational or articulated transitions, and relate prediction errors to downstream failures.

We clarify the target and report two additional RLBench2 experiments:

  1. Target scope. Observed OST-Maps contain node orientations and relations, but the reported model's future target directly supervises only node positions. Relative motion among multiple nodes can capture some rotational or articulated changes; the future orientation of a single-node object is not directly supervised (Section 3.3).
  2. Future-state accuracy and failures. We compare OSTRA's predicted future structural-node positions with positions derived from simulator states at the same four future keyframes, using 100 rollouts per task for each of three checkpoints across all seven RLBench2 tasks. Mean error is 2.1 cm overall, increasing from 1.2 cm at the first future keyframe to 3.0 cm at the fourth. It is 1.9 cm on successful rollouts versus 3.4 cm on failed rollouts. Drawer and Laptop, which involve articulated motion, have overall errors of 2.2 cm and 2.4 cm, respectively. This is an association between prediction error and outcome, not a causal attribution of failures (Appendix A.6).
  3. Future-orientation ablation. Under the same RLBench2 protocol, mean task success is 84.5% without future prediction, 89.8% with future node-position prediction, 85.6% with future node-rotation prediction, and 89.9% with both position and rotation. Adding rotation to position improves success by only 0.1 percentage point, indicating that future position prediction accounts for nearly all of the gain in this comparison (Appendix A.6).

Future-state position prediction error on RLBench2 (cm; lower is better).

Taskh=1h=1h=2h=2h=3h=3h=4h=4All horizonsSuccessful rolloutsFailed rollouts
Box1.11.51.92.41.71.72.8
Ball1.21.62.12.71.91.83.2
Drawer1.31.92.53.22.22.03.7
Laptop1.42.02.73.52.42.13.8
Dustpan1.01.41.82.31.61.62.9
Tray1.21.72.33.02.12.03.2
Handover Easy1.52.23.03.92.72.34.2
Mean1.21.82.33.02.11.93.4

Future-state orientation ablation on RLBench2 (task success, %; higher is better).

TaskNo future predictionPositionRotationPosition + rotation
Box95.098.396.098.3
Ball88.093.088.093.0
Drawer80.087.082.087.3
Laptop75.082.776.083.0
Dustpan95.097.796.097.7
Tray87.092.088.092.0
Handover Easy71.577.773.278.0
Mean84.589.885.689.9
Reviewer_cwHdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OSTRA, a framework for training diffusion policies with model object-state transition supervision to improve long-horizon manipulation. The key idea is to leverage an Object-State Transition Map, which represents task-relevant objects as structural graphs and uses historical object states to predict future object states. The predicted future states are then used to condition TCP keypose prediction and continuous action generation through a hierarchical denoising process. In their experiments, OSTRA outperforms several imitation learning baselines across three simulation benchmarks (RLBench2, ManiSkill, and LIBERO) and seven real-world manipulation tasks.

Critical strengths

The paper is overall well-written and addresses an important problem in long-horizon robotic manipulation. The idea of explicitly supervising object-state transitions instead of relying solely on action prediction is interesting and well-motivated. The experimental results are strong, and the method is compared to a decent number of baselines.

Critical weaknesses

W1. The method seems to rely heavily on task-specific templates to construct OST-Maps. As described in Appendix B.4, it assumes access to a structural template for each task, which is a strong assumption that limits the general applicability of the method. Since the baselines do not appear to have access to the same prior information, I am not sure whether the comparison is entirely fair. In particular, OSTRA is provided with oracle-like information, such as the number and identities of task-relevant objects, which could give it a significant advantage over the baselines.

W2. Some important implementation details regarding task conditioning are unclear. In particular, OSTRA's policy inputs appear to consist of RGB-D observations and robot states, but the paper evaluates on LIBERO, which is a language-conditioned multi-task benchmark. How are language instructions or task goals provided to OSTRA? Are separate policies trained for each task, or is a single policy trained across multiple tasks? It is also unclear how the task-specific structural templates are selected at inference time.

W3. The details of the real-world experimental setup are missing. In particular, how many demonstrations are used for training for each task?

Questions for authors

Q1. Are the SAM 3 prompts simply the names of task-relevant objects, or do they require additional designs? How are these prompts determined for each task?

Q2. Since OSTRA uses additional pretrained perception models and structured object information, I am curious about whether the baselines have access to comparable observations and prior information.

Q3. I am also curious about the OOD generalization capabilities of this framework. If OSTRA were trained on a diverse set of tasks with different structural templates, how well would it generalize to an entirely unseen task with a new structural template at test time? Would the model be able to handle previously unseen structural templates without additional training?

3 comments
AuthorsOpenReview ↗

Responses to Questions

Thank you for the constructive review. We clarify how OST-Map templates are built, how language conditions the LIBERO policy, and how the real-world evaluation is conducted. We also state the scope of the structural prior and generalization claims.


(Q1) SAM 3 prompts

Q1. Are the SAM 3 prompts simply the names of task-relevant objects, or do they require additional designs? How are these prompts determined for each task?

SAM 3 uses the name or short description of each task-relevant object as its text prompt. At the start of an episode, GPT-5.6 selects relevant object templates from the first RGB frame and task description and returns these names or descriptions. They are passed directly to SAM 3 for segmentation; no separate prompt is designed for each task. This connection is clarified in the implementation details (Appendix A.1) and the construction algorithm (Algorithm 2, Appendix B.4).


(Q2) Comparable observations and priors

Q2. Since OSTRA uses additional pretrained perception models and structured object information, I am curious about whether the baselines have access to comparable observations and prior information.

  1. Structural knowledge. Several baselines also use object-level representations that provide meaningful points of comparison: PPI predicts object pointflow and gripper keyposes, SPOT predicts object pose trajectories, and Im2Flow2Act uses object flow. Their representations differ from OST-Map, but they likewise exploit structured object information. OST-Map's explicit structural representation is itself a contribution and advantage of OSTRA. Its reusable object templates are selected from the first observation and task description, rather than supplying ground-truth task objects or states. We compare the complete methods and use OST-Map ablations to assess the contribution of this representation.
  2. Observations. OSTRA and the baselines use the same camera observations and robot proprioceptive information under the corresponding benchmark protocols. The comparison therefore holds the available sensory inputs and evaluation conditions fixed while allowing each method to use its own representation.
  3. Matched structured-input experiment. We also give PPI and SAM2Act+ the same historical OST-Map tokens used by OSTRA on RLBench2, and separately add the same future-node prediction target as auxiliary supervision. With OST-Map input and future-node supervision, PPI reaches 85.0% success and SAM2Act+ reaches 86.9%, compared with 89.8% for OSTRA under the same evaluation protocol. The controlled results, including each baseline's original and OST-Map-only variants, are reported in the revision (Appendix A.6).
MethodAvg. success (%) ↑BoxBallDrawerLaptopDustpanTrayHandover Easy
PPI80.896.789.379.746.398.792.062.7
PPI + OST-Map input82.497.090.381.750.398.993.065.6
PPI + OST-Map input + future-node target85.097.591.584.756.199.395.070.9
SAM2Act+84.595.288.081.571.897.089.568.5
SAM2Act+ + OST-Map input85.695.488.682.474.097.190.571.2
SAM2Act+ + OST-Map input + future-node target86.995.689.383.676.697.391.774.2
OSTRA89.898.393.087.082.797.792.077.7

(Q3) Unseen scenes

Q3. I am also curious about the OOD generalization capabilities of this framework. If OSTRA were trained on a diverse set of tasks with different structural templates, how well would it generalize to an entirely unseen task with a new structural template at test time? Would the model be able to handle previously unseen structural templates without additional training?

For a scene with a different drawer instance or size, the existing template, estimator parameter slots, and trained policy can be reused when the drawer's part topology and the policy's node vocabulary remain unchanged; perception updates the instance dimensions and pose. A substantially different task, such as moving from drawer opening to pouring water, requires a policy trained for that task family.

Within a covered task family, existing object templates can be selected and reused in new scenes, while perception updates instance dimensions and poses. Continuous geometric differences are represented by template parameters, and an imperfect local shape can be approximated with an existing primitive when the part topology remains covered. Unspecified cross-object relations receive the unconstrained Free type. A genuinely new topology requires an additional object template; if it adds nodes or parameter slots, the policy vocabulary or estimator heads must also be extended. Deformable objects fall outside the current fixed-primitive representation.

AuthorsOpenReview ↗

Response to Weakness 1

(W1) Structural priors and comparison

W1. The method seems to rely heavily on task-specific templates to construct OST-Maps. As described in Appendix B.4, it assumes access to a structural template for each task, which is a strong assumption that limits the general applicability of the method. Since the baselines do not appear to have access to the same prior information, I am not sure whether the comparison is entirely fair. In particular, OSTRA is provided with oracle-like information, such as the number and identities of task-relevant objects, which could give it a significant advantage over the baselines.

Thank you for raising these concerns. We clarify two points:

  1. General applicability. We understand that the original wording “each task defines a structural template” (Appendix B.4) could suggest that a template must be manually built for every task. In fact, the task-specific template used by OST-Map is not manually constructed for each task. We first define eight structural primitive types and compose them, together with within-object relations, into reusable object templates. Given the first RGB frame and task description, GPT-5.6 selects the relevant object templates and combines them into the task schema; cross-object pairs receive Free relations, while instance masks, poses, and geometric parameters are estimated from RGB-D observations. The revision details the primitive vocabulary (Appendix B.2) and this construction procedure (Appendix B.4). Because the primitives and object templates are reusable, new tasks can use existing structures without manually authoring a task-specific template. Together with automated template selection, this makes OSTRA applicable across tasks whose object structures are covered by the library; genuinely new topologies require an additional object template.

  2. Fair comparison. We argue that the comparison is fair for four reasons:

    1. Within each benchmark, OSTRA and the baselines follow the same evaluation protocol and use the same benchmark demonstration dataset.
    2. OSTRA's structural prior comes from the reusable primitive and object-template library described above. Crucially, OSTRA is given no privileged task-specific scene information in advance, including the number or identities of objects in a particular scene: it selects relevant templates from the first image and task description, and estimates instance states from observations. Obtaining structured object information is a key strength of OST-Map; the library is a design component of our method, not extra oracle information exposed to the policy.
    3. Our comparisons also include object-centric methods such as PPI, which provides its policy with predicted object pointflow and gripper keyposes. Comparing against such representations helps assess the advantage of OST-Map's explicit object structure.
    4. Although OSTRA uses pretrained perception models, several baselines also use resources beyond the benchmark demonstrations. For example, PPI uses pretrained Grounding DINO, SAM, and DINOv2 models, while MemoryVLA builds on a vision-language backbone further pretrained on the Open-X Embodiment robot dataset.
AuthorsOpenReview ↗

Response to Weakness 2, 3

(W2) LIBERO task conditioning

W2. Some important implementation details regarding task conditioning are unclear. In particular, OSTRA's policy inputs appear to consist of RGB-D observations and robot states, but the paper evaluates on LIBERO, which is a language-conditioned multi-task benchmark. How are language instructions or task goals provided to OSTRA? Are separate policies trained for each task, or is a single policy trained across multiple tasks? It is also unclear how the task-specific structural templates are selected at inference time.

We clarify two points:

  1. Language conditioning on LIBERO. We encode the task instruction with a frozen CLIP ViT-B/32 text encoder, project its embedding into a language token, and concatenate that token directly with the Semantic Field scene tokens before policy cross-attention. We use the same input during training and evaluation, as detailed in the revision (Appendix A.3). This simple adaptation lets OSTRA follow LIBERO's language-conditioned evaluation protocol for a fair comparison; we do not introduce a specialized language-processing architecture.
  2. Shared multi-task policy. Following MemoryVLA's setup, we train one shared policy for each of the Spatial, Object, and Goal suites, each containing 10 tasks. Long-10 and Long-90 jointly train one Long policy, and we report its Long-10 evaluation. We use the filtered LIBERO RLDS demonstrations released for MemoryVLA. Before training each policy, the union of objects selected across its training tasks defines a fixed vocabulary and node order. At inference, GPT-5.6 selects active templates from the first RGB frame and instruction; slots for absent objects are zero-filled. The revision specifies this procedure (Appendix A.3; Algorithm 2, Appendix B.4).

(W3) Real-world demonstrations

W3. The details of the real-world experimental setup are missing. In particular, how many demonstrations are used for training for each task?

We use 100 real-robot demonstrations to train each of the seven task policies, 700 demonstrations in total. The platform comprises a 7-DoF Realman 75-BF arm, a Pika parallel gripper, and an Intel RealSense D415 RGB-D camera. We evaluate each method for 20 trials per task and report full-task success rate. The revision states the demonstration count (Appendix A.5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.