- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents OSTRA for multi-stage robot manipulation. The main idea is to keep track of how task-relevant objects have changed and use predictions of their future states to guide robot actions. Objects are represented using identity-consistent structural graphs, and the policy predicts future object positions, end-effector subgoals, and continuous actions in sequence. Experiments on three simulation benchmarks and seven real-world tasks report improvements over several imitation learning baselines.
Critical strengths
-
The paper addresses a meaningful problem: a robot performing a multi-stage task needs to understand what has already happened to the objects, not just their current appearance. Using identity-consistent object histories and future targets is a sensible way to make this information easier to learn.
-
The evaluation covers several simulation benchmarks and seven real-world tasks. The architectural ablations and perception-error analysis provide useful evidence that the proposed representation and prediction hierarchy contribute to the reported performance.
Critical weaknesses
-
The training-data pipeline is insufficiently specified for reproducibility. Although the architecture and losses are described, the paper does not clearly explain how the structural estimator is trained, how historical and future keyframes are selected, or how future object-position and TCP targets are obtained and aligned. These are central components of the method, so the omissions make it difficult to reproduce the results and assess the additional supervision required.
-
The dependence on task-specific preparation is not sufficiently evaluated. Appendix B.4 specifies task-dependent structural templates, but the effort needed to construct them and their reuse across objects or tasks remain unclear. The reported evaluations do not isolate transfer to unseen objects with an unchanged template. Consequently, it is difficult to assess how much of the practical benefit depends on task-specific knowledge and preparation.
Questions for authors
-
Data preparation: Which templates, affordance annotations, geometric labels, and keyframes are manually specified or additionally supervised? How is the structural estimator trained, and how are future object and TCP targets obtained and aligned? Please clarify the preparation cost for a new task and whether baselines receive comparable information.
-
Generalization: Can the method handle unseen object instances or sizes without changing templates or retraining specialized regressors? Such an evaluation would help establish its transferability. Please also clarify whether LIBERO uses per-task or shared policies, and how language instructions or task identities enter the model.
-
Future-state targets: Since future supervision contains node positions but not orientations, how are rotations handled for single-node objects? A position-only versus position-and-orientation comparison on an alignment-sensitive task would help assess this limitation.
-
Method clarity: Could the authors provide stage-wise input/output equations with tensor dimensions and a training algorithm? Please explicitly distinguish direct object regression from TCP/action denoising, identify frozen/trainable modules, and specify action representations, noise schedules, and loss normalization.
-
Deployment: Does the reported policy latency include the full reverse-diffusion process, and with how many steps? How are perception, replanning, and action execution scheduled? Please report full-system peak GPU memory and latency variability; a success–latency comparison at different update frequencies would help assess practical responsiveness.
6 comments
Responses to Question 2, 3
(Q2) Unseen instances and LIBERO conditioning
Q2. Can the method handle unseen object instances or sizes without changing templates or retraining specialized regressors? Such an evaluation would help establish its transferability. Please also clarify whether LIBERO uses per-task or shared policies, and how language instructions or task identities enter the model.
- Transfer scope. For a new instance, size, or arrangement within a covered task family, the existing template, estimator parameter slots, and trained policy can be reused when the part topology and policy node vocabulary remain unchanged; perception updates instance dimensions and poses. A genuinely new part topology requires an additional object template. If it adds parameter slots or nodes, the estimator heads or policy vocabulary must be extended accordingly.
- LIBERO policies and language.
- Language conditioning on LIBERO. We encode the task instruction with a frozen CLIP ViT-B/32 text encoder, project its embedding into a language token, and concatenate that token directly with the Semantic Field scene tokens before policy cross-attention. We use the same input during training and evaluation (Appendix A.3). This simple adaptation lets OSTRA follow LIBERO's language-conditioned evaluation protocol for a fair comparison; we do not introduce a specialized language-processing architecture.
- Shared multi-task policy. Following MemoryVLA's setup, we train one shared policy for each of the Spatial, Object, and Goal suites, each containing 10 tasks. Long-10 and Long-90 jointly train one Long policy, and we report its Long-10 evaluation. We use the filtered LIBERO RLDS demonstrations released for MemoryVLA. Before training each policy, the union of objects selected across its training tasks defines a fixed vocabulary and node order. At inference, GPT-5.6 selects active templates from the first RGB frame and instruction; slots for absent objects are zero-filled. This procedure is specified in the revised paper (Appendix A.3; Algorithm 2 in Appendix B.4).
(Q3) Future-state targets and rotation
Q3. Since future supervision contains node positions but not orientations, how are rotations handled for single-node objects? A position-only versus position-and-orientation comparison on an alignment-sensitive task would help assess this limitation.
- Why future supervision focuses on position. The future-state target is designed to give the policy a simple, memory-like account of long-horizon object-state transitions. It therefore supervises changes in node positions rather than future node rotations. Rotation changes needed for precise alignment are more closely tied to short-horizon action generation, which the TCP and action stages handle using current observations and historical OST-Maps that include observed orientations. A single-node object's future orientation is not directly predicted, but the policy can still generate rotational actions.
- Effect of adding rotation supervision. We compare four variants on all seven RLBench2 tasks: no future prediction, future position, future rotation, and future position plus rotation. They use the same demonstrations, OST-Map templates, perception pipeline, training budget, and evaluation seeds; the prediction variants use future keyframes. We report mean success over three checkpoints with 100 rollouts per task. Adding rotation to position improves mean success by just 0.1 percentage point (89.8% to 89.9%), with a gain of at most 0.3 points on any task. The negligible gain across the evaluated tasks supports our choice to use position for future-state supervision and leave precise rotational control to the policy. Full experimental details are provided in the revision (Appendix A.6).
| RLBench2 task | No future prediction (%) | Position (%) | Rotation (%) | Position + rotation (%) |
|---|---|---|---|---|
| Box | 95.0 | 98.3 | 96.0 | 98.3 |
| Ball | 88.0 | 93.0 | 88.0 | 93.0 |
| Drawer | 80.0 | 87.0 | 82.0 | 87.3 |
| Laptop | 75.0 | 82.7 | 76.0 | 83.0 |
| Dustpan | 95.0 | 97.7 | 96.0 | 97.7 |
| Tray | 87.0 | 92.0 | 88.0 | 92.0 |
| Handover Easy | 71.5 | 77.7 | 73.2 | 78.0 |
| Overall | 84.5 | 89.8 | 85.6 | 89.9 |
Responses to Question 4
(Q4) Prediction stages and training details
Q4. Could the authors provide stage-wise input/output equations with tensor dimensions and a training algorithm? Please explicitly distinguish direct object regression from TCP/action denoising, identify frozen/trainable modules, and specify action representations, noise schedules, and loss normalization.
The revised paper lists the stage-wise inputs, outputs, supervised targets, and tensor dimensions in the new tensor-layout table (Table 7 in Appendix A.1). The same appendix gives the joint minibatch training procedure; we summarize the key distinctions below.
- Stage targets and action representation. The object stage directly regresses normalized future node positions ; its target is not diffused. The TCP stage predicts noise . For 7D actions, the action stage predicts pose noise for and directly predicts gripper openness with a separate head. For 4D actions , only XYZ is diffused; the fourth output directly supervises gripper openness. A 4D TCP target differs: all four channels are diffused. The reference horizons are and .
- Noise schedules. Each training item independently samples a timestep from and Gaussian noise for TCP and action targets. Both schedulers use epsilon prediction without sample clipping: position uses a scaled-linear beta schedule, and quaternion channels use a squared-cosine schedule. Inference uses eight DDIM reverse steps; each step reruns the object, TCP, and action stages, while the object prediction itself receives no scheduler update.
- Loss normalization and training algorithm. The object, TCP-noise, action-noise, and gripper losses are separate mean terms over their valid elements; a node mask, when present, restricts the object loss to valid XYZ scalars. Their reference weights have equal ratios relative to the action loss. For each minibatch, we normalize targets, sample timesteps and noise, build noisy TCP/action inputs, execute object TCP action, compute the four losses, and jointly update the trainable parameters. Stop-gradient is applied to the latent plans passed between stages, while shared context encoders still receive gradients.
- Frozen and trainable modules. The pretrained visual backbone and LIBERO CLIP ViT-B/32 text encoder are frozen; SAM 3 and FoundationPose supply observation-derived inputs outside the gradient graph. The structural estimator, Semantic Field and OST-Map encoders, language-token projection, prediction backbone, and output heads are trainable. AdamW updates the trainable parameters, and an EMA copy is updated after optimizer steps. Staged inference is specified separately (Algorithm 1).
Responses to Question 1
(Q1) Data preparation and target alignment
We address the five parts of the question in order: