acceptodds
Desk rejected at ICLR 2027

Solis: Streaming Unified World Models for Robotic Interaction

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Robotic interaction couples actions, changes in the environment, and progress toward a task. Modeling these modalities jointly offers a foundation for both generating actions toward desired outcomes and anticipating the consequences of actions. We present Solis, a streaming unified world model that learns a joint distribution over observations, actions, and progress-derived values using a single video diffusion transformer. Using different inputs as conditions, the same model can be used for simulation, planning, or evaluation. For practical deployment, closed-loop feedback is important for adapting to changes in the environment, yet restarting diffusion sampling after every observation makes frequent feedback costly. To address this, we employ temporal block-causal attention and natively pretrain our model with independently sampled chunk noise levels. At inference, we adopt a rolling diffusion schedule to maintain future video, action, and value chunks at progressively increasing noise levels: the leading chunk becomes executable while later chunks remain partially denoised. After execution, the horizon shifts with a new observation and a fresh noise chunk enters. This design raises feedback frequency without extra computational cost. With cross-embodiment pretraining and per-benchmark post-training, Solis achieves 98.4% average closed-loop success across four LIBERO suites and 88.8% and 89.1% across 12 RoboTwin 2.0 tasks in clean and randomized scenes. Furthermore, Solis shows strong future prediction and accurate value estimation for task evaluation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, World Models, Diffusion Models, Diffusion Forcing, Streaming Generation, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ivkrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Solis, a unified video diffusion transformer that jointly models future observations, actions, and progress-derived values. It combines temporal block-causal attention, diffusion forcing, and rolling sampling to retain partially denoised trajectories across control cycles, enabling frequent closed-loop feedback at reduced inference cost. It reports 98.4% average success across the four LIBERO suites and 88.8%/89.1% on a 12-task RoboTwin 2.0 subset (clean/randomized), and it is demonstrated on two real-robot manipulation tasks.

Critical strengths

The proposed model supports action generation, future prediction, and value estimation through different conditioning modes, providing a versatile framework for robotic tasks. The method achieves competitive LIBERO performance, reasonable RoboTwin results, with real-robot validation.

Critical weaknesses
  • The paper lacks ablation studies to isolate the contributions of video and value prediction to policy performance. All main closed-loop results use joint generation. World-model mode is not used for planning, and evaluator mode is not validated for action selection. Therefore, the practical advantages of jointly modeling all three modalities remain unclear.

  • The rolling sampler improves average LIBERO success by only 0.3 percentage points over the non-rolling baseline. More importantly, the comparison is mixed with different feedback frequencies (rolling replans every four actions, whereas non-rolling executes 16 actions open loop). Therefore, the gains cannot be distinctively attributed to retaining partially denoised trajectories.

Questions for authors
  • How does an action-only variant with the same backbone and rolling schedule perform? Can the authors demonstrate that joint video and value prediction improves closed-loop control, or that evaluator mode can meaningfully distinguish successful from failed trajectories?

  • Can the authors compare rolling and non-rolling sampling under matched feedback frequencies, and report the performance–computation trade-off? This would clarify whether the benefits arise from retaining partially denoised states rather than simply from observing more frequently.

  • Can the authors clarify how the 9.1 Hz feedback rate (in Table 2) is measured, including hardware and timing methodology?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ivkr (Part 1/2)

Thank you for your review and questions. We address the contributions of video and value prediction, the sampling comparison, and the feedback-rate measurement below.

W1 / Q1: Contributions of video and value prediction

Training ablations. We compare three fine-tuning variants: action prediction only, action and future-frame prediction, and the full model with value prediction. All variants use the same backbone, start from the same jointly pretrained model, and are evaluated on LIBERO with the K=4 rolling sampler. The prediction objectives are changed during fine-tuning. The table reports average success rates (%).

VariantFuture framesValueSuccess Rate (%)
Action onlyNoNo91.65
Action + future framesYesNo97.8
Full modelYesYes98.3

The action-and-frame model reaches 97.8%, compared with 91.65% for action only, a difference of 6.15 percentage points. The full model reaches 98.3%, a further 0.50 points above the variant without value prediction and 6.65 points above action only. The full model achieves the highest control success rate among the three variants.

Role of value. Value is an auxiliary progress signal that can indicate the progress of the current task. Enabling the model to be aware of this provides a global understanding of the task which can benefit the task execution. We have included a clearer statement in the revised paper.

The table below shows predictions from five failed RoboTwin rollouts. The values in the table are rescaled to 0–1, where 1 corresponds to the start of a demonstration and 0 to its end.

Task and settingObserved behaviorExecuted action stepsPredicted remaining value
Three-bowl stacking, randomized (Trajectory 1)Bowls remain separate after unsuccessful placement; then regrasp, lift, and stack two bowls160 → 248 → 3520.596 → 0.552 → 0.311; slow decrease (0.044 over 88 steps), then faster decrease (0.241 over 104 steps)
RGB block ordering, randomized (Trajectory 2)Return to the uncollected green block and retry the grasp; then lift, transport, and release it488 → 520 → 536; 568 → 600 → 624Rises during grasp retries (0.049 → 0.125 → 0.137), then falls during lifting, transport, and release (0.129 → 0.057 → 0.002), reflecting changes in local task progress
Bottle disposal, clean (Trajectory 3)Repeated unsuccessful grasping; the bottle remains on the table396–544 (149 steps)Remains on a plateau during repeated unsuccessful grasping, fluctuating within 0.418–0.490
Bottle disposal, randomized (Trajectory 4)Repeated manipulation attempts; a bottle remains on the table814 → 913 → 1088 → 1137 → 1232Shows two rise-and-fall cycles during manipulation retries (0.002 → 0.264 → 0.004 → 0.178 → 0.006), reflecting repeated revisions of the remaining-value estimate while the task remains unfinished
Bottle disposal, randomized (Trajectory 5)Repeated grasp attempts; the bottle remains on the table1136 → 1201 → 1296 → 1329 → 1408Rises and falls repeatedly during grasp retries (0.018 → 0.135 → 0.016 → 0.145 → 0.002), showing a response to repeated attempts

These examples show that predicted value can reflect local stagnation, repeated attempts, and renewed progress during execution. For a clearer visualization of this correspondence, please refer to the aligned video keyframes and value curves in Appendix D, Figs. 5–9 of the revised manuscript.

Action selection with world modeling and value guidance. We also use jointly predicted values to select action candidates on three RoboTwin tasks, using rolling sampling with the same model weights as in the 50-task RoboTwin evaluation reported in this rebuttal. Each candidate path contains jointly predicted future observations, actions, and values, so world modeling participates in candidate generation, while the predicted value decrease provides the selection criterion. For each block, we generate two candidates from different noise samples, maintaining 24=162^4=16 candidate paths across four blocks. At each query, we select the candidate for the leading block with the larger predicted value decrease and retain the eight paths sharing that block. As the horizon shifts, each retained path is extended with two candidates for the new trailing block, restoring the total to 16 paths. Candidate noise seeds are determined by the base seed (0), environment seed, block index, and candidate index.

We evaluate both methods using the same 100 environment seeds for each task in each of the clean and randomized settings. Each cell reports ordinary rolling / value-guided success rates.

TaskCleanRandomized
handover_block57.0% / 75.0%52.0% / 71.0%
pick_diverse_bottles79.0% / 89.0%80.0% / 86.0%
pick_dual_bottles82.0% / 100.0%87.0% / 96.0%
AuthorsOpenReview ↗

Response to Reviewer ivkr (Part 2/2)

W2 / Q2: Matched feedback and sampling cost

Matched-feedback comparison. Following your suggestion, we compare rolling and non-rolling sampling using the same checkpoint and joint-generation mode within each benchmark, with matched feedback intervals, open-loop execution lengths, and forward passes per query. Both obtain a new observation after every four executed actions on LIBERO or 16 on RoboTwin and use one network evaluation per query after initialization.

Rolling retains partially denoised future chunks across queries and refines them under new observations. Non-rolling generates the trajectory from fresh noise at each query and executes only its first action chunk.

LIBERO. Results average four suites with 500 episodes each.

SamplerTrajectory reuseFeedback intervalOpen-loop lengthForward passes per querySuccess (%)
RollingYes44198.3
Non-rollingNo44197.20

RoboTwin. Results cover all 50 tasks, with 100 episodes per task in each setting.

SamplerTrajectory reuseFeedback intervalOpen-loop lengthForward passes per queryClean (%)Randomized (%)
RollingYes1616188.9487.60
Non-rollingNo1616186.9885.40

With feedback intervals, open-loop execution lengths, and forward passes per query matched, rolling exceeds non-rolling by 1.10 percentage points on LIBERO, 1.96 points on RoboTwin clean, and 2.20 points on RoboTwin randomized.

Q3: Measurement of the 9.1 Hz feedback rate

The 9.1 Hz feedback rate reported in Table 2 was measured on an NVIDIA RTX PRO 6000 GPU. The measured inference time includes VAE encoding of visual observations and denoising for action generation, corresponding to approximately 110 ms per policy update. We measure the average inference latency over repeated policy updates and report its reciprocal as the feedback rate.

AuthorsOpenReview ↗

Invitation to Discuss Any Remaining Concerns

Dear Reviewer ivkr,

Thank you for the time and care you have devoted to reviewing our work. As the discussion period draws to a close, we would be grateful if you could take another look at our responses and revised manuscript when your schedule permits. Your feedback has helped us improve the paper, and we would welcome any further questions or concerns.

We appreciate your time and consideration.

Best regards, The Authors

Reviewer_BxDqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Solis uses a single video diffusion transformer to jointly model future observations, actions, and demonstration-time-derived progress labels. Different conditioning modes support policy generation, world-model prediction, and trajectory evaluation. Block-causal attention and independently sampled chunk noise enable rolling inference that retains partially denoised trajectories and updates them as new observations arrive. Experiments cover LIBERO, RoboTwin 2.0, two real-robot tasks, and video and progress prediction.

Critical strengths
  1. Unified modeling is integrated into a concrete deployment procedure. A shared backbone supports action generation, future prediction, and progress estimation through conditioning changes, reducing interfaces between separate models.

  2. Rolling sampling has a clear computational motivation. Retaining and refining partially denoised joint trajectories distributes sampling across control cycles, providing a concrete mechanism for more frequent feedback.

  3. Experiments cover multiple control settings. LIBERO performance is strong and largely maintained with a reduced sampling budget. RoboTwin and real-robot experiments further demonstrate applicability across tasks and embodiments.

Critical weaknesses
  1. The streaming comparison changes multiple factors simultaneously. In Table 4, the rolling sampler refreshes observations every four actions, whereas the non-rolling sampler does so every sixteen actions. Trajectory reuse, feedback frequency, and open-loop execution length therefore change together. The average success difference is only 0.3 percentage points, making attribution to trajectory retention difficult.

  2. Progress fitting does not establish task-evaluation capability. Labels are normalized remaining time and assume steady demonstration progress, while quantitative evaluation covers successful trajectories only. Low prediction error does not establish recognition of failure, stagnation, or regression.

  3. Training ablations do not isolate the control benefits of joint modalities. The analyses primarily vary sampling or conditioning while fixing checkpoints. Without models trained without video or value prediction, it remains unclear whether strong control performance depends on joint world modeling or mainly on action learning and large-scale pretraining.

  4. Comparison protocols and evaluation scale limit the conclusions. LIBERO baselines come from different papers, and some strong RoboTwin baselines use more tasks and training data. Real-robot evaluation covers only two tasks with 30 trials each and lacks training-randomness analysis. These results more clearly establish feasibility than a stable performance advantage.

Questions for authors

1.Does rolling trajectory reuse remain beneficial against restart sampling when observations are refreshed every four actions and actual computational budgets are matched?

2.On failed, stalled, or regressing trajectories, do predicted values reflect task state rather than elapsed time?

3.How does control success change when video-prediction and value-prediction objectives are removed separately?

4.Are matched-protocol baselines or repeated experiments available to establish whether the differences in Table 4 are stable?

4 comments
AuthorsOpenReview ↗

Response to Reviewer BxDq (Part 1/3)

Thank you for your review and questions. We address the sampling comparison, value evaluation, training ablations, and evaluation protocols below.

W1 / Q1 / Q4: Matched feedback and sampling cost

Matched-feedback comparison. Following your suggestion, we compare rolling and non-rolling sampling with matched feedback intervals, open-loop execution lengths, and forward passes per query. Both obtain a new observation after every four executed actions on LIBERO or 16 on RoboTwin and use one network evaluation per query after initialization.

Rolling retains partially denoised future chunks across queries and refines them under new observations. Non-rolling generates the trajectory from fresh noise at each query and executes only its first action chunk.

LIBERO. Results average four suites with 500 episodes each.

SamplerTrajectory reuseFeedback intervalOpen-loop lengthForward passes per querySuccess (%)
RollingYes44198.3
Non-rollingNo44197.20

RoboTwin. Results cover all 50 tasks, with 100 episodes per task in each setting.

SamplerTrajectory reuseFeedback intervalOpen-loop lengthForward passes per queryClean (%)Randomized (%)
RollingYes1616188.9487.60
Non-rollingNo1616186.9885.40

With feedback intervals, open-loop execution lengths, and forward passes per query matched, rolling exceeds non-rolling by 1.10 percentage points on LIBERO, 1.96 points on RoboTwin clean, and 2.20 points on RoboTwin randomized.

AuthorsOpenReview ↗

Response to Reviewer BxDq (Part 2/3)

W2 / Q2: Value prediction on failures, stalls, and regressions

Thanks for the suggestion. We have analyzed if the value can indicate failure or stagnation. The table below pairs value predictions with observed behavior in five failed RoboTwin rollouts, covering stalled execution, repeated attempts, and renewed progress. The values in the table are rescaled to 0–1, where 1 corresponds to the start of a demonstration and 0 to its end. For a clearer visualization of this correspondence, please refer to the aligned video keyframes and value curves in Appendix D, Figs. 5–9 of the revised manuscript.

Task and settingObserved behaviorExecuted action stepsPredicted remaining value
Three-bowl stacking, randomized (Trajectory 1)Bowls remain separate after unsuccessful placement; then regrasp, lift, and stack two bowls160 → 248 → 3520.596 → 0.552 → 0.311; slow decrease (0.044 over 88 steps), then faster decrease (0.241 over 104 steps)
RGB block ordering, randomized (Trajectory 2)Return to the uncollected green block and retry the grasp; then lift, transport, and release it488 → 520 → 536; 568 → 600 → 624Rises during grasp retries (0.049 → 0.125 → 0.137), then falls during lifting, transport, and release (0.129 → 0.057 → 0.002), reflecting changes in local task progress
Bottle disposal, clean (Trajectory 3)Repeated unsuccessful grasping; the bottle remains on the table396–544 (149 steps)Remains on a plateau during repeated unsuccessful grasping, fluctuating within 0.418–0.490
Bottle disposal, randomized (Trajectory 4)Repeated manipulation attempts; a bottle remains on the table814 → 913 → 1088 → 1137 → 1232Shows two rise-and-fall cycles during manipulation retries (0.002 → 0.264 → 0.004 → 0.178 → 0.006), reflecting repeated revisions of the remaining-value estimate while the task remains unfinished
Bottle disposal, randomized (Trajectory 5)Repeated grasp attempts; the bottle remains on the table1136 → 1201 → 1296 → 1329 → 1408Rises and falls repeatedly during grasp retries (0.018 → 0.135 → 0.016 → 0.145 → 0.002), showing a response to repeated attempts

These examples show plateaus or slow decreases during stalls, rebounds during retries, and decreases with renewed progress. This correspondence with observed behavior supports value as a signal of execution state, rather than a strictly monotonic countdown.

Action selection with world modeling and value guidance. We also use jointly predicted values to select action candidates on three RoboTwin tasks, using rolling sampling with the same model weights as in the 50-task RoboTwin evaluation reported in this rebuttal. For each block, we generate two candidates from different noise samples, maintaining 24=162^4=16 candidate paths across four blocks. At each query, we select the candidate for the leading block with the larger predicted value decrease and retain the eight paths sharing that block. As the horizon shifts, each retained path is extended with two candidates for the new trailing block, restoring the total to 16 paths. Candidate noise seeds are determined by the base seed (0), environment seed, block index, and candidate index.

We evaluate both methods using the same 100 environment seeds for each task in each of the clean and randomized settings. Each cell reports ordinary rolling / value-guided success rates.

TaskCleanRandomized
handover_block57.0% / 75.0%52.0% / 71.0%
pick_diverse_bottles79.0% / 89.0%80.0% / 86.0%
pick_dual_bottles82.0% / 100.0%87.0% / 96.0%
AuthorsOpenReview ↗

Response to Reviewer BxDq (Part 3/3)

W3 / Q3: Training ablations

Training ablation. We compare three training variants: action prediction only, action and future-frame prediction, and the full model with value prediction. We evaluate the resulting models on the LIBERO benchmark.

VariantFuture framesValueSuccess Rate (%)
Action onlyNoNo91.65
Action + future framesYesNo97.8
Full modelYesYes98.3

The action-and-frame model reaches 97.8%, compared with 91.65% for action only, a difference of 6.15 percentage points. The full model reaches 98.3%, a further 0.50 points above the variant without value prediction and 6.65 points above action only. The full model achieves the highest control success rate among the three variants.

W4 / Q4: Evaluation scale and comparison protocols

Thanks for the suggestion.

LIBERO. LIBERO provides standardized task suites and evaluation environments, supporting comparison with results reported in prior papers. Using published baseline results is common practice, as in LingBot-VA and Fast-WAM. We follow this convention and strictly adhere to the standard LIBERO evaluation protocol, evaluating 50 episodes per task, 500 per suite, and 2,000 overall. Table 1 identifies the source paper for each baseline result.

Across three evaluation seeds, with the same model checkpoint and sampling settings, Solis achieves average success rates of 98.30%, 98.35%, and 98.50%, yielding 98.38 ± 0.10% (mean ± sample standard deviation). These results show consistent performance across evaluation seeds.

RoboTwin. Following your suggestion, we extended evaluation from the 12-task subset to the full 50-task RoboTwin benchmark. The results and conclusion are consistent with results on the 12-task subset reported in the original submission. We evaluate a single model trained jointly on all 50 tasks, with 100 episodes per task in each setting. Solis obtains 88.94% on clean and 87.60% on randomized tasks. The table shows average success rates over all 50 tasks:

MethodClean (%)Randomized (%)
Solis88.9487.60

Given the time and compute constraints of the rebuttal period, we have not yet completed more extensive training. We will continue training the current 50-task RoboTwin model and report updated evaluation results.

Real-robot evaluation. Following your suggestion, we expanded the real-robot evaluation from two to more tasks including: block stacking, number sorting, arithmetic and logic, chessboard setup, and letter formation. The expanded evaluation also includes π0.5\pi_{0.5} as a baseline. Both methods use 30 trials per task.

MethodBlock stackingNumber sortingArithmetic and logicChessboard setupLetter formationAvg. (%)
π0.5\pi_{0.5}22/30 (73.3%)18/30 (60.0%)13/30 (43.3%)21/30 (70.0%)18/30 (60.0%)61.3
Solis24/30 (80.0%)19/30 (63.3%)22/30 (73.3%)28/30 (93.3%)18/30 (60.0%)74.0
AuthorsOpenReview ↗

Invitation to Discuss Any Remaining Concerns

Dear Reviewer BxDq,

Thank you for the time and care you have devoted to reviewing our work. As the discussion period draws to a close, we would be grateful if you could take another look at our responses and revised manuscript when your schedule permits. Your feedback has helped us improve the paper, and we would welcome any further questions or concerns.

We appreciate your time and consideration.

Best regards, The Authors

Reviewer_RMZvOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Solis, a streaming unified world model that represents future video latents, actions, and demonstration-progress values as frame-shaped slots and models their joint distribution with a single video diffusion transformer. Temporal block-causal attention couples modalities within a chunk while preventing later chunks from affecting earlier ones. Trained with diffusion forcing and sampled with a rolling schedule, Solis keeps partially denoised chunks across control cycles and refreshes them with new observations.

Critical strengths

This paper presents Solis, a streaming unified world model that represents future video latents, actions, and demonstration-progress values as frame-shaped slots and models their joint distribution with a single video diffusion transformer. Temporal block-causal attention couples modalities within a chunk while preventing later chunks from affecting earlier ones. Trained with diffusion forcing and sampled with a rolling schedule, Solis keeps partially denoised chunks across control cycles and refreshes them with new observations.

Critical weaknesses

1.The main text would benefit from a clearer overview paragraph before the method section. In particular, the transition from the three conditional modes in Section 3.1 to the slot construction and block-causal attention in Section 3.2 is abrupt, so the motivation for the temporal mask and per-chunk noise design becomes clear only later.

2.The paper should also disclose more complete training details, such as GPU-hours, to support reproducibility and cost assessment.

3.A removal/ablation study, or a clearer statement repositioning value as an auxiliary progress signal rather than a decision variable, would make the contribution boundary sharper.

Questions for authors

1.Could the authors add a short overview paragraph at the beginning of Section 3 that maps the three conditional modes to the architectural choices in Sections 3.2 and 3.3?

  1. Could you report training duration or equivalent compute statistics, in addition to batch sizes and step counts?
3 comments
AuthorsOpenReview ↗

Response to Reviewer RMZv (Part 1/2)

Thank you for your time and review. We address the questions about method overview, training cost, and role of value below.

W1 / Q1: Connecting the conditional modes to the architecture and sampling design

Thanks for the suggestion. We agree that the transition from the conditional formulation to the architectural details needs a clearer explanation. We have added a brief overview at the beginning of Section 3 and strengthened the transition between Sections 3.1 and 3.2:

In this section, we present Solis, a unified world model that generates future observations, actions, and value trajectories from the observed context. Section 3.1 formulates the unified world modeling and its support for three conditional modes: world model, policy, and evaluator. Section 3.2 further introduces the instantiation of the unified formulation by introducing a shared multimodal trajectory representation and the temporal block-causal attention to support streaming refinement. Section 3.3 details the training and sampling strategy.

We have also added a transition sentence at the end of Section 3.1:

To instantiate the unified world model formulated in Section 3.1, we organize each future chunk into aligned visual, action, and value slots and employ a temporal block-causal attention design to support closed-loop streaming refinement.

W2 / Q2: Training cost

Thank you for the suggestion. We provide additional training details below, including the optimizer steps, training time, and computational cost. All stages use 8 × A800 80GB GPUs.

StageOptimizer stepsApprox. training timeApprox. GPU-hours
Pretraining500,00014.4 days2,756
LIBERO fine-tuning18,0009.3 days1,782
RoboTwin fine-tuning (50 tasks)32,00013.2 days2,534

The corresponding input and optimization settings are:

StageResolution (H × W)Prediction horizon NGlobal batch sizePeak learning rate
Pretraining224 × 22416645e-5
LIBERO fine-tuning224 × 224161,0245e-5
RoboTwin fine-tuning (50 tasks)192 × 256643845e-5

We optimize all DiT parameters with AdamW (β₁=0.9, β₂=0.95, weight decay=0.01), using BF16 and activation checkpointing, while keeping the VAE frozen. Global batch size denotes the number of samples per optimizer update, accumulated over eight GPUs and two gradient-accumulation steps. The per-GPU microbatch sizes are 4 for pretraining, 64 for LIBERO, and 24 for RoboTwin. The learning rate uses 1,000 steps of linear warmup followed by linear decay. The minimum learning-rate multiplier is 0.1 for all three stages. The observation context contains four frames: three history frames and the current frame. Each prediction horizon is divided into four chunks, with four future steps per chunk for pretraining and LIBERO (N=16), and 16 future steps per chunk for RoboTwin (N=64).

AuthorsOpenReview ↗

Response to Reviewer RMZv (Part 2/2)

W3: Contribution of value prediction

Training ablation. We compare three training variants: action prediction only, action and future-frame prediction without value prediction, and the full model with value prediction. We evaluate the resulting models on the LIBERO benchmark.

VariantFuture framesValueSuccess Rate (%)
Action onlyNoNo91.65
Action + future framesYesNo97.8
Full modelYesYes98.3

The action-and-frame model reaches 97.8%, compared with 91.65% for action only, a difference of 6.15 percentage points. The full model reaches 98.3%, a further 0.50 points above the variant without value prediction and 6.65 points above action only. The full model achieves the highest control success rate among the three variants.

Role of value. We agree that value is an auxiliary progress signal that can indicate the progress of the current task. Enabling the model to be aware of this provides a global understanding of the task which can benefit the task execution. We have included a clearer statement in the revised paper.

The table below shows predictions from five failed RoboTwin rollouts. The values in the table are rescaled to 0–1, where 1 corresponds to the start of a demonstration and 0 to its end. For a clearer visualization of this correspondence, please refer to the aligned video keyframes and value curves in Appendix D, Figs. 5–9 of the revised manuscript.

Task and settingObserved behaviorExecuted action stepsPredicted remaining value
Three-bowl stacking, randomized (Trajectory 1)Bowls remain separate after unsuccessful placement; then regrasp, lift, and stack two bowls160 → 248 → 3520.596 → 0.552 → 0.311; slow decrease (0.044 over 88 steps), then faster decrease (0.241 over 104 steps)
RGB block ordering, randomized (Trajectory 2)Return to the uncollected green block and retry the grasp; then lift, transport, and release it488 → 520 → 536; 568 → 600 → 624Rises during grasp retries (0.049 → 0.125 → 0.137), then falls during lifting, transport, and release (0.129 → 0.057 → 0.002), reflecting changes in local task progress
Bottle disposal, clean (Trajectory 3)Repeated unsuccessful grasping; the bottle remains on the table396–544 (149 steps)Remains on a plateau during repeated unsuccessful grasping, fluctuating within 0.418–0.490
Bottle disposal, randomized (Trajectory 4)Repeated manipulation attempts; a bottle remains on the table814 → 913 → 1088 → 1137 → 1232Shows two rise-and-fall cycles during manipulation retries (0.002 → 0.264 → 0.004 → 0.178 → 0.006), reflecting repeated revisions of the remaining-value estimate while the task remains unfinished
Bottle disposal, randomized (Trajectory 5)Repeated grasp attempts; the bottle remains on the table1136 → 1201 → 1296 → 1329 → 1408Rises and falls repeatedly during grasp retries (0.018 → 0.135 → 0.016 → 0.145 → 0.002), showing a response to repeated attempts

These examples show that predicted value can reflect local stagnation, repeated attempts, and renewed progress during execution.

Action selection with world modeling and value guidance. We also use jointly predicted values to select action candidates on three RoboTwin tasks, using rolling sampling with the same model weights as in the 50-task RoboTwin evaluation reported in this rebuttal. For each block, we generate two candidates from different noise samples, maintaining 24=162^4=16 candidate paths across four blocks. At each query, we select the candidate for the leading block with the larger predicted value decrease and retain the eight paths sharing that block. As the horizon shifts, each retained path is extended with two candidates for the new trailing block, restoring the total to 16 paths. Candidate noise seeds are determined by the base seed (0), environment seed, block index, and candidate index.

We evaluate both methods using the same 100 environment seeds for each task in each of the clean and randomized settings. Each cell reports ordinary rolling / value-guided success rates.

TaskCleanRandomized
handover_block57.0% / 75.0%52.0% / 71.0%
pick_diverse_bottles79.0% / 89.0%80.0% / 86.0%
pick_dual_bottles82.0% / 100.0%87.0% / 96.0%
AuthorsOpenReview ↗

Invitation to Discuss Any Remaining Concerns

Dear Reviewer RMZv,

Thank you for the time and care you have devoted to reviewing our work. As the discussion period draws to a close, we would be grateful if you could take another look at our responses and revised manuscript when your schedule permits. Your feedback has helped us improve the paper, and we would welcome any further questions or concerns.

We appreciate your time and consideration.

Best regards, The Authors

Comments on the paper

2 comments
Area_Chair_jDtHOpenReview ↗

Discussion

Dear reviewers,

We have mixed signals from the review ratings.

  • RMZv: 3: Weak acceptance
  • ivkr: 2: Weak rejection
  • BxDq: 2: Weak rejection

Could you take a look at each other's reviews and the author responses and initiate a discussion to reach consensus?
Were your concerns resolved and questions answered?
How do you weigh the strengths and weaknesses?

Please note that ICLR 2027 has Policy on LLM use.

Best, Your AC

AuthorsOpenReview ↗

Kind Request to Review Our Responses and Continue the Discussion

Dear Reviewers,

Thank you for your time and thoughtful feedback. We would appreciate it if you could take a look at our responses and the updates to the manuscript when you have a chance. We would be very happy to discuss any remaining concerns and provide further clarification. Thank you again for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.