acceptodds
Under review as a conference paper at ICLR 2027

Structure Beyond Pixels: Aligning Video with Structural Preferences for Robust Policy Generation

Abstract

Video generation models can serve as world models for robotic policy generation. However, because they are trained to produce visually plausible videos without physical constraints, the generated rollouts often exhibit an executability gap, leading to downstream task failures. To address this, we propose Structure-Aligned Direct Preference Optimization (Struct-DPO), which aligns a pre-trained video generator with structural preferences derived from environment interaction. Specifically, we treat physically grounded simulator observations as winning samples and model rollouts as losing samples, automatically identifying the most severe violations via structural metrics for viewpoint and geometric consistency. Notably, this approach requires neither architectural changes nor explicit 3D supervision; in our experiments, it trains only 3.95% of the base model parameters at 0.27% of the additional step of standard supervised fine-tuning, yielding high parameter and computational efficiency. Experiments on the RoboCasa benchmark across 24 manipulation tasks show consistent gains in task success, and alignment learned from a subset of tasks transfers to unseen tasks, indicating that the distilled physical priors generalize across tasks and environments.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Video Policy, Structural Preference Optimization, Manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WzitOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Struct-DPO to align video-based robot policies using structural preferences between generated videos and their corresponding environment executions. It combines depth reprojection, edge alignment, and spatial displacement for preference selection, with hindsight relabeling for failed episodes. Experiments show improvements on RoboCasa and a physical robot, but stronger controls and more consistent reporting are needed.

Critical strengths

1,Environment-executed rollouts provide a practical source of preference supervision without manual preference annotation. 2,The method evaluates downstream task success, including encouraging real-robot improvements from 52% to 82% in the single-task setting. 3,Component ablations, phase-wise execution analysis, and failure cases provide useful insights.

Critical weaknesses

1,The structural metrics do not isolate physical violations. Differences between generated and executed videos can also arise from action-decoder errors, timing offsets, or appearance changes. In particular, same-timestep RAFT displacement measures spatial disagreement rather than temporal dynamics. Independent contact or object-motion evaluation is needed to support the interpretation of these errors as physical inconsistency.

2,The paper could broaden its discussion of structural grounding to include OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generation, ORV: 4D Occupancy-centric Robot Video Generation, and OmniNWM: Omniscient Driving Navigation World Models. These provide relevant perspectives on occupancy-based generation, downstream policy benefits, and geometry-grounded rewards, respectively. Discussing their supervision requirements and the distinction between explicit structural representations and preference-based alignment would sharpen the paper’s positioning.

3, Since training includes an SFT term on environment observations, the gains may partly come from additional supervised adaptation. Please compare against LoRA-SFT on the same observations and DPO with randomly selected pairs, using matched data sizes and optimization budgets. Metric-removal ablations should also control the number of selected pairs.

4,The main simulation improvement is 52.46% to 55.65%, corresponding to 22 additional successes over 690 trials; five tasks deteriorate. Please report paired confidence intervals and variability across training seeds. These results support an average improvement, but not uniformly improved performance across tasks.

Questions for authors

1,Section 4.3 states 50 evaluation runs, but Table 4 reports 65% success, which cannot arise from a single set of 50 binary outcomes. Please provide success counts and explain any averaging.

2,Table 7 states 20 demonstrations per task but reproduces Struct-DPO’s 55.65% result from the 30-trial protocol. Were all methods evaluated on identical episodes? Please also reconcile the different baseline subsets used in Tables 5 and 6.

3,The real-world subset and all-pair variants use 102 and 951 optimization steps. A shared training-loss stopping rule does not eliminate this confound; please provide a matched-budget comparison.

1 comment

We thank the reviewer for the detailed review.

W1. Not isolating physical violations

We partially agree with the reviewer's concerns regarding the interpretation of our structural metrics.

Discrepancies between generated and executed videos can arise not only from physical violations but also from action-decoder errors and appearance variations. Nevertheless, since our metrics primarily capture geometric discrepancies between generated scenes and actual observations, we argue that these discrepancies serve as important indicators of physical execution errors. Moreover, in a video-based policy, the video generator and action decoder jointly constitute the overall policy. Consequently, errors introduced during action decoding can also contribute to discrepancies between predicted and actual physical outcomes.

Our results demonstrate the promise of our approach in identifying physical execution errors manifested as geometric discrepancies. We acknowledge that a more comprehensive assessment of physical consistency, including contact dynamics and object motion, would provide additional evidence. While such an evaluation is beyond the scope of the present work, we consider these aspects valuable directions for future research and will explicitly discuss them in the revised manuscript.

Regarding temporal offsets, each execution cycle begins with an actual observed conditioning frame, from which the corresponding video chunk is generated and its associated action chunk decoded (Appendix B). This design anchors generation and execution to the same initial observation at each cycle, thereby mitigating the accumulation of temporal misalignment across successive cycles.

W2. Related work We thank the reviewer for suggestion. We will add OccScene, ORV, and OmniNWM. These studies demonstrate the effectiveness of incorporating explicit geometric or occupancy representations into scene and video generation. Struct-DPO keeps RGB-only output and uses structure only as a preference signal, so it needs no curated occupancy dataset; it does need environment rollouts, success signals, calibrated cameras, and depth estimation. We will expand the related-work discussion to clarify these methodological differences, including their supervision requirements and the roles of structural representations.

W4. Our experiments address: “Does Struct-DPO improve performance across diverse tasks?” Across 23 tasks, ours achieves an average gain and improves most tasks. While the improvement may appear modest, recently [1] show that SOTA models are often separated by only 1–2%. Thus, our gain supports our claim. Meanwhile, video policy incurs substantial computational costs in experiments, forcing a trade-off between statistical power and task diversity. We chose the latter, as expected of modern video policy, and thus omitted variance. Still, Table below reports standard deviations by treating the Table 1 as six repetitions, which should be interpreted cautiously due to task variation and few repetitions.

MethodPnPnon-PnPOverall
Baseline39.17±6.0639.17\pm6.0659.56±4.0159.56\pm4.0152.46±2.8952.46\pm2.89
Ours (All)42.50±10.6142.50\pm10.6160.00±3.2760.00\pm3.2753.91±4.0053.91\pm4.00
Ours (PnP Only)41.25±9.3241.25\pm9.3263.33±2.7663.33\pm2.7655.65±4.4055.65\pm4.40
[1] Moo Jin Kim et al., Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning, 2026.

Q1. We thank the reviewer for pointing out this inconsistency. The success rates in the real-world experiments were manually recorded by the experimenter, and a recording error led to the incorrect reporting of 65%. The actual result was 32 successes out of 50 trials, corresponding to 64%. We have corrected the reported success rate accordingly.

Q2. We agree with the reviewer that the evaluation settings in Table 7 are inconsistent. However, the methodological observation remains valid: our metric can be replaced by an epipolar criterion within the same pair-construction pipeline. None of the main results in the paper depends on Table 7. We will revise the table and its discussion to make this distinction clear.

Regarding the small baseline difference between Tables 5 and 6, Table 5 uses RoboCasa's automated success evaluation. In contrast, Table 6 relies on manual annotation because RoboCasa does not provide automated metrics for grasp and transport success. We also manually evaluated final task success to maintain consistency across all three milestones. As visually observed task completion may not always agree with the simulator's predefined success conditions, this difference in evaluation criteria may explain the slight discrepancy. We have clarified this point in the revised manuscript.

Reviewer_eQ9LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Struct-DPO to improve the physical consistency of video-based robotic policies by aligning generated rollouts with environment observations using geometric and visual consistency preferences. Experiments on RoboCasa and real robots demonstrate improved task success rates, with promising cross-task transfer from PnP to Non-PnP tasks.

Critical strengths
  1. The finding that geometric and edge consistency can improve downstream robotic task success is interesting and practically relevant.
  2. In Table 1, the “PnP Only” variant achieves larger improvements on Non-PnP tasks excluded from DPO alignment, suggesting promising cross-task transferability. This is an interesting finding.
Critical weaknesses
  1. Experiments mainly focus on internal ablations, lacking competitive baselines to establish the method's effectiveness.
  2. The method largely applies existing DPO techniques with geometry and consistency preferences. The overall improvements are relatively small.
Questions for authors

No

1 comment

We thank the reviewer for the thoughtful assessment and for recognizing the cross-task transfer results.

W2. Novelty: The key innovation is the source of preference supervision We use Diffusion-DPO without modifying its optimizer (Sec. 3.3). Our contribution is to derive preference supervision from the gap between generated predictions and actual execution outcomes. Execution-grounded preferences. We execute actions decoded from a generated rollout and pair the rollout with the resulting observation. The preferred sample thus represents the actual outcome of the same plan, directly exposing the gap between what the policy predicts and what it can execute. Unlike prior methods (Secs. 2.2 and 2.3), which score generated videos using geometric criteria, learned rewards, or VLM judgments, our preferences are grounded in actual environment execution. Structural pair selection. We select pairs with large cross-view depth-reprojection errors or substantial edge and displacement discrepancies, focusing alignment on the most structurally inconsistent rollouts. Learning from failures. Hindsight relabeling allows failed episodes to contribute preference pairs. These account for 71% of PnP-only pairs and 59% of all-task pairs (Sec. 4.2). Minimal changes to the policy. The policy retains its RGB output and requires no human preference labels, learned reward model, or curated 3D or occupancy data. Structural supervision is extracted directly from environment rollouts.

W2. Magnitude of the improvements Simulation. Across 23 tasks, Struct-DPO achieves an average performance gain and improves most tasks. While the improvement may appear modest, recent work [1] shows that state-of-the-art models often differ by only 1–2 percentage points. Thus, our 3.19-point gain is meaningful and supports our claim. Moreover, PnP-only improves all three structural metrics, as well as FVD, which is not directly optimized, and grasp, transport, and final success. Real robot. In the single-task and multi-task setting. Compared with simulation, real-world deployment involves more frequent physical inconsistencies, diverse physical constraints, and visual disturbances such as lighting and reflections. The substantial improvement therefore provides strong evidence that incorporating physical priors can improve real-world policy performance. Cross-task transfer. Aligning exclusively on Pick-and-Place pairs improves success on 15 Non-PnP tasks from 59.56% (268/450) to 63.33% (285/450), with 10 tasks improving, 2 unchanged, and 3 declining (Table 5). This demonstrates that structural preferences derived from one task group can benefit performance on others, even though the base policy was originally SFT-trained on all tasks.

W1. Baselines As a post-hoc alignment method, Struct-DPO is primarily evaluated by comparing the same Video Policy before and after alignment, with all non-LoRA parameters frozen (Sec. 3.3). We additionally compare using all preference pairs against structurally selected pairs (Table 4), demonstrating the value of informative pair selection. Table 7 further illustrates that alternative geometric criteria, such as epipolar constraints, can be incorporated into the same pipeline. Together, these results establish Struct-DPO as an execution-grounded alignment approach that extracts informative structural preferences from both successful and failed rollouts. Its improvements across diverse simulation tasks and real-robot settings demonstrate the effectiveness of the proposed approach.

[1] Moo Jin Kim et al., Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning, 2026.

Reviewer_3iZJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper fine-tunes Video Policy (an SVD-based video model with an action head) to make its predicted videos more physically consistent. The trained policy is run in the simulator, and for each step the real simulator observation is used as the winning sample and the model's generated video as the losing sample. Pairs are kept if they are in the top 20% error under any of three metrics (depth reprojection across views, edge Chamfer distance, optical flow), and failed episodes get a generic prompt. A LoRA is then trained with Diffusion-DPO plus an SFT loss on the simulator videos. On 23 RoboCasa tasks success goes from 52.46% to 55.65%, and on a real UR5e cup task from 52% to 82%.

Critical strengths
  1. The problem is real, generated videos can look fine but be physically wrong, which breaks the decoded actions.
  2. The method is simple, needs no architecture change, and is cheap to train (LoRA, 1K steps). The real robot experiment is a plus.
Critical weaknesses
  1. In practice what this paper does is mostly collecting more in-domain data with the policy's own rollouts and fine-tuning on it, which is expected to help. On the other hand, it lacks the simple baseline of plain SFT on the same collected rollouts (no DPO).
  2. The comparison with epipolar-DPO (Table 7) is not on the same protocol: the baseline and epipolar rows use 20 demos per task (baseline 50.87 instead of 52.46 in Table 1), while the Struct-DPO number is copied from Table 1.
Questions for authors

Please see the weaknesses part.

2 comments

We thank the reviewer for the careful summary. We address the central question first: is Struct-DPO more than fine-tuning on additional in-domain data?

W1. Relation to plain SFT The data are not additional demonstrations. The alignment data consist of the policy's own executions, collected on the same environment configurations used to train the base policy, while evaluation uses unseen layout-and-style configurations (App. B). No new scenes or expert behavior are introduced. Moreover, 71% of the selected pairs in PnP-only and 59% in All come from failed episodes (Sec. 4.2). Plain SFT on these rollouts would therefore train the policy to reproduce its own, mostly unsuccessful, executions. What Struct-DPO adds is the pairing. Each generated rollout is paired with the observation obtained by executing the actions decoded from that same rollout, and we retain pairs with the largest structural discrepancies. The generated (losing) sample captures what the policy predicted but could not realize in execution. This contrast between generation and execution contributes to training through the preference term, rather than the SFT term. The SFT term in our objective serves to regularize against degenerate solutions of the preference loss (Sec. 3.3). This alignment is applied to a base policy that had already received 369K SFT steps in simulation and, on the real robot, was trained with SFT until its training loss stopped decreasing (App. B, D). Our primary contribution is therefore not simply applying DPO to additional data, but selecting informative preference pairs based on structural discrepancies between generated and executed outcomes. Across 23 simulation tasks, Struct-DPO improves the overall success rate from 52.46% to 55.65%, with improvements on 15 tasks, ties on 3, and declines on 5. On the real robot, success on the evaluated task increases from 52% (26/50) to 82% (41/50). Table 4 further illustrates the practical value of structural pair selection. DPO using all collected pairs achieves 78% success, whereas DPO using the geometrically selected subset achieves 82%, despite using only 45% of the preference pairs and substantially fewer optimization steps. This shows that the selected subset is sufficient to achieve a stronger result with considerably less preference data and optimization, supporting the practical value of structural pair selection.

W2. Inconsistent evaluation in Table 7 We agree with the reviewer that the evaluation settings in Table 7 are inconsistent. However, the methodological observation remains valid: our metric can be replaced by an epipolar criterion within the same pair-construction pipeline. None of the main results in the paper depends on Table 7. We will revise the table and its discussion to make this distinction clear.

Thank you for the clarification. I appreciate the clarification of execution-grounded preference supervision and the additional discussion of cross-task transfer. But my main concerns remain. The evaluation still relies primarily on internal ablations, without sufficient comparisons against competitive alternatives or matched-budget baselines to establish the specific contribution of structural preference alignment. Moreover, the methodological novelty remains relatively limited, and the simulation improvements are modest without further statistical analysis. I maintain my original assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.