acceptodds
Under review as a conference paper at ICLR 2027

Self-Supervised VLA Preference Optimisation via Temporally Shifted Guidance

Abstract

Vision-language-action (VLA) models learn diverse robotic behaviors from demonstrations, but improving them beyond imitation often requires costly sources of feedback. Preference optimization offers a direct and effective post-training strategy, yet obtaining useful action comparisons can still depend on human intervention, additional interactions with environments, or external evaluators. We introduce temporally shifted classifier-free guidance (TS-CFG) for flow-matching action policies, using their learned action structure to construct hard negatives from demonstrations. The central idea is to steer a demonstration's reconstruction through the model's conditional predictions, targeting hard negatives with subtle deviations from demonstrated behavior. TS-CFG adapts classifier-free guidance by combining predictions under the original context and a nearby temporal context from the same episode. Temporal proximity provides related conditioning, intended to avoid mixing markedly different action distributions while introducing a structured perturbation to generation. Treating each demonstration as preferred over its temporally shifted alternative provides a preference-learning signal using only existing demonstrations and the pretrained policy. This construction supports direct preference post-training without new environment interaction, human preference annotation, or auxiliary world-model or reward evaluation. The additional computation is confined to training, leaving the policy's deployment procedure unchanged. Extensive experiments on RoboTwin and RoboCasa show improvements in task success over the starting policies.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision Language Action Models, Post Training, Preference Optimisation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JVocOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TS-CFG, a method that constructs DPO preference pairs for flow-matching VLAs using only existing demonstration data. It first inverts demonstration actions into latent variables within the original context and then reconstructs them using a frozen policy; the velocity field blends predictions derived from the original context and the context of adjacent frames within the same episode, with the weight of the temporal context decaying from 1 to 0 during reconstruction. This process yields hard negative samples, against which the demonstration actions are prioritized using a Diffusion-DPO-style loss. Improvements were achieved on the π0.5 model across the RoboTwin (Easy/Hard) and RoboCasa365 benchmarks.

Critical strengths

Strengths:

1.This is a simple and practical method. It requires neither new rollout or manual annotation stages nor reward or world models, and the deployment-time sampler remains unchanged. "Demonstration inversion combined with temporal context guidance" is a novel approach with a clear rationale, where the generated hard negative samples are tied to specific demonstrations.

2.The setup for the simulation experiment assignments is meticulous and reliable.

Critical weaknesses

Weakness:

1.The evidence supporting the main reported results is relatively weak; all results reflect values ​​from a single random seed, and the improvement over π0.5\pi_{0.5}in the primary experiments is marginal. Furthermore, the simulation experiments only evaluated π0.5\pi_{0.5}, and real-world robot experiments are missing.

2.There is no evidence that negative samples of action a−a^{-} generated via perturbation are inferior, nor is there a description of how the perturbation alters the original demonstration.

Questions for authors
  1. Please provide the mean and standard deviation (std) results across multiple random seeds.
  2. What happens when "aaaa" is introduced into the environment, and what is the success rate?
  3. Can the TS-CFG method be generalized to other VLA models or real-world robots?
2 comments

Response to Reviewer JVoc (Part 1/2)

Thank you for recognizing the demonstration-specific construction and the value of self-supervised preference generation. We have added repeated runs, direct execution analysis and a second policy backbone.

W1 / Q1. Mean and standard deviation across training seeds

We completed three independent training seeds for every main-table post-training method, using the same settings without retuning. Each policy is evaluated on 50 tasks with 50 rollouts per task. Results are mean +/- sample standard deviation (%):

MethodRoboTwin EasyRoboTwin HardRoboCasa
Base56.2458.3618.00
BC55.13 +/- 0.9558.75 +/- 0.7018.23 +/- 0.15
DITTO*58.60 +/- 1.0360.95 +/- 0.9519.00 +/- 0.38
VTLA*58.59 +/- 0.1762.03 +/- 0.3718.01 +/- 0.45
TS-CFG61.39 +/- 0.5663.99 +/- 0.7421.33 +/- 0.44
TS-CFG + LN61.05 +/- 0.4665.27 +/- 0.3721.84 +/- 0.42

TS-CFG exceeds the strongest baseline mean by 2.79 / 1.96 / 2.33 percentage points, with improvements across independent training runs on all three evaluations.

The performance gap becomes substantially larger with full-parameter fine-tuning. We used LoRA in the main experiments for computational efficiency. We additionally evaluated all three preference methods with full updates on RoboTwin Hard50, using the same initialization, data, global batch 8 and 5k-update budget. These are single-seed comparisons at seed 31; full-parameter runs share LR 1e-8, while the original LoRA runs use 5e-7.

MethodLoRA (%)Full-parameter (%)Change (pp)
DITTO*62.0459.84-2.20
VTLA*62.2061.24-0.96
TS-CFG64.0868.60+4.52

Neither baseline improves over its own LoRA counterpart in this comparison, whereas TS-CFG gains 4.52 points. Under the matched full-parameter settings, TS-CFG exceeds DITTO* and VTLA* by 8.76 and 7.36 points, respectively. Its advantage therefore extends beyond the parameter-efficient setting and is substantially larger under full updates.

W2 / Q2. How is the demonstration changed, and what happens during execution?

We interpret "aaaa" in Q2 as referring to executing the temporally perturbed actions in the environment. Please let us know if you intended something different; the following addresses that interpretation.

The negative is a modified reconstruction of a particular demonstration, not an independently sampled action. For demonstrated action a+a^+, current context oo and nearby context hh, we obtain the demonstration's latent under the current context and reconstruct it with temporal guidance:

z⋆=To−1(a+),a−=TDEC(z⋆).z^{\star}=T_o^{-1}(a^+),\qquad a^-=T_{\mathrm{DEC}}(z^{\star}).

At denoising step kk, the field is

vk(x,τ)=vo(x,τ)+λk[vh(x,τ)−vo(x,τ)],λk=1−k/9,k=0,…,9.v_k(x,\tau)=v_o(x,\tau)+\lambda_k[v_h(x,\tau)-v_o(x,\tau)],\qquad \lambda_k=1-k/9,\quad k=0,\ldots,9.

Both velocities are evaluated at the same intermediate action state and time. The difference between the temporal and current velocity fields changes the reconstruction toward what the policy would predict under a nearby manipulation state. Its weight decreases to zero, so reconstruction finishes under the current context. Inversion preserves the link to the demonstrated behavior; temporal guidance supplies a structured deviation from it. The original demonstration remains the positive.

This design aims at hard negatives that are close in task and behavior but differ in compatibility with the current state. Under the same DPO objective, it reaches 64.08% on Hard50, compared with 60.56% for zero-guidance/noise reconstruction, 58.16% for random initialization and 58.32% for random same-episode conditioning.

Direct execution of the perturbation. We apply guidance from the context five frames in the past to the base policy during online rollouts, using the same linear 1-to-0 schedule. Both guided and unguided rollouts start from random noise because no demonstrated action is available online. With matched execution settings, success falls from 58.24% to 49.48%, a drop of 8.76 points; success decreases on 37 of 50 tasks.

Execution conditionRoboTwin Hard50 (%)
Unguided base58.24
Temporal guidance throughout the rollout49.48

The perturbed policy still completes 49.48% of rollouts: temporal guidance reduces execution reliability without simply destroying task-directed behavior. This is the intended balance for preference construction: expose errors while remaining sufficiently plausible to provide informative negatives for preference training. During training, inversion additionally ties each alternative to its demonstrated positive.

Response to Reviewer JVoc (Part 2/2)

W2 / Q2. Which behavioral changes follow preference training?

The clearest change is more reliable completion after the robot has already made useful progress. On RoboTwin Hard, TS-CFG improves over base on 35 of 50 tasks, using its three-seed mean. Grouping all 50 tasks by their manipulation requirements gives the following descriptive breakdown; each task appears once and rates are averaged equally across tasks:

Task groupTasks improved / total, vs. baseBaseTS-CFG
Articulated opening and switching2/346.0062.22
Button pressing and direct gripper contact3/368.6779.33
Hand-to-hand object transfer2/235.0045.33
Multi-object placement, ordering and stacking8/1055.6063.47
Container manipulation sequences4/551.2056.67
Lifting and held-object manipulation8/967.3370.74
Single-object relocation and pose adjustment8/1659.3862.71
Held-tool targeting0/268.0063.00

Seven of eight group averages improve, with gains in both handover tasks, all three direct-contact tasks and eight of ten multi-object tasks. This points to better execution through changing contact and task stages, rather than an improvement confined to one object or task family.

Representative full-evaluation results are:

TaskBase (%)TS-CFG, three-seed mean (%)
Cup placement40.0084.67
Microwave opening30.0074.00
Two-can placement38.0068.67
Microphone handover42.0058.00

The saved rollouts show where these differences arise. We inspected a fixed, outcome-independent sample of ten scenes per task and policy, using the original TS-CFG training seed:

  • Cup placement: both policies reach the coaster region in 10/10 scenes. Base completes placement in 4/10, versus 10/10 for TS-CFG. The separation occurs after the cup has reached the correct target region, not in identifying where to move it.
  • Two-can placement: at least one can reaches the box in 8/10 base scenes and 10/10 TS-CFG scenes, but both are placed in 1/10 versus 6/10. The larger difference is in progressing from a partially completed task to the second successful placement.
  • Microwave opening: base can initiate door movement but fail to complete the opening; TS-CFG rollouts continue the interaction to the successful open state. This is a difference in completing the contact-dependent motion, not simply reaching the appliance.

An additional paired microphone-handover example illustrates object retention. Both policies grasp the microphone. Base loses it during the transfer between grippers; TS-CFG retains it and finishes the handover. The relevant distinction is maintaining the object through a change in which hand controls it.

These stages explain why temporally nearby alternatives are useful. The task and objects remain the same across nearby frames, but the action needed before contact, while holding an object, or after reaching its target changes. A policy can therefore produce a task-relevant action that is mistimed for the current hand-object relationship. TS-CFG builds this local mismatch into the reconstruction of the same demonstration and decreases its influence before reconstruction ends. Preference training then favors the demonstrated, current-state action over that plausible alternative. This connects the construction to the observed improvements in sustaining transfer, completing placement and continuing a multi-object sequence, rather than attributing them only to generic task recognition.

W1 / Q3. Other VLA models and real robots

We also evaluated TS-CFG with GR00T N1.6 on RoboCasa full50:

GR00T N1.6 conditionSuccess (%)
Base19.72
TS-CFG22.32

The 2.60-point improvement extends the evidence to a second flow-policy backbone, without changing its deployment sampler. The method acts through the conditional velocity field, so its construction is not specific to pi0.5's architecture. Together with the diverse bimanual and household tasks in RoboTwin and RoboCasa, this supports applicability across simulated tasks, environments and backbones. Real-robot evaluation is the next extension to test the same construction under physical sensing and execution conditions.

Reviewer_VLUROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TS-CFG, an offline preference post-training method for flow-matching VLA policies. It inverts demonstrated actions through a frozen policy and reconstructs them using a mixture of the original and nearby temporal conditioning, treating the resulting actions as dispreferred. A reference-relative flow-error objective then updates the policy. Experiments with pi0.5 on RoboTwin and RoboCasa show improved success rates, with ablations of the negative construction and guidance settings.

Critical strengths
  1. The method uses existing demonstrations and the pretrained policy, while leaving deployment unchanged.
  2. The matched perturbation and conditioning ablations provide useful evidence that the negative construction matters.
Critical weaknesses
  1. Statistical reliability and model selection. Each benchmark is reported using one training seed, without uncertainty across independent runs. Several improvements over DITTO* are only around 1–2 percentage points, making their robustness important to establish. The manuscript also does not clarify whether hyperparameters were selected using a separate validation split.
  2. Incomplete attribution to temporal guidance. LN-only uses nonzero latent noise and therefore does not replace the missing zero-guidance, zero-noise inversion–reconstruction control. Without this baseline, the contribution of temporal conditioning cannot be fully separated from reconstruction error or reference-policy bias. A direct audit of whether generated alternatives are actually inferior to demonstrations is also missing.
Questions for authors

Additional Limitations 3. Influence of padded channels. Generated values in padded action channels are retained and included in the preference loss, although they do not control the robot. A physical-channel-only ablation and a decomposition of the preference margin would help determine whether these dimensions materially affect the results. 4. Limited comparative coverage. DITTO* and VTLA* are adapted negative-construction baselines and do not fully represent the original methods. Evaluation within the π0.5 family and simulated environments also leaves applicability to other policies and real robots uncertain. 5. Uncontrolled context changes. Temporal contexts can differ in both direction and instruction wording, alongside changes in observations and state. Separate past/future comparisons and fixed-instruction controls would clarify which aspects of the conditioning change produce the benefit. 6. Weak evidence of unseen-task transfer. RoboCasa unseen composite-task success increases from 1.00% to 1.25%, corresponding to only two additional successes across 800 rollouts. This provides limited evidence of transfer beyond the post-training task set, despite the stronger improvements on seen tasks. 7. Unreported training cost. Each generated negative requires approximately thirty reference velocity evaluations through inversion and guided reconstruction. Wall-clock time and peak memory relative to the baselines are needed to assess the practical cost of the method.

Questions For Authors

  1. What happens when both guidance and latent noise are zero, and how often are generated actions actually worse than demonstrations?
  2. Are the gains robust across training seeds, and how were hyperparameters selected?
  3. Do the results persist after excluding padded channels and controlling instruction wording and temporal direction?
5 comments

Response to Reviewer VLUR (Part 1/3)

Thank you for the specific suggestions. We have added repeated training runs and controls for reconstruction, padding and instruction wording, and expanded the comparisons and execution analysis.

W1 / Q2. Training-seed reliability and hyperparameter selection

We completed three independent training seeds for every main-table post-training method. Each policy is evaluated on 50 tasks with 50 rollouts per task; values are mean +/- sample standard deviation (%).

MethodRoboTwin EasyRoboTwin HardRoboCasa
Base56.2458.3618.00
BC55.13 +/- 0.9558.75 +/- 0.7018.23 +/- 0.15
DITTO*58.60 +/- 1.0360.95 +/- 0.9519.00 +/- 0.38
VTLA*58.59 +/- 0.1762.03 +/- 0.3718.01 +/- 0.45
TS-CFG61.39 +/- 0.5663.99 +/- 0.7421.33 +/- 0.44
TS-CFG + LN61.05 +/- 0.4665.27 +/- 0.3721.84 +/- 0.42

TS-CFG improves over the strongest baseline mean by 2.79 / 1.96 / 2.33 percentage points. Additional seeds use the same configurations without retuning.

The performance gap becomes substantially larger with full-parameter fine-tuning. We used LoRA in the main experiments for computational efficiency. We additionally evaluated all three preference methods with full updates on RoboTwin Hard50, using the same initialization, data, global batch 8 and 5k-update budget. These are single-seed comparisons at seed 31; full-parameter runs share LR 1e-8, while the original LoRA runs use 5e-7.

MethodLoRA (%)Full-parameter (%)Change (pp)
DITTO*62.0459.84-2.20
VTLA*62.2061.24-0.96
TS-CFG64.0868.60+4.52

Neither baseline improves over its own LoRA counterpart in this comparison, whereas TS-CFG gains 4.52 points. Under the matched full-parameter settings, TS-CFG exceeds DITTO* and VTLA* by 8.76 and 7.36 points, respectively. Its advantage therefore extends beyond the parameter-efficient setting and is substantially larger under full updates.

For the main LoRA experiments, we follow established DPO and LoRA fine-tuning practice, without TS-CFG-specific optimization-hyperparameter tuning. The conservative learning rate, 5e-7, matches the original DPO implementation, while adapter rank and scaling follow the OpenPI backbone configurations. Optimization settings and the update budget are held fixed across the preference comparisons and additional seeds.

The method-specific choices are nearby conditioning (gap 5) and linear decay to the current context, examined through the gap and schedule ablations. No separate hyperparameter-validation split was used. The main pi0.5 comparisons use the fixed 5k checkpoint, with the same settings across methods and additional seeds and no per-experiment retuning.

W2 / Q1. Does temporal guidance add anything beyond reconstruction error?

We added the requested zero-guidance, zero-noise control: invert the demonstration under its current context, reconstruct under that same context, and use the reconstruction as the DPO negative.

Negative constructionRoboTwin Hard50 (%)
Base policy58.36
Cycle-only: guidance=0, noise=060.56
LN-only59.60
TS-CFG64.08
TS-CFG + LN65.52

With the same training seed and DPO objective, temporal guidance adds 3.52 points over Cycle-only. Thus, numerical reconstruction can contribute, but does not account for the full improvement from demonstration-anchored temporal reconstruction.

W2 / Q1. What happens when temporally perturbed actions are executed?

We directly execute temporally guided actions using the base checkpoint. Online rollouts have no demonstrated action to invert, so both conditions start from random noise. They differ only in whether the context from five frames earlier is used with guidance decreasing linearly from 1 to 0 during reconstruction. Execution settings are matched.

Execution conditionRoboTwin Hard50 (%)
Unguided base58.24
Temporal guidance throughout the rollout49.48

Guidance lowers success by 8.76 points, with lower success on 37 of 50 tasks, while still completing 49.48% of rollouts. The perturbation therefore reduces execution reliability without simply destroying task-directed behavior. This is the intended balance: alternatives should differ enough to expose errors, while remaining sufficiently plausible to provide informative negatives for preference training.

Response to Reviewer VLUR (Part 2/3)

Limitation 3 / Q3. Are padded channels responsible for the improvement?

The improvement persists when unused action channels are removed from the preference signal or zeroed in the final generated action:

ConditionHard50 (%)
TS-CFG64.08
Preference loss on physical channels only64.80
Zero padding after negative generation64.04

The first control retains the original loss scaling and full-dimensional generation, changing only which channels contribute to the preference loss. The second changes only the final negative, after inversion and reconstruction.

We also split the reference-adjusted preference margin into the contribution from the 14 physical control channels and the 18 padding channels. Across 500 demonstration anchors, averaging 16 matched noise/time draws per anchor, the mean absolute padding contribution is only 0.37% of the physical-channel contribution. Removing padding flips the sign of that averaged margin for none of the 500 anchors. In plain terms, the unused coordinates contribute very little to the preference signal, and the trained controls retain the improvement without them.

Limitation 5 / Q3. Instruction wording and temporal direction

We first keep the instruction unchanged while replacing only the images and robot state with those from the nearby frame:

ConditionHard50 (%)
Original TS-CFG64.08
Fixed instruction64.92

The fixed-instruction result shows that changing instruction wording is not needed for the gain.

Separately, we compare past and future contexts, then reverse the guidance sign for the future context. All three use current-context demonstration inversion, a linear 1-to-0 guidance schedule, 5k LoRA updates and the original training seed, with the same data, loss and optimization hyperparameters:

Temporal contextReconstruction guidanceHard50 (%)
Past 5 framesInterpolation (TS-CFG)64.08
Future 5 framesInterpolation46.24
Future 5 framesExtrapolation67.04

For future context ff, extrapolation changes only the reconstruction rule to

vk(x,τ)=vo(x,τ)+λk[vo(x,τ)−vf(x,τ)],λk=1−k/9,k=0,…,9.v_k(x,\tau)=v_o(x,\tau)+\lambda_k[v_o(x,\tau)-v_f(x,\tau)],\qquad \lambda_k=1-k/9,\quad k=0,\ldots,9.

The two fields are evaluated at the same intermediate action state and time. The original demonstration remains the positive, and the reconstructed action is the negative. No latent noise is added. The table reports post-training evaluation with the ordinary sampler, not execution of the guided negatives.

Past and future contexts are not interchangeable sources of negatives. Past contexts describe states the robot has already moved beyond, so their guidance can introduce stale actions that remain plausible but are mistimed for the current state. Future contexts instead describe progress toward the goal and can steer reconstruction toward useful continuations. Penalizing those continuations risks suppressing task progress. Extrapolation reverses this direction, constructing alternatives that move away from the future-conditioned continuation. Its 67.04% success, compared with 46.24% for future interpolation and 64.08% for the original TS-CFG, supports this temporal interpretation: useful negatives depend on the direction of the perturbation relative to progress, not context mismatch alone.

Response to Reviewer VLUR (Part 3/3)

Limitation 4. How do the adapted baselines relate to the original methods?

The starred baselines were designed to isolate negative construction, holding the backbone, data, demonstration positive, flow-preference objective and update budget fixed. DITTO* retains demonstration-versus-generated preferences. VTLA* retains GT-distance-based selection, using the farthest of 16 candidates as the negative and the demonstration as the common positive.

Some adaptation is necessary: DITTO operates on language outputs, while VTLA uses a token-based vision-tactile policy. Our policies instead generate continuous action chunks through flow matching. Keeping GT as the positive in the starred comparison is an additional deliberate control, so a gain cannot be attributed to changing both positive and negative supervision at once.

To test closer versions of the original preference procedures, we also restore DITTO's iterative generation/replay and VTLA's generated-versus-generated ranking:

MethodPreference constructionHard50 (%)
DITTO*GT versus a frozen-policy sample62.04
DITTO-ITERFour generation/update rounds, with expert, replay and inter-policy pairs60.52
VTLA*GT versus the farthest of 16 generated candidates62.20
VTLA-PAIRTwo generated actions: closer preferred over farther by physical-action L1 distance to GT58.08
TS-CFGGT versus its temporally guided reconstruction64.08

The original pair-construction principles are therefore tested both under a matched GT-positive comparison and with the additional iterative or generated-pair components restored. TS-CFG remains stronger in both comparisons. All use the same flow-VLA backbone and training budget; the asterisks identify the controlled adaptations.

Limitation 4. Other backbones and real robots

TS-CFG also improves GR00T N1.6 on RoboCasa full50:

GR00T N1.6 conditionSuccess (%)
Base19.72
TS-CFG22.32

The 2.60-point gain extends the results beyond pi0.5 to a second flow-policy backbone, with its deployment sampler unchanged. Together, RoboTwin's diverse bimanual tasks, RoboCasa's household tasks and the GR00T N1.6 experiment provide evidence across tasks, environments and policy architectures. Real-robot evaluation is the next extension to test these gains under physical sensing and execution conditions.

Limitation 6. Unseen composite-task transfer

The three-seed RoboCasa results are:

MethodAtomic seenComposite seenComposite unseen
Base44.565.131.00
TS-CFG49.85 +/- 1.499.04 +/- 0.721.54 +/- 0.40
TS-CFG + LN49.96 +/- 1.2510.13 +/- 0.571.92 +/- 0.51

Unseen compositions are difficult even for the pretrained policy, whose success is 1.00%. The increases to 1.54% and 1.92% correspond to approximately 54% and 92% relative improvement, respectively, although absolute success remains low. The small absolute scale should therefore be read alongside the low starting rate.

Our central claim is refinement of a pretrained policy using its existing demonstrations, without collecting new task supervision. The strongest gains occur on the seen-task distribution, including nearly doubled seen-composite success. The unseen results indicate positive transfer from that refinement; solving novel long-horizon compositions is a broader objective than the post-training problem addressed here.

Limitation 7. Training cost and peak memory

The matched profiling protocol uses one NVIDIA L40S, global batch 8, identical precision and LoRA settings, and three repetitions of 100 measured updates after 20 warm-up updates. It includes online negative generation and optimizer computation, excluding data loading, diagnostic monitoring and checkpointing for all methods.

MethodSeconds/update, mean +/- SDNegative generation, s/updatePeak allocated GiB
BC1.213 +/- 0.0040.0009.80
DITTO*3.475 +/- 0.0090.40717.36
VTLA*9.689 +/- 0.0286.60517.36
TS-CFG4.288 +/- 0.0081.21817.36

TS-CFG takes 1.23x the DITTO* time per update at the same peak allocated memory, and 3.53x the BC time. Its negative construction costs 1.218 seconds/update, without generating and ranking a candidate pool. Deployment cost is unchanged: neither inversion nor temporal guidance is used by the deployed policy.

Reviewer_kF6HOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces temporally shifted classifier-free guidance (TS-CFG), a self-supervised method for constructing hard negatives from demonstration data to post-train flow-matching VLA policies. The policy re-generates each demonstrated action while guidance pushes the generation toward what it would produce given a slightly earlier observation. This produces a negative that stays close to the demonstration but deviates in a structured way. The demonstration is treated as preferred over this negative, and the pairs are used to fine-tune the policy with DPO, requiring no rollouts, rewards, or human labels. Post-training a $\pi$0.5 policy with TS-CFG improves success on RoboTwin 2.0 and RoboCasa365 over continued behavior cloning and adapted versions of prior self-supervised preference methods (DITTO, VTLA). Ablations isolate the main design choices: using a nearby past observation is more effective than a randomly chosen frame from the same episode or corrupted inputs, and both inverting the demonstration and the decaying guidance schedule matter. Overall, this is a cheap, self-supervised approach to improving imitation-learning policies that provides modest but consistent gains on two commonly used simulation benchmarks.

Critical strengths
  1. The ablations are strong and directly test the paper's main claims about TS-CFG's design choices (e.g., anchoring to the demonstration, using a nearby observation, the guidance schedule). The method also improves over the baselines on both benchmarks.

  2. The method is self-supervised and cheap: it improves policy success without additional rollouts, human labels, or external evaluators. It requires only LoRA fine-tuning and leaves the deployed policy's sampling procedure unchanged.

Critical weaknesses
  1. Qualitative descriptions of where the method succeeds are not included, so it is hard to understand what about the method actually helps compared with the baselines. On which types of tasks do the improvements occur, and at which stage of those tasks? This analysis could also shed light on why temporally shifted observations produce good negative examples.

  2. The central appeal of TS-CFG is that it is cheap and requires no extra supervision, which would make it a natural default for post-training imitation policies. However, the paper does not establish that the gains are robust enough for this. The improvement over baselines is modest, and the results appear to come from a single training run with no variance reported. Results are also sensitive to the temporal offset and guidance weight: a 10-frame offset or a constant guidance weight of 0.5 performs no better than, or worse than, the original policy.

Questions for authors
  1. Where in the rollouts does the method improve over prior approaches? Is it during grasping, placing, choosing the right object to move toward, etc.?

  2. The training method seems lightweight (LoRA updates only) and requires no manual annotation, but the paper does not report GPU type or wall-clock training time. Since negatives are regenerated on every training iteration, how does the compute cost compare with continued BC and DITTO*?

  3. Is full fine-tuning or a longer training time going to degrade the learning dynamics? Or will the method keep improving?

  4. Can you provide confidence intervals on any of these results? With relatively modest gains it would be useful to have a better understanding of the variance of this method.

3 comments

Response to Reviewer kF6H (Part 1/3)

Thank you for highlighting the practical value of TS-CFG and asking where its improvements arise. We have expanded the behavioral analysis, repeated the main experiments across training seeds, and added the requested controls.

W1 / Q1. Which tasks and execution stages improve?

The clearest pattern is better completion of state-dependent manipulation: maintaining an object during transfer, finishing a placement after reaching the target, and progressing from one object to the next. On RoboTwin Hard, TS-CFG improves over base on 35 of 50 tasks, using its three-seed mean.

Grouping all 50 tasks by the manipulation they require gives the following descriptive breakdown. Each task belongs to one group; rates are percentages, averaged equally across tasks.

Task groupTasks improved / total, vs. baseBaseTS-CFG
Articulated opening and switching2/346.0062.22
Button pressing and direct gripper contact3/368.6779.33
Hand-to-hand object transfer2/235.0045.33
Multi-object placement, ordering and stacking8/1055.6063.47
Container manipulation sequences4/551.2056.67
Lifting and held-object manipulation8/967.3370.74
Single-object relocation and pose adjustment8/1659.3862.71
Held-tool targeting0/268.0063.00

Thus, gains occur in seven of eight group averages, with particularly clear improvements in transfer, multi-object manipulation and direct contact. The pattern is less uniform for single-object pose adjustment and tool targeting.

Rollout inspection localizes the changes beyond initial reaching. Representative tasks from the full evaluations are below; TS-CFG values are three-seed means:

TaskBaseTS-CFGStage where the difference is visible
Cup placement40.0084.67Completing placement after transporting the cup to the coaster
Microwave opening30.0074.00Continuing initial door movement to a sufficiently open state
Two-can placement38.0068.67Completing the second placement after handling the first can
Microphone handover42.0058.00Retaining the object through the transfer between grippers

We examined a fixed, outcome-independent video sample of ten scenes per task and policy, using the original TS-CFG training seed for this inspection:

  • Reaching versus placing. In cup placement, both policies reach the coaster region in 10/10 scenes, but complete placement in 4/10 and 10/10, respectively. The difference appears after transport to the correct target, rather than in finding or approaching it.
  • Partial versus complete multi-object execution. In two-can placement, at least one can reaches the box in 8/10 versus 10/10 scenes, whereas both are placed in 1/10 versus 6/10. The larger separation is in completing the sequence, not merely manipulating one can.
  • Initiating versus sustaining contact-dependent motion. Microwave rollouts show base beginning to move the door without completing the opening, while TS-CFG continues to the successful open state.

An additional paired microphone-handover example illustrates object retention: both policies initially grasp the microphone; base drops it during transfer, whereas TS-CFG retains it and completes the handover.

Why temporal negatives target these errors. Successful manipulation requires an action to match the current stage, not merely the right task or object. A movement appropriate before contact, during transport or before release can become inappropriate a few frames later, despite looking very similar. Nearby contexts preserve the task and objects while changing exactly these hand-object and contact relationships. TS-CFG uses this difference to perturb the reconstruction of the same demonstrated action, then reduces guidance so that reconstruction finishes under the current context. This supplies a plausible, locally mismatched alternative against which the demonstration is preferred.

The observed improvements therefore fit the intended learning signal: complete the placement once the object is at the target, retain it through a transfer, and proceed to the next object after partial progress. The strongest behavioral evidence is improved execution conditional on progress already made, rather than object recognition alone.

Response to Reviewer kF6H (Part 2/3)

W2 / Q4. Are the gains reproducible, and how sensitive are the settings?

We completed three independent training seeds for every main-table post-training method, with 50 tasks and 50 rollouts per task. Results are mean +/- sample standard deviation (%):

MethodRoboTwin EasyRoboTwin HardRoboCasa
Base56.2458.3618.00
BC55.13 +/- 0.9558.75 +/- 0.7018.23 +/- 0.15
DITTO*58.60 +/- 1.0360.95 +/- 0.9519.00 +/- 0.38
VTLA*58.59 +/- 0.1762.03 +/- 0.3718.01 +/- 0.45
TS-CFG61.39 +/- 0.5663.99 +/- 0.7421.33 +/- 0.44
TS-CFG + LN61.05 +/- 0.4665.27 +/- 0.3721.84 +/- 0.42

TS-CFG exceeds the strongest baseline mean by 2.79 / 1.96 / 2.33 percentage points. Its 95% Student-t confidence intervals for the mean across training seeds are [59.99, 62.78] / [62.14, 65.84] / [20.24, 22.43]. Base is a fixed checkpoint; the training-seed statistics apply to the post-trained methods.

The performance gap becomes substantially larger with full-parameter fine-tuning. We used LoRA in the main experiments for computational efficiency. We additionally evaluated all three preference methods with full updates on RoboTwin Hard50, using the same initialization, data, global batch 8 and 5k-update budget. These are single-seed comparisons at seed 31; full-parameter runs share LR 1e-8, while the original LoRA runs use 5e-7.

MethodLoRA (%)Full-parameter (%)Change (pp)
DITTO*62.0459.84-2.20
VTLA*62.2061.24-0.96
TS-CFG64.0868.60+4.52

Neither baseline improves over its own LoRA counterpart in this comparison, whereas TS-CFG gains 4.52 points. Under the matched full-parameter settings, TS-CFG exceeds DITTO* and VTLA* by 8.76 and 7.36 points, respectively. Its advantage therefore extends beyond the parameter-efficient setting and is substantially larger under full updates.

The sensitivity results on Hard50 are:

SettingValuesSuccess (%)
Temporal gap1 / 3 / 5 / 1062.36 / 63.40 / 64.08 / 58.84
Initial guidance, decreasing to zero0.50 / 0.75 / 1.0062.88 / 64.28 / 64.08
Constant guidance0.10 / 0.25 / 0.5063.28 / 62.88 / 56.56

Our objective is a subtle, demonstration-linked deviation, not the strongest possible corruption. Gaps 1-5 and all three decreasing schedules improve over base; performance does not depend on one exact gap or initial weight. Gap 10 can span a larger change in manipulation state. Constant 0.5 keeps substantial donor influence even in the final reconstruction steps, unlike the decreasing schedule, which returns to the current context. Both move away from the intended local perturbation followed by current-conditioned refinement. Their weaker results support this design rationale: making alternatives more different is not necessarily useful hard-negative construction. Constant 0.1 and 0.25 also improve over base, showing that annealing is a useful way to control intervention strength rather than an absolute requirement for any gain.

Response to Reviewer kF6H (Part 3/3)

Q2. What are the training-time and memory costs?

The matched profiling protocol uses one NVIDIA L40S, global batch 8, the same precision and LoRA configuration, and 100 measured updates after 20 warm-up updates, repeated three times. We time negative generation and the training update, excluding data loading and diagnostic overhead for all methods.

MethodSeconds/update, mean +/- SDNegative generation, s/updatePeak allocated GiB
BC1.213 +/- 0.0040.0009.80
DITTO*3.475 +/- 0.0090.40717.36
VTLA*9.689 +/- 0.0286.60517.36
TS-CFG4.288 +/- 0.0081.21817.36

TS-CFG takes 1.23x the DITTO* time per update with the same peak allocated memory, and 3.53x the BC time. BC performs only the supervised update, whereas the preference methods also evaluate positive/negative pairs and the reference policy. VTLA* generates and ranks 16 candidates; TS-CFG constructs one demonstration-linked negative without candidate mining. There is no additional deployment-time cost: the post-trained policy uses the original sampler.

Q3. What happens with longer training or full-parameter fine-tuning?

Full-parameter fine-tuning improves performance at the same 5k-update budget: TS-CFG reaches 68.60%, versus 64.08% with LoRA. The matched comparison in W2 / Q4 above shows that DITTO* and VTLA* do not obtain a corresponding gain.

Longer training is a separate question. Following a single LoRA run beyond 5k, with the data, objective and learning rate unchanged, gives:

Training updates3k5k6k8k10k
RoboTwin Hard50 (%)59.5663.4062.3243.0822.24

Success peaks at 5k among these checkpoints and declines substantially after 6k. The logs show that BC loss increases even as preference loss continues to decrease. Our objective optimizes a relative preference margin without an auxiliary BC anchor, so improving that margin does not require preserving absolute demonstration-reconstruction accuracy. This is consistent with overoptimization documented for DPO and related direct-alignment methods (Rafailov et al., 2024). The results support a bounded post-training stage rather than indefinite preference optimization.

Note that this longer-training experiment uses an independent run, separate from the original TS-CFG run reported in the main table.

Comments on the paper

1 comment
Area_Chair_CKAtOpenReview ↗

Please review the authors’ rebuttal and update your assessment

Dear reviewers,

Thank you for your reviews, and thank you to the authors for their responses. Please read the authors’ rebuttal and discuss whether it addresses your main concerns. Please update your review and rating if warranted, and briefly explain your current assessment, including any concerns that remain unresolved. If your assessment is unchanged, please explain why.

Thank you for your continued engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.