acceptodds
Under review as a conference paper at ICLR 2027

Guidance under Shift: Deployment-Aligned Semantic Interfaces for Vision–Language–Action Policies

Abstract

Explicit chain-of-thought (CoT) can improve vision–language–action (VLA) policies by exposing task progress and spatial intent, but guidance available during training is often more reliable and temporally aligned than online predictions at deployment, which may be incorrect, missing, or outdated. This mismatch can make a policy trained with reliable guidance brittle at deployment. Our flow-matching analysis bounds the deployment loss by training error, guidance-distribution coverage, and conditional drift, motivating two training principles: covering deployment-time guidance conditions and exposing the policy to temporal staleness. We therefore train a VLA policy with subtask descriptions, affordance regions, and image-space trajectories drawn from annotated, generated, corrupted, and missing guidance, while varying their physical age—the elapsed time from the source observation to policy use. Current observations and action supervision remain aligned. An asynchronously refreshed dependency graph overlaps reasoning and action generation while preserving an explicit and inspectable CoT interface. Controlled evaluations vary guidance reliability and physical age. We evaluate closed-loop success on LIBERO and on training and testing tasks of RoboTwin 2.0, as well as a real-world “desktop-cleaning” task and its variants, demonstrating robustness and generalization on physical robots.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Embodied Reasoning, Semantic Guidance, Distribution Shift

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8G3tOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the mismatch between semantic guidance used to train vision-language-action policies and the guidance available during deployment, where predictions may be incorrect, missing, or temporally stale. It introduces a Perceiver–Worker framework: the Perceiver maintains task memory and predicts subtask descriptions, affordance regions, and image-space trajectories, while a flow-matching Worker conditions on current observations and the latest guidance, accompanied by field-specific physical ages. The Worker is trained using a mixture of annotated, predicted, perturbed, and missing guidance, with temporal sampling that preserves the alignment between current observations and demonstrated actions. An asynchronous cache allows guidance fields to refresh independently. The paper also derives a squared flow-matching risk decomposition and a transfer bound involving training error, input coverage, and conditional drift. Evaluation includes LIBERO, a training/held-out task split of RoboTwin 2.0, controlled guidance interventions on fixed demonstration states, and real-robot desktop-cleaning experiments with several scene variations.

Critical strengths
  1. The paper identifies a concrete deployment problem and develops a coherent, inspectable solution. The distinction between incorrect, missing, and stale guidance is useful for reasoning-conditioned robot policies. The proposed interface makes the source age of each guidance field explicit, preserves access to current observations, and allows action generation to continue while semantic predictions are refreshed. The training procedure is consistent with this design: offline predictions use causal observation histories, and temporal perturbations change the guidance while keeping the current action supervision aligned (Sections 4.2–4.4 and Appendix F.3). The analysis provides a useful conceptual account of when guidance can help or hurt. Equation (1) separates guidance information gain from the learned policy’s excess error, and Proposition 1 appears correct under its stated assumptions. The authors also correctly acknowledge that deterministic guidance can help a finite learned model without adding information to the Bayes-optimal predictor. Together, these elements make the work informative for designing practical semantic interfaces.
  2. The main simulation results provide credible evidence that the combined recipe improves closed-loop performance. The strongest evidence is the authors’ own simulation evaluation. Their observation-only π0.5 control follows the same continued-pretraining and downstream training stages, and the RoboTwin fine-tuning ablations share the general Worker checkpoint, Perceiver, data, and training budget (Appendix C). The method improves LIBERO mean success from 97.1% to 99.0%, including a gain from 93.5% to 97.8% on LIBERO-Long, and improves held-out RoboTwin success from 19.48% to 25.04% (Tables 3–4). The evaluation is substantial, with 500 rollouts per LIBERO task and 100 rollouts per RoboTwin task. Table 5 further supports the usefulness of the overall training recipe through closed-loop ablations. The real-robot experiments provide additional evidence of feasibility across changed objects, layouts, and partially completed tasks, although their sample size and baseline matching limit the strength of the conclusions drawn from them.
Critical weaknesses
  1. The controlled robustness evidence does not yet establish that the policy preserves useful guidance dependence under held-out, closed-loop deployment shifts. Table 6 is a useful sensitivity diagnostic, but Appendix E specifies that it uses 75 and 77 selected states from the Worker’s training demonstrations. The ERR intervention also reuses the corruption operators used during training. In these tests, all three guidance fields receive the same status, whereas the motivating deployment problem includes combinations of valid, incorrect, missing, and differently aged fields. Moreover, the full model has nearly identical action errors with correct, erroneous, and missing guidance in Table 6(a): 1.96, 1.97, and 1.99, respectively, in units of (10-2) rad. This demonstrates low sensitivity on those states, but leaves open how much useful online guidance the policy retains. It does not establish that the model ignores guidance; the relevant same-checkpoint, closed-loop comparison is missing. The paper does report held-out closed-loop success in Tables 4–5, which is valuable. However, those results do not isolate guidance usefulness or robustness under controlled shifts. A focused evaluation on held-out tasks, comparing the same full model with natural guidance, masked guidance, and mixed-field failures, would directly connect the proposed mechanism to deployment performance.
  2. The incremental contribution of mixed-source training is not fully isolated, and some supporting comparisons have different training histories. The GT-only and PRED-only ablations change more than the guidance-source mixture. Appendix C states that A1 and A4 also disable random emptying, while the full recipe includes artificial field dropout. Consequently, the comparisons establish the benefit of a package of changes, but do not isolate the contribution of GT/NOISE/ERR mixing beyond predicted guidance with the same dropout, age inputs, and temporal jitter. A matched PRED-plus-dropout control is particularly important given the related work on reasoning dropout and delayed-input training already acknowledged by the authors. The closest existing comparison is F versus A4, with held-out success of 25.04% versus 23.56%. The reported downstream setup specifies one training seed, and uncertainty for this difference is not reported. Paired task-level results and repeated runs for this pivotal comparison would clarify its reliability. Training-history matching also requires clarification outside the main simulation comparison. Appendix E explicitly states that the observation-only reference in Table 6 uses a different initialization and learning rate. For the real robot, Appendix D describes the proposed model as starting from the general Worker and the baseline as starting from pretrained π0.5. These details limit attribution of the supporting gains specifically to the proposed guidance-training recipe.
Questions for authors
  1. Targeted evidence: useful guidance under deployment shift. Can the same trained full model be evaluated on a representative subset of held-out tasks with natural Perceiver guidance, all guidance masked, and selected mixed-field failures or delays? Including naturally occurring predictor mistakes or a corruption type excluded from training would help establish whether the policy retains a measurable benefit from informative guidance while remaining robust to failures.
  2. Targeted evidence: source mixing beyond dropout. What is the performance of a PRED-only variant that retains the full model’s field-dropout probability, temporal jitter, age inputs, and training budget? How consistent is the full model’s advantage across held-out tasks and training seeds, especially given the 1.48 percentage-point gap between F and A4?
  3. Clarification and targeted evidence: the role of numerical age. How should the large A2 degradation in Table 5—25.04% to 17.48% held-out success—be reconciled with Appendix E’s finding that replacing a stale field’s true age with 0.2 s changes action error on the tested subtask states by at most 0.01? These interventions differ, so the findings need not conflict. However, same-checkpoint closed-loop results with true, hidden, or shuffled age tags would clarify whether numerical age controls reliance at inference or mainly affects training.
  4. Clarification: baseline initialization and continued pretraining. Did the real-robot π0.5 baseline receive the same continued-pretraining data and budget as the general Worker before the 6,000 adaptation updates? Please provide the checkpoint lineage and optimization settings for the main simulation baseline, the Table 6 reference, and the real-robot baseline. If their histories differ, which comparisons should be interpreted as tests of the overall system, and which isolate the guidance-training procedure?
  5. Clarification: measured training–deployment alignment. How were the source probabilities and delay ranges selected, and what are the effective per-field age, missingness, and failure distributions during deployment? Appendix D reports ages from one successful Base rollout sampled every 10 seconds; broader measurements would better support the alignment claim. Please distinguish actual PRED ages from nominal lookback ranges, and report the Worker query cadence and Perceiver/Worker latency statistics on the shared GPU.
4 comments
AuthorsOpenReview ↗

Reply to Reviewer 8G3t (1/4)

Thank you for your careful review and for recognizing our treatment of a practical deployment problem and the consistency between the semantic interface and training design. We also appreciate your recognition of the conceptual value of the analysis, the evidence from the main simulation experiments for the combined recipe, and the substantial evaluation effort. Below, we address your questions in turn.

R1: The role of guidance under deployment variation (W1, Q1)

Thank you for suggesting that the same full model be compared on held-out tasks under natural guidance, masked guidance, and mixed-field failures. Following this suggestion, we fixed the F checkpoint and ran closed-loop comparisons, with the all-masked result as the direct evidence of the closed-loop benefit of guidance.

The natural online baseline already retains the Perceiver's prediction errors, missing fields, and the ages produced by asynchronous execution, with no additional artificial injection; the interventions below change this input in a controlled way. F is trained with ERR corruptions and random field dropout (Appendix F.5); the comparison below tests what F does with guidance at inference.

1.1 Setup

We fix the F checkpoint and evaluate on the 25 training tasks (Train) and 25 held-out tasks (Test) of RoboTwin 2.0, 100 scenes per task, 2,500 rollouts per cell, seed 42. The three inference-time interventions are:

  • All guidance masked: the Worker input receives no guidance.

  • Additional training-style errors: at each read, each present subtask, affordance, and trajectory field is selected for the ERR operators of Appendix E with probability 12.5%, 8%, and 8%, taken from the ERR source weights of F's training recipe before dropout. These are selection probabilities applied on top of PRED's own natural errors, not the realized error rate of the input.

  • Corruptions not used in training: each present field is replaced with probability 30%; the subtask is replaced by another task's subtask, and the box and trajectory are mirrored horizontally.

Injection is independent per field and applied on top of the natural Perceiver output: fields that are not replaced keep their natural output, and fields that were already missing stay missing. These per-field interventions allow combinations of correct, wrong, and missing fields to occur; these closed-loop tests keep the natural field ages, and the fixed-state 3 s staleness test is discussed in R3.

1.2 Closed-loop results

ConditionF Train (%)Drop (pp)F Test (%)Drop (pp)
Natural online guidance89.12—25.04—
All guidance masked85.483.6419.405.64
Injected training-style errors (subtask / box / trajectory per field 12.5% / 8% / 8%)87.321.8023.042.00
Injected corruptions not used in training (per field 30%)87.042.0820.204.84

The drop is the decrease in success relative to natural online guidance, in percentage points (pp).

Masking all guidance lowers F's Train / Test success by 3.64 / 5.64 percentage points, so the same checkpoint obtains a measurable closed-loop benefit from online guidance. In this evaluation the drop is larger on the held-out tasks, where Test success falls from 25.04% to 19.40%.

Under the two tested error interventions, Train / Test success falls by 1.80 / 2.00 percentage points for training-style errors and by 2.08 / 4.84 percentage points for corruptions not used in training; in all four cells success remains above that of full masking. This result applies to the intervention types and rates tested here.

AuthorsOpenReview ↗

Reply to Reviewer 8G3t (2/4)

R2: The role of source mixing beyond dropout (W2, Q2)

Thank you for suggesting a PRED-only control that keeps the same dropout to isolate the contribution of source mixing.

2.1 Matched control

In the paper, A4 also disables random dropout, so the comparison between F and A4 indeed cannot separate this contribution.

Following your suggestion, we trained PRED + dropout: guidance comes only from PRED and every other setting matches F, including initialization, Perceiver, data, the 5,000-step training budget, the per-field dropout probability of 0.20, temporal jitter, and numerical age. The control differs from F only in the GT, NOISE, and ERR sources.

2.2 Closed-loop results

RecipeSourcesDropout per fieldTrain success (%)Test success (%)
F (paper Table 4)GT + NOISE + PRED + ERR0.2089.1225.04
PRED + dropout control (newly trained here)PRED only0.2087.1622.84
A4 (paper Table 5)PRED only087.4823.56

Under the same dropout rate, temporal jitter, and age input, F's Test success is 2.20 percentage points above the PRED + dropout control (1.96 on Train), so source mixing gives an additional gain beyond dropout in this matched comparison. The A4 row differs from F in both sources and dropout, which is why the matched control was needed.

The comparisons above use training seed 42. We commit to reporting the mean and standard deviation over three training seeds for all ablation experiments in the final version (see the reply to all reviewers).

R3: The role of numerical age (Q3)

Thank you for pointing out that the large A2 degradation and the small change in Appendix E need to be reconciled, and for suggesting a comparison of age tags on the same checkpoint. We first explain how the two differ and then report the added closed-loop test. A2 and the Appendix E test apply different interventions and measure different quantities.

3.1 Experimental settings

A2 replaces the numerical age with present in both training and deployment, so it changes the trained model and the prompt format together and is measured by closed-loop success.

Appendix E fixes the trained models and the demonstration state: on states just after a subtask switch, only the subtask is made stale while the box and trajectory stay fresh, and the subtask's true age is replaced by 0.2 s; for F and A3, two separately trained and frozen models, the action error changes by at most 0.01 (×10⁻² rad). This comparison measures the change in error relative to demonstration actions; the predicted actions themselves may still change.

Table 6(b) is a separate protocol in which all three fields are stale together. The original manuscript did not distinguish these settings, and we apologize for it; the revision states them in Sections 5.1.2 and 5.1.3 of the paper and in Appendix E.

To test the role of age at inference on the same checkpoint, we fix F's weights and change only the age text in the Worker input, while the age-cap check and scheduling still use the true age. The two variants are: (1) hidden age: the age of each present field is written as the word present, which changes the representation;

(2) random age: at each read, the age of each present field is drawn independently and uniformly from 0.1–2.0 s, which keeps the numerical format and perturbs only the value.

3.2 Closed-loop results

Age the Worker sees (F, Test)Test success (%)
True age (paper Table 4)25.04
Hidden age (present)18.28
Random age (uniform 0.1–2.0 s)23.20

Test success under true age is 1.84 percentage points above random age, so the age value itself affects closed-loop performance of this checkpoint. Under hidden age success falls to 18.28%, a condition that changes the age information and the prompt format together.

AuthorsOpenReview ↗

Reply to Reviewer 8G3t (3/4)

R4: Baseline initialization and continued pretraining (Q4)

Thank you for asking us to state the lineage and optimization settings of each baseline, and which comparisons test the whole system versus the training recipe.

Comparisons among F, A1–A4, and the PRED + dropout control isolate the guidance training recipe: these models share one general Worker checkpoint and differ only in downstream guidance training. Comparisons with observation-only π0.5 (the RoboTwin main table, the Table 6 reference, and the real-robot baseline) test the whole system, because the baseline's continued pretraining has no guidance.

ModelContinued pretraining (DROID + AgiBot World)Downstream trainingDownstream optimization (AdamW, cosine decay)What this comparison tests
F, A1–A4, PRED + dropout controlwith guidance, sharing one general WorkerRoboTwin training tasks, 5,000 stepspeak learning rate 5×10⁻⁵, 250 warmup steps, decay to 2.5×10⁻⁷the guidance training recipe itself
RoboTwin observation-only π0.5 (revised Table 4, also the reference for Table 6)without guidanceRoboTwin training tasks, 5,000 stepspeak learning rate 5×10⁻⁵, 250 warmup steps, decay to 2.5×10⁻⁷the whole system
Real-robot GoT-VLAwith guidance, general Worker866 real-robot demonstrations, 30,000-step schedule, evaluated at step 6,000peak learning rate 5×10⁻⁵, 1,500 warmup steps, decay to 2.5×10⁻⁷the whole system
Real-robot observation-only π0.5without guidance866 real-robot demonstrations, 30,000-step schedule, evaluated at step 6,000peak learning rate 5×10⁻⁵, 1,500 warmup steps, decay to 2.5×10⁻⁷the whole system

Every observation-only π0.5 baseline starts from official π0.5, undergoes continued pretraining on DROID and AgiBot World without guidance, and is then fine-tuned downstream; our method performs continued pretraining with guidance on the same data. "Starting from pretrained π0.5" in Appendix D and "different initialization" in Appendix E both refer to this guidance-free continued-pretraining checkpoint. The initial version did not state the baseline lineage clearly, and we apologize for it; the revision states it in the Table 6 caption and in Appendices D and E.

The "different learning rate" mentioned in the initial Appendix E refers to the RoboTwin observation-only baseline using the fine-tuning learning rate recommended by the official π0.5 release at that time. We retrained and re-evaluated that baseline with the same downstream optimization settings as F: Train success 77.44%, Test 19.56%, versus 77.28% and 19.48% in the initial Table 4. The revision adopts these results (see the reply to all reviewers) and the table above lists this setting; the observation-only references in Table 6 are recomputed accordingly, with the Missing column changing from 3.10, 2.27 to 2.95, 2.21, and the conclusion unchanged.

On the real robot the two models use identical optimization settings. The training loss had largely stabilized by step 6,000; given our compute constraints, we evaluated the checkpoint at that step. The real-robot baseline uses the same schedule and is likewise evaluated at step 6,000. The RoboTwin downstream optimization setting (including the observation-only π0.5 baseline) is given in the "Downstream optimization" paragraph of Appendix F.5, and the real-robot schedule and evaluated checkpoint are given in Appendix D.

AuthorsOpenReview ↗

Reply to Reviewer 8G3t (4/4)

R5: Source weights and the guidance distribution at deployment (Q5)

Thank you for asking us to distinguish the nominal training lookback from the actual PRED ages and to report measured deployment statistics. We explain the training settings below and compiled a real-robot deployment record.

The source weights, field dropout rates, and training lookback ranges of each benchmark were set empirically before the held-out evaluation and then held fixed; the values are in Appendix F.5, and R2 tests the recipe under these settings.

The nominal lookback is a training-time sampling setting. For each field, training samples a historical time in physical time within the benchmark's range (0.12–0.66 s on RoboTwin, 0.1–1.0 s for the real-robot Worker). Annotated and perturbed fields take this sampled age, whereas a PRED field carries the source time of the selected prediction record, so its age can be older than the sampled lookback (Appendix F.5).

Deployment involves no lookback sampling: each field's age is computed from its true source time and the time it is read.

For the distribution at deployment, we compiled a real-robot deployment record: the Perceiver and the Worker share one RTX 5090, a client queried the Worker at the deployment cadence of one query per 25 executed steps, and the record covers the responses of the per-field request loops and the 53 Worker reads over 60 consecutive seconds.

LoopCompleted (rate)Latency (s)Age (s)Used / missing
Memory5 (one per window)0.90 / 1.12——
Subtask112 (1.87 Hz)0.47 / 0.700.70 / 0.87 / 1.4353 / 0
Affordance116 (1.93 Hz)0.52 / 0.740.97 / 1.30 / 1.6752 / 1
Trajectory116 (1.93 Hz)0.86 / 1.180.97 / 1.30 / 1.6751 / 2
Worker53 queries (0.88 Hz)0.27 / 0.66——

Completed is the number of responses that loop returned during the 60 s recording, and the rate is that number divided by 60 s. Latency is the time from request submission to return (p50 / max).

The last two columns are computed over the 53 Worker queries rather than over the whole recording: age is the value of that field at those query times (p50 / p90 / max), and used / missing counts the queries in which the field was provided to the Worker or was blanked, so the two entries in each row sum to 53. Memory is not an input to the Worker.

In this recording, the subtask (1.87 Hz) and the two geometry fields (1.93 Hz) refreshed faster than the Worker queried (0.88 Hz), while memory produced one summary per completed 10 s window.

Across these 53 reads, the median age was 0.70 s for subtask and 0.97 s for affordance and trajectory, with a maximum of 1.67 s; the three fields were missing in 0, 1, and 2 reads.

This table reports age, missingness, refresh rate, and latency; it does not measure whether field contents are correct. "Used" means the field was present, not that it was correct, and "missing" does not cover all failures.

Online states have no frame-level semantic labels, so we have not measured an online error rate; the Perceiver's prediction-error statistics on training and held-out demonstrations are in our reply to Reviewer aYEZ, and they come from an evaluation on demonstrations. The revision adds this table to Appendix F.6 and states in Appendix D that the earlier age description comes from one successful trial.

Due to the time and computational resource constraints during the rebuttal period, we prioritized covering a broader range of experimental settings rather than running multiple seeds for each individual experiment. Nevertheless, we commit to reporting the mean and standard deviation across three independent training seeds for all simulation experiments in the final version.

Updated on November 18, 20:40 AoE: We have updated the 3-seed experiments (including the means and standard deviations) for some experimental settings. The results are presented on page 3 of the Common Response.

Reviewer_aYEZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the mismatch between reliable training-time guidance and imperfect online guidance in vision-language-action policies. It decomposes the benefit of guidance under squared flow-matching loss and bounds guided deployment excess risk using training excess risk, input coverage, and conditional drift. The proposed Perceiver-Worker system trains the action policy with annotated, predicted, perturbed, and missing guidance, with independently sampled historical content and explicit field ages. Current observations and action targets remain aligned, and guidance is refreshed asynchronously at inference. Evaluation includes LIBERO, a held-out-task split of RoboTwin 2.0, controlled guidance interventions on fixed states, and real-robot desktop cleaning.

Critical strengths
  1. The paper addresses a concrete deployment problem with a coherent, implementable training recipe. Preserving current visual observations while varying guidance content, availability, and physical age is a useful design principle. The analysis distinguishes information gain from learned-policy excess risk and clarifies why fitting a clean training distribution alone need not resolve guidance shift. This makes the work informative even though several individual components have precedents.

  2. The controlled RoboTwin comparisons provide meaningful evidence for the recipe. The variants share the general Worker checkpoint, Perceiver, downstream data, and training budget. The full method improves held-out-task success over both GT-only and Pred-only fine-tuning, while paired fixed-state tests show substantially reduced degradation under wrong and stale guidance. The small fresh-guidance accuracy cost of temporal jitter also reveals a useful robustness trade-off. These results, together with closed-loop simulation and real-robot demonstrations, support the method's practical value.

Critical weaknesses
  1. The central unresolved issue is robustness to correlated guidance errors that remain internally consistent. An incorrect subtask can induce a plausible affordance and trajectory for the same wrong operation, whereas the reported ERR interventions do not establish such coherence. Since the transfer bound concerns the joint input distribution, coverage of individual field errors alone does not establish adequate coverage of failures caused by a shared upstream mistake. This leaves a specific gap between the controlled robustness evidence and the deployment-alignment claim.

  2. How the Worker handles errors that propagate through the Perceiver. A mistaken subtask can produce a plausible target box and trajectory for the wrong operation. The ERR tests in Table 6 use swapped subtasks and displaced geometry, leaving the Worker's response to internally consistent errors unclear. PRED may already contain such cases. The paper gives little detail on their frequency or whether independently sampling fields and source times preserves their correlations. This matters for the deployment-alignment claim, since Proposition 1 depends on coverage of the joint input distribution.

  3. In Table 6(a), the full model has almost identical errors with correct, wrong, and missing guidance (1.96, 1.97, and 1.99 in the reported units). The small differences make it difficult to judge how much online guidance contributes to task performance. Comparing the same checkpoint with and without guidance on held-out tasks would clarify how much of the improvement depends on the Perceiver at inference time.

Questions for authors
  1. Does PRED contain cases where the subtask and geometry agree on the same wrong operation? How common are they, and does independent sampling of field sources and times preserve these relationships? Examples from the existing logs would help.

  2. Could you compare correct, missing, and internally consistent incorrect guidance using the same checkpoint and comparable guidance ages? A small closed-loop test with actual Perceiver errors would be useful, using cases where the instruction or current image makes the correct target clear.

  3. How much does held-out task success change when the full Worker runs with all guidance missing? A paired comparison on the same scene seeds would help quantify the benefit of online guidance.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer aYEZ (1/3)

Thank you for your careful review and for recognizing the value of preserving current observations while varying the content, availability, and age of guidance. We also appreciate your positive assessment of the evidence from the controlled RoboTwin comparisons and the practical value of the training recipe. Below, we address your questions in turn.

R1: Robustness to correlated guidance errors (W1, W2, Q1)

Thank you for pointing out internally consistent errors caused by a shared upstream mistake, and the question of whether independent sampling preserves the relations between fields. Such errors do exist.

PRED generates in causal order: the affordance and 2D trajectory of the current frame are conditioned on the subtask generated before them, so when the subtask is wrong, the error can propagate into the geometry fields. To address this, we quantify in offline evaluation how often such errors occur and how much of this kind of input the training recipe gives the Worker, examine how independent sampling affects the generation relation, and show an example.

1.1 Occurrence rate in offline evaluation

We freeze the Perceiver in offline evaluation, generate PRED in the causal generation order used at deployment, and compare each generated triple field by field with the annotation, on 1,250 training demonstrations and on 250 demonstrations from the 25 held-out tasks. This evaluation uses demonstrations only and does not represent the distribution at deployment:

Error type (share of all evaluated guidance triples; 60,502 training triples, 7,949 held-out triples)Training demonstrationsHeld-out
Subtask error3.0%16.1%
All three fields wrong0.15%12.4%
of which: internally consistent error0.05%2.0%

The judgment uses the annotation of the frame the Worker reads as reference: guidance that disagrees with the current operation counts as a subtask error, including time-offset errors that lag the current frame and hallucinated errors that also disagree with the current operation.

Judgments are made independently by models that take no part in training: the subtask is judged by Qwen3.8-27B against the annotation and the predicted value, and for all evaluated triples, Doubao-Seed-2.1-Pro separately judges whether the affordance and the 2D trajectory endpoint point to the result or object described by the subtask. An all-three-wrong triple whose geometry points to that wrong subtask is one where the wrong subtask has led the geometry to the wrong object, which is an internally consistent error.

All-three-wrong means: the subtask is wrong; the IoU between the affordance and the annotated box is below 0.5; and the trajectory has inconsistent arm state or a discrete Fréchet distance above 50 pixels (images 320×240).

These results support two observations. First, upstream errors are five times more frequent on held-out data than on training demonstrations (16.1% versus 3.0%), and this is the quantity that most needs coverage. Second, all-three-wrong triples account for 12.4%; within these fully wrong inputs, the geometry actually points to the wrong subtask in about 16% of cases, and in the remaining approximately 84% the geometry does not itself agree with that wrong upstream subtask. Internally consistent errors account for 2.0% of all evaluated triples.

1.2 Exposure to internally consistent errors during training

Internally consistent errors make up only 0.05% of PRED on training demonstrations, so a Worker trained on PRED alone would rarely see such inputs.

We generate 20,000 loader items following F's training recipe, with source sampling, the ERR operators, random dropout, temporal jitter, and the age pathway running as usual, and evaluate each item with the same adjudication models and thresholds as the table above: internally consistent errors account for 1.5% of all items. These inputs are constructed by the training recipe, and their frequency is of the same order as the 2.0% the Perceiver produces on held-out demonstrations.

Both rates use all judged items as the denominator and the same adjudication rules, so they can be compared as occurrence frequencies: training exposes the Worker to such inputs at a frequency close to that on unseen tasks. The comparison does not concern the distribution of error content or field age.

AuthorsOpenReview ↗

Reply to Reviewer aYEZ (2/3)

1.3 Overall closed-loop behavior

The online condition uses the same Perceiver. Row (1) of the R2 closed-loop results table below is measured under natural online input that includes the Perceiver's actual errors and reflects F's overall behavior: F reaches 25.04%, falling to 19.40% when all guidance is removed.

1.4 How independent sampling affects the generation relation

In F's training loader, among samples in which all three fields come from PRED, 85.8% have at least one geometry field generated from exactly the subtask the Worker reads, so the Worker sees the true generation relation; in the remaining 14.2%, the geometry was generated under a different subtask, so the correspondence is deliberately broken.

AuthorsOpenReview ↗

Reply to Reviewer aYEZ (3/3)

R2: Closed-loop comparison of the same checkpoint under different guidance (W3, Q2, Q3)

2.1 Technical limitations of the requested comparison

Thank you for suggesting a same-checkpoint comparison of correct, missing, and internally consistent incorrect guidance at comparable ages. Our existing error injections use rule-based text replacements and geometric perturbations applied to online guidance.

Injecting correct or internally consistent incorrect guidance requires using the current observation and task progress to select an operation and target and jointly construct the corresponding subtask, affordance box, and trajectory. Correct guidance must support an appropriate operation for the current state, whereas internally consistent incorrect guidance must make all three fields support the same wrong operation. Our current rule-based injection procedure cannot jointly generate such guidance on demand as the rollout state changes, so we cannot reliably carry out this specific comparison at comparable guidance ages.

2.2 Approximate experimental setting

Nevertheless, to address your concern as fully as possible, we added feasible approximations to the proposed experimental setting and provide complementary evidence in two parts. The offline statistics and example in R1 show whether such errors occur and whether the training recipe exposes the Worker to them. Below, we fix the same F checkpoint, tasks, and scene seeds, and compare natural online guidance, full masking, and two per-field error injections to measure the overall closed-loop benefit of guidance in the presence of natural prediction errors and performance under controlled corruption. The two injections preserve natural field ages and leave already missing fields missing; they are not designed to construct internally consistent errors. Row (2) directly addresses the comparison with all guidance masked requested in W3 and Q3.

The two injections are sampled with a fixed random seed and superimposed on the natural output: the injection positions are the same across evaluations for the same task, scene seed, and decision step; where a field is itself missing, the perturbation at that position has no effect.

  1. Natural online guidance: all guidance comes from the online Perceiver, that is, the official results of Table 4 of the paper.

  2. All guidance masked: using F's Worker checkpoint with no guidance at all; subtask text, geometry overlays, and age markers are all left empty.

  3. Injected training-style errors: at each read, the present subtask, affordance, and trajectory are selected for the ERR operators of Appendix E with probabilities 12.5%, 8%, and 8% respectively; these probabilities are taken from the ERR source weight in F's training recipe before dropout, and they are selection probabilities applied on top of PRED's own natural errors, not the realized error rate of the input.

  4. Injected corruptions not used in training: at each read, each present field is independently corrupted with probability 30%, with the subtask replaced by another task's subtask and the affordance box and trajectory mirrored horizontally.

2.3 Closed-loop results

F (seed 42), 25 tasks × 100 scene seeds per cellTrain (25 training tasks)Test (25 held-out tasks)
(1) Natural online guidance89.1225.04
(2) All guidance masked85.4819.40
(3) Injected training-style errors (subtask / box / trajectory per field 12.5% / 8% / 8%)87.3223.04
(4) Injected corruptions not used in training (per field 30%)87.0420.20
Reference: observation-only π0.5\pi_{0.5}77.4419.56

Row (1) is taken from Table 4 of the paper; observation-only π0.5\pi_{0.5} is the result aligned with F's optimization settings, see the reply to all reviewers.

Removing online guidance from the same F checkpoint lowers Test success from 25.04% to 19.40%, showing that natural online guidance provides an overall benefit even in the presence of actual prediction errors. Test success under the two injections is 23.04% and 20.20%, both above the 19.40% of full masking.

These results measure overall performance under natural online input and the tested corruptions; closed-loop success specifically under internally consistent errors at comparable ages remains unmeasured. The Worker is a learned black-box policy whose response depends on the current observation and specific guidance; the error category alone does not determine its behavior.

Due to the time and computational resource constraints during the rebuttal period, we prioritized covering a broader range of experimental settings rather than running multiple seeds for each individual experiment. Nevertheless, we commit to reporting the mean and standard deviation across three independent training seeds for all simulation experiments in the final version.

Updated on November 18, 20:40 AoE: We have updated the 3-seed experiments (including the means and standard deviations) for some experimental settings. The results are presented on page 3 of the Common Response.

Reviewer_MbVvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses the train/deploy mismatch for chain-of-thought-guided VLA policies: the action policy is trained on hindsight-annotated guidance but deployed on online predictions that can be wrong, missing, or stale. It contributes 1/ a flow-matching decomposition and a covariate-shift bound expressing the guided policy’s deployment excess risk through training excess risk, a density-ratio coverage coefficient, and conditional drift; 2/ a two-model system: a PERCEIVER VLM that predicts memory, subtask, affordance boxes, and a 2D trajectory, and a pi-0.5-based WORKER fine-tuned on a mixture of annotated, perceiver-generated, corrupted, and missing guidance with per-field physical ages and temporal jitter, run with asynchronous per-field refresh.

Critical strengths

The problem is real and the paper’s organizing distinction: coverage of deployment guidance inputs versus drift in the guidance to action relation, maps cleanly onto two different training choices (corruption/dropout/jitter versus perceiver-generated guidance). The stale-guidance experiment is a fair extrapolation test. The evaluation effort is broad (two simulators, a real robot) and the appendix is unusually complete on prompts, source weights, splits, and the annotation pipeline.

Critical weaknesses

1/ The analysis contribution is a definition plus a textbook bound, and the notation hides its scope.

Claimed (Contribution 1, also in the abstract): “Guidance helps a learned policy only when the information it adds exceeds the error of using it; a bound relates this error to training fit, coverage, and drift.” But this derivation is trivial: Eq. (1) is the Pythagorean identity for conditional means under squared loss with its two terms named (A.2); it cannot fail and is not a finding. Prop. 1 is the triangle inequality plus an L-inf change of measure: the standard covariate-shift bound, and Cor. 1 follows in one line. None of the variables is ever estimated; beta is an L-inf density ratio over continuous boxes, trajectories, and ages, so the “sufficient condition” C_U < G + e_X is not checkable even in principle, and the bound says nothing about mixture weights, which are tuned per benchmark (GT weight 0.25 on LIBERO, 0.125 on RoboTwin, 0.10 on the robot). The setup (section 3) requires training and deployment to share the law of (X, A), so the analysis applies only to training-demonstration states: exactly the fixed-state test of section 5.1.3, and is silent on every transfer result the paper is built around; WORKER pretraining is not modeled at all. The two “training principles” it motivates are already in TIC-VLA, CloudEdgeVLA, and ERVLA. Section 3 is a three-sentence motivation written as a theorem.

2/ The method’s novelty is combinational, and it is a weaker version of the technique it most resembles.

Claimed (Contribution 2): training the WORKER on “annotated, generated, perturbed, and missing guidance tagged with its age, matching what it receives at deployment.” Every ingredient has precedent: delayed inputs with current action targets (TIC-VLA, CloudEdgeVLA), reasoning dropout (ERVLA), synthetic input corruption (DART-style noise injection), and training a downstream module on upstream predictions. The combination does not deliver the deployment match it claims. PRED is generated by the PERCEIVER on the same demonstrations it was fine-tuned on, so the WORKER trains on in-sample predictions that understate online error, and that gap is never measured. Because the WORKER is never run during data collection, guidance produced under its own induced states: replanning after a failed grasp, real asynchronous timing, is absent from training, even though Fig. 4 presents exactly that case as a highlight. The paper’s own logs show geometry arriving 1.0–2.1 s old on the robot against a trained range of 0.1–1.0 s. A closed-loop, on-policy collection would address all three and is feasible here (RoboTwin provides scripted experts), but is not attempted. The one ingredient specific to this setting, the numeric age tag, is undercut by the paper’s own evidence: hiding it costs 7.56 pp on Test (A2, Tab. 5), yet App. E reports that replacing the true age with 0.2 s changes action error by at most 0.01. The interface (subtask as text, geometry as an image overlay, in the style of TraceVLA/RT-Affordance) and the four-field decomposition are never ablated, so there is no evidence of which field does any work.

Minor: “GT” guidance is hindsight VLM annotation with boxes forward-filled across each phase (F.1); annotation quality is not reported, so “correct guidance” throughout is relative to unmeasured label noise. \beta is defined differently in Tab. 1 and Eq. (11).

Questions for authors

Q1 (Section 3). Eq. (1) is the Pythagorean identity for conditional means with its terms named, and Prop. 1 is the standard covariate-shift bound (triangle inequality plus an L-inf change of measure). None of the variables are estimated, \beta cannot be computed over continuous boxes, trajectories, and ages, and the setup assumes training and deployment share the law of (X, A), so the result applies only to training-demonstration states. What does Section 3 predict in advance, and which experiment tests that prediction? If every result in Section 5 would have been equally consistent with the analysis, please clarify that its role is motivation rather than a result.

Q2 (novelty). By the paper’s own account, each component has a precedent: training on delayed guidance with current action targets (TIC-VLA, CloudEdgeVLA), dropping or corrupting reasoning inputs (ERVLA), asynchronous semantic buffers feeding a fast controller (AsyncVLA, DuoCore-FS), subtask language from a VLM (π0.5), and geometry rendered onto the image (TraceVLA, RT-Affordance). Please state precisely what this work adds beyond assembling these. If the answer is the per-field physical age tag, note that App. E finds the WORKER’s output changes by ≤0.01 when the age value is altered, so the one new ingredient appears unused. If the answer is the combination itself, please provide an experiment that isolates it: a baseline that applies the existing pieces (delay training + dropout + async refresh) without your additions, so readers can see what the combination contributes over its parts.

Q3 (ablations). Several claims rest on ablations that are not in the paper. Please report, on RoboTwin Train and Test with at least three seeds: (a) F’s own WORKER run closed-loop with all guidance fields empty, to show how much of 89.12/25.04 is due to guidance at all; (b) subtask only, geometry only, and no memory, to show which field contributes; (c) closed-loop success, not open-loop RMSE, under wrong, stale, and missing guidance, including corruptions not used in training; and (d) the PERCEIVER’s error on the training demonstrations used for PRED versus on closed-loop rollouts, to substantiate “matching what it receives at deployment.”

6 comments
AuthorsOpenReview ↗

Reply to Reviewer MbVv (1/6)

Thank you for your careful review and for recognizing the practical deployment problem we study, the distinction between input coverage and conditional drift, and the breadth of our simulation and real-robot evaluations. We also appreciate your careful attention to and positive assessment of the detailed prompts, source weights, data splits, and annotation pipeline in the appendix. Below, we address your questions in turn.

R1: How the risk decomposition guides training design (W1, Q1)

Thank you for asking what Section 3 predicts in advance and which experiment tests it, and for identifying, item by item, how these derivations relate to standard tools. Below we explain the role of Section 3 in the training design and list the recipe hypotheses it raises together with the corresponding controls.

Eq. (1) rests on the projection property of the conditional mean under squared loss, and Prop. 1 follows the standard route for covariate shift; we do not present them in the paper as new inequalities. The role of Section 3 is to split the broad question "does guidance help at deployment" into two parts — the information gain from guidance, and the error incurred by using it — and then to bound the deployment excess risk through training excess risk, input coverage, and conditional drift. These three quantities inform different training choices, which makes the recipe hypotheses testable by controls.

1.1 Sufficient condition and the three terms

The sufficient condition we obtain is

CU=(βϵtr+Γte)2<G+eX.C_U=\left(\sqrt{\beta\epsilon_{\mathrm{tr}}}+\sqrt{\Gamma_{\mathrm{te}}}\right)^2<G+e_X.

Here ϵtr\epsilon_{\mathrm{tr}} measures training excess risk, β\beta measures how well the training distribution covers deployment inputs, and Γte\Gamma_{\mathrm{te}} measures the difference in the guidance-to-action conditional relation between training and deployment; the left side CUC_U bounds the deployment excess risk eU,tee_{U,\mathrm{te}} when guidance is used, GG is the information gain of guidance over observation alone, and eXe_X is the deployment excess risk of the observation-only policy itself. All of these quantities are defined under the squared flow-matching risk.

Appendix A.4 explains that each source can affect multiple terms: wrong and missing guidance change the training distribution and are included primarily to improve coverage; the PRED source carries outputs of the deployed predictor and is included primarily to reduce conditional drift; temporal jitter changes both the content and the age of guidance, and can therefore affect coverage, conditional drift, and training difficulty at once. The controls below test "whether a training choice acts in its intended direction"; they do not confine any source to a single risk term.

1.2 What the population training optimum implies

Under the coverage condition of Cor. 1, training that reaches the population optimum gives eU,te=Γtee_{U,\mathrm{te}}=\Gamma_{\mathrm{te}}: once the fit term vanishes, deployment excess risk is determined by conditional drift. Hence, with the population distributions and the guidance-generating mechanism of training and deployment fixed, adding independent demonstrations from the same distribution can reduce the error due to finite samples, but does not change β\beta or Γ\Gamma; training on multiple sources therefore has a definite target.

1.3 When coverage fails

Write qc=pte(c)q_c=p_{\mathrm{te}}(c) and wc=ptr(c)w_c=p_{\mathrm{tr}}(c), where cc ranges over the eight patterns in which the three fields are present or missing. If every pattern that occurs at deployment satisfies wc>0w_c>0, and the conditional distribution within a pattern is absolutely continuous, then β=max⁡c:qc>0qcwcβc\beta=\max_{c:q_c>0}\frac{q_c}{w_c}\beta_c, where βc\beta_c is the essential supremum of the conditional density ratio for the complete input (S,U)(S,U) in that pattern. If some pattern of positive deployment mass satisfies wc=0w_c=0, then β\beta is infinite and the finite-coverage condition fails.

Random dropout acts by assigning training probability to missing patterns, making wc>0w_c>0 for those patterns; ERR mainly changes the content distribution within present patterns. This gives the design rationale for increasing training exposure to missing and wrong inputs.

1.4 Weights are set empirically, and controls test the recipe

The source weights, dropout rates, and lookback ranges of each benchmark are set empirically (Section 3.3 of the paper, Appendix F.5). A single source changes both the pattern mass wcw_c and the training conditional mean given the complete input (S,U)(S,U), and therefore affects coverage, conditional drift, and training difficulty at once; the optimal weights do not come from solving this inequality. The effect of the recipe under these settings is tested by controlled comparisons.

AuthorsOpenReview ↗

Reply to Reviewer MbVv (2/6)

1.5 The experiments test the recipe hypotheses

For the empirical content of Q1 we test three things: whether temporal jitter improves tolerance to stale guidance, whether the complete recipe with wrong and missing training exposure is less sensitive to the corresponding inputs, and whether multi-source training still helps under common settings. On set (b) with fixed demonstration states (23 tasks, 77 states), 3 s stale guidance raises F's action error by 0.11 and A3's (extra temporal jitter removed) by 0.42 (×10⁻² rad), a difference in error increases of 0.31, with 21 of 23 tasks in the same direction; the observation-only reference for set (b) is 2.21.

On set (a) (25 tasks, 75 states), A1 gives no weight to the error and missing sources during fine-tuning, and after injecting ERR and MISSING its error rises by 0.88 and 0.67, versus +0.01 and +0.03 for F.

Recipe hypothesisControl and measurementAvailable result
Temporal jitter improves tolerance to stale guidanceF versus A3, set (b), injected 3 s stale guidanceincreases of +0.11 and +0.42 (×10⁻² rad), difference in error increases 0.31, 21/23 tasks in the same direction
The complete recipe with wrong and missing training exposure is less sensitive to the corresponding inputsFull recipe F versus A1 (GT only) and A4 (PRED only), set (a), injected ERR and MISSINGincreases: A1 +0.88 / +0.67, A4 +0.43 / +0.36, F +0.01 / +0.03
Under common jitter, dropout, and age settings, multi-source training still helpsF versus the PRED + dropout control, closed-loop successTrain 89.12% versus 87.16%, Test 25.04% versus 22.84%, see R2

1.6 Scope of the state distribution and of pretraining

The paper assumes that training and deployment share the distribution of (X,A,ξ,τ)(X,A,\xi,\tau), so that the change in guidance can be analyzed in isolation; that distribution can be any state distribution satisfying the assumptions, and is not mathematically restricted to a particular set of training demonstrations.

Appendix A.1 shows that, under second-moment and coverage conditions, the in-domain decomposition and the change-of-measure relation also hold for different state distributions, in which case the coverage coefficient absorbs changes in both the state and the guidance distributions.

Transfer performance is measured by held-out closed-loop evaluation. Worker pretraining enters the analysis through the policy it finally learns and that policy's error; the recipe controls share the same general Worker initialization, which is an experimental control condition.

In addition, β\beta appears in Tab. 1 and in Eq. (11) in the measure-domination and density-ratio forms respectively; the two are equivalent under absolute continuity and otherwise both are infinite. The revision states this equivalence in Section 3.2 of the paper, and repeats it after Eq. (11) in Appendix A.3.

1.7 Role of Section 3

We agree that ϵtr\epsilon_{\mathrm{tr}}, β\beta, and Γte\Gamma_{\mathrm{te}} cannot be reliably estimated term by term in our setting, so the sufficient condition is not checked numerically. Eq. (1), Prop. 1, and Cor. 1 are mathematical results under the stated assumptions; in this paper they guide and explain the training design: they separate the information gain of guidance from the error of using it, take the observation-only policy as the reference, and relate that error to training excess risk, input coverage, and conditional drift, which yields the recipe hypotheses in the table of Section 1.5; Section 5 and Appendix E test these hypotheses.

Using a bound that contains terms which cannot be estimated to guide method design has a precedent in domain adaptation. The bound of Ben-David et al. (2010, Thm. 2) contains the error λ\lambda of the ideal joint hypothesis. This term depends on target labels and in general cannot be estimated from labeled source data and unlabeled target data alone (Sec. 7.2); in their experiments the authors treat λ\lambda as negligible and approximate the divergence term by a lower bound. Ganin et al. (2016, Secs. 2 and 4) derive DANN directly from this theory and train the network to reduce a proxy of its divergence term. Section 3 plays a similar role in our work: the decomposition indicates which risk term each training choice targets, e.g., random dropout gives missing patterns positive training mass (Section 1.3), and the controls in Section 1.5 check whether these choices act in the intended direction.

References

  • Ben-David et al. A theory of learning from different domains. Machine Learning 79, 2010.
  • Ganin et al. Domain-adversarial training of neural networks. JMLR 17(59), 2016.
AuthorsOpenReview ↗

Reply to Reviewer MbVv (3/6)

R2: The method's incremental contribution, the offline setting, and the age evidence (W2, Q2)

Thank you for carefully tracing the prior work behind each component and for suggesting a combined control to test the increment of the full recipe.

Individual components do have precedent; our contribution is to place these training choices within one risk analysis of CoT-guided policies, to explain why they need to be combined and what each addresses, and to test the full recipe on robotic manipulation tasks.

2.1 A control on top of the shared components

For the combined control you asked for, we built the PRED + dropout control: guidance comes only from PRED, and the random dropout rate and the asynchronous inference setting are the same as in F, as are the general Worker initialization, the Perceiver, the downstream demonstrations, the 5,000-step training budget, temporal jitter, and numerical age; F additionally uses the GT, NOISE, and ERR sources. A4 has no additional random dropout, so this control is reported separately from the strict PRED-only A4; A4's results also show why a control with a matched dropout rate was needed.

RecipeSourcesRandom dropout per fieldTrain success (%)Test success (%)
FGT + NOISE + PRED + ERR0.2089.1225.04
PRED + dropout controlPRED only0.2087.1622.84
A4PRED only087.4823.56

With initialization, jitter, age, budget, and the asynchronous setting held fixed and the per-field dropout probability equal at 0.20, F's Train / Test success is higher than the PRED + dropout control by 1.96 / 2.20 percentage points; multi-source training gives an incremental gain on top of these shared settings. A4 and the PRED + dropout control differ only in the random dropout rate, and both are below F on Train and Test.

2.2 Offline training and online data collection

This paper trains the Worker offline on existing demonstrations and their CoT annotations. Using on-policy collection for training would require running the policy, collecting the states it visits, obtaining supervision for those states, and training for another round, which is beyond the scope of this paper. The retries in Fig. 4 are an observation made during deployment.

2.3 The role of the age input

The ≤0.01 reported in Appendix E is the change in error relative to the demonstration action, which is a different quantity from the change in the Worker's own output: two predictions can have similar error relative to the same demonstration action while the predicted actions themselves differ.

A2 replaces the numerical age with present in both fine-tuning and inference, changing the training recipe and the input representation together. Appendix E fixes the checkpoint and the demonstration state, lets only the subtask be stale while the geometry stays fresh, and then replaces the subtask age. To isolate the age input, we fix the same F checkpoint and replace only the age shown for fields that are already present, at the moment the Worker reads them; missing fields stay missing, and cache selection, expiry checks, and scheduling still use the true age:

Worker age input (same F checkpoint)Test success (%)
True age25.04
Hidden age (replaced by present)18.28
Random age (each present field drawn uniformly from 0.1–2.0 s per read)23.20

Random replacement keeps the numerical format, and Test success under true age is 1.84 percentage points above random age; hidden age also changes the representation.

2.4 The training age range includes the general training stage

The lookback range of the Worker's general training stage is 0.1–2.0 s, and that of real-robot downstream fine-tuning is 0.1–1.0 s; in one successful trial recorded in Appendix D, the geometry age reaches 2.1 s, beyond the general training range. The revision lists both ranges in the age description of Appendix D.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Common Response For All Reviewers and AC (1/2)

1. Thanks

We thank the Area Chair for the time and effort devoted to coordinating the review, and all three reviewers for carefully reading the paper and appendix and recognizing specific aspects of the research problem, method, and experiments. All the reviewers recognize that the paper addresses a real deployment problem, and we greatly appreciate this shared assessment.

In particular, we thank MbVv for recognizing the distinction between input coverage and conditional drift, the breadth of the simulation and real-robot evaluations, and the detailed experimental information in the appendix. We thank aYEZ for recognizing the design principle of preserving current observations while varying the content, availability, and age of guidance, and the value of the training recipe supported by the controlled RoboTwin comparisons. We thank 8G3t for recognizing the consistency between the semantic interface and training procedure, the conceptual value of the analysis, and the evidence from the main simulation experiments for the combined recipe. The reviewers’ suggestions helped us distinguish the closed-loop benefit of guidance from the added contribution of the training recipe.

Below we first address the closed-loop benefit of online guidance and the additional contribution of source mixing, then clarify the related experimental settings and the scope of each comparison, and finally summarize the revisions to the manuscript, the results that are complete but not yet in the manuscript, and the planned follow-up experiments. The remaining questions from each reviewer are answered point by point in the individual replies.

2. Core questions and new evidence

Both comparisons in this section are run on RoboTwin 2.0, with 25 tasks each in Train and Test, 25 tasks × 100 scenes per cell, and seed 42.

2.1 Does the same Worker benefit from online guidance? (aYEZ, 8G3t, MbVv)

In Table 6(a), F's action errors under correct, wrong, and missing guidance are 1.96 / 1.97 / 1.99 (×10⁻² rad). These are action errors on fixed training states and measure tolerance to wrong and missing guidance; they do not yet quantify the contribution of online guidance to closed-loop task success, which requires a closed-loop comparison of the same checkpoint with and without guidance.

We therefore fix the same F checkpoint and, on the same tasks and scene seeds, change only the guidance the Worker receives at inference: natural online guidance, all guidance masked, and two types of errors injected into the natural output. All four rows below are inference-time interventions on this checkpoint.

Condition (same F checkpoint)Train (%)Test (%)
Natural online guidance89.1225.04
All guidance masked85.4819.40
Injected training-style errors (subtask / box / trajectory, per field 12.5% / 8% / 8%)87.3223.04
Injected corruptions not used in training (per field 30%)87.0420.20

Masking all guidance lowers Train / Test success by 3.64 / 5.64 percentage points, showing that natural online guidance provides a measurable benefit in this closed-loop evaluation. In this evaluation, masking guidance causes a larger drop on held-out tasks, where Test falls from 25.04% to 19.40%.

We further examine performance when guidance is degraded. Training-style errors use the ERR operators of Appendix E; the corruption not used in training replaces the subtask with another task's subtask and mirrors the box and trajectory horizontally. Both interventions are applied independently per field: fields that are not replaced keep their natural output and fields that were missing stay missing, so combinations of correct, wrong, and missing fields can occur. Under these two tested interventions, Test success is 23.04% and 20.20%, still above the 19.40% of full masking.

2.2 Does source mixing still contribute once dropout and other settings are matched? (MbVv, 8G3t)

In the paper, A1 (GT only) and A4 (PRED only) also disable random emptying, so the comparison between F and A4 cannot isolate the contribution of source mixing.

We trained a new PRED + dropout control: guidance comes only from PRED, while initialization, Perceiver, data, the 5,000-step training budget, per-field emptying probability 0.20, temporal jitter, and numerical ages are identical to F; the two differ only in using PRED alone versus a mixture of sources. The table below compares two separately trained models, unlike the same-checkpoint interventions of question 1; the evaluation protocol is the same as in question 1.

RecipeTrain (%)Test (%)
F (mixed sources)89.1225.04
PRED + dropout control87.1622.84

F is higher than the control by 1.96 / 2.20 percentage points on Train / Test, supporting an additional benefit from source mixing under these matched settings. The plan for additional training seeds is in Section 6.

AuthorsOpenReview ↗

Common Response For All Reviewers and AC (2/2)

3. Clarifying experimental settings and the scope of comparisons

  1. Natural asynchronous latency versus controlled staleness interventions. Latency is part of the setting: the Perceiver and the Worker run asynchronously, and the guidance the Worker reads at each step was generated from an earlier frame, with its true age attached. The RoboTwin results of 89.12% / 25.04%, the real-robot results, and the new closed-loop results in Section 2 were all measured under this condition, and therefore already include fields of unequal age as they occur at run time. The evidence that varies staleness in a controlled way comes from the fixed-state experiment of Table 6(b): under 3 s stale guidance, the action-error increase is 0.11 for F and 0.42 for A3 (×10⁻² rad). This result measures action error on fixed states, not closed-loop success.

  2. A training ablation versus an age intervention on the F/A3 checkpoints. A2 (Test 25.04% → 17.48%) replaces the numerical age with present in both training and deployment, changing the age information and the prompt format together. The "≤0.01" (×10⁻² rad) in Appendix E fixes the checkpoints in F/A3 and the demonstration state and replaces only the age of stale subtasks. We apologize that the original appendix did not state this setting clearly, and have added the clarification.

  3. Training lineage of the observation-only π0.5 baseline. Like our method, every observation-only baseline first undergoes continued pretraining on DROID and AgiBot World (without guidance). We also retrained the RoboTwin π0.5 baseline with the same downstream optimization settings as F: Train 77.44%, Test 19.56% (previously 77.28% and 19.48%). The conclusion is unchanged.

4. Revisions made in the revised manuscript

Revised passages are marked in red in the manuscript.

  1. The RoboTwin observation-only baseline now uses the retrained result aligned with F's optimization settings, and the references in Table 6 are recomputed accordingly (the Missing column changes from 3.10, 2.27 to 2.95, 2.21).

  2. We state the continued-pretraining lineage of every observation-only baseline, and the training schedule and evaluated checkpoint shared by the two real-robot models (Appendix D, E, and the Table 6 caption).

  3. The main text now includes the result of masking guidance for the same Worker and the PRED control with matched dropout, and states that A1 and A4 also disable random dropout (Section 5.1).

  4. We distinguish the two fixed-state test settings in Appendix E, and state the two training-stage ranges that correspond to real-robot guidance age (general training 0.1–2.0 s, real-robot fine-tuning 0.1–1.0 s).

  5. Section 3 now states the role of the assumption that training and deployment share a state distribution and the conditions under which it extends, and explains how the analysis organizes the training choices; contribution 2 changes from "matching what it receives at deployment" to "covering the kinds of guidance shift that occur at deployment"; "GT" is marked as machine annotation, with boxes forward-filled within a phase.

  6. We corrected the object name in the Figure 2 caption.

  7. We also revised the risk terminology and related wording in the abstract, introduction, and conclusion, so that the statements about the theoretical conclusions and about recovery behavior are more accurate.

5. Completed results to be added to the manuscript

The following results have been reported in this round of responses (see the corresponding individual replies) but are not yet in the revised manuscript; they will go into the appendix of the final version:

  • closed-loop success under injected training-style errors (per field 12.5% / 8% / 8%) and under injected corruptions not used in training (per field 30%);

  • closed-loop success for the three field conditions: subtask only, geometry only, and memory removed;

  • closed-loop results for hidden age and random age on the same checkpoint;

  • Perceiver prediction-error statistics on training and held-out demonstrations, and the audit of internally consistent error supply in the training recipe.

6. Planned follow-up experiments

Due to the time and computational resource constraints during the rebuttal period, we were unable to complete the three-seed training for all ablation experiments. However, we commit to including the results across three seeds, reported as mean and standard deviation, in the final version.

AuthorsOpenReview ↗

Additional RoboTwin 2.0 Results Across Three Seeds (Updated November 18, 2026, at 20:40 AoE)

We thank the area chair for coordinating the discussion and the reviewers for recognizing the deployment problem, analysis, and closed-loop evaluations, and for proposing further controls.

The paper already shows that the full deployment-aligned recipe improves closed-loop success on the evaluated tasks. Here we report the three-seed ablations completed so far as part of our commitment in the common response. Success varies much less within a setting across seeds than across recipes or inference conditions.

7. Results over three seeds

We evaluated F, A1, A4, and field interventions on F on RoboTwin 2.0 with seeds 42–44. Each seed, condition, and split has 25 tasks × 100 scenes (2,500 rollouts). Comparisons between the paper's natural-guidance results for seed 42 and new controls are descriptive cross-batch differences. Tables 1–2 report mean ± sample SD of success (%).

Table 1. Training recipes (trained separately for each seed)

RecipeTrainTest
F (mixed sources)89.13 ± 0.1025.17 ± 0.17
A4 (PRED only)87.11 ± 0.3323.32 ± 0.22
A1 (GT only)85.71 ± 0.3021.65 ± 0.08

Table 2. Inference-time field interventions on F

ConditionTrainTest
Natural online guidance89.13 ± 0.1025.17 ± 0.17
Geometry only88.24 ± 0.2924.51 ± 0.17
Subtask only86.53 ± 0.2022.67 ± 0.38
All guidance masked85.27 ± 0.2419.49 ± 0.23
  • Online guidance improved closed-loop success at every seed. With F fixed, full masking reduced success by 3.87 ± 0.21 points on Train and 5.68 ± 0.11 on Test.
  • F > A4 > A1 at every seed on both splits. F led A4 by 2.03 ± 0.36 / 1.85 ± 0.38 points (Train / Test) and A1 by 3.43 / 3.52 points on average. A4 and A1 both disable per-field random dropout. Their comparison supports using PRED in this setting.
  • The Worker did best with both subtask and geometry at every seed on both splits. Either input alone beat full masking. For geometry only, the Perceiver still used subtask text to generate geometry for the Worker.

8. Closed-loop controls for seed 42

Source mixing improved success with matched dropout. F led the PRED control by 1.96 points on Train and 2.20 on Test with the same per-field random dropout rate, temporal jitter, numerical age inputs, and training budget.

Numerical age values affected success. With F's weights fixed, randomizing only the age values shown to the Worker reduced Test success from 25.04% to 23.20%.

Guidance retained most of its observed gain under training-type errors. Test success was 23.04%, 3.64 points above full masking (19.40%). With corruptions outside those used in training, Test success was 20.20%, close to full masking.

9. Variation across seeds and settings

All seeds share the same initialization, tasks, and scenes.

Table 3. Success-rate SDs within and across settings

Success (%)TrainTest
Pooled within-setting SD (recipes / field conditions)0.26 / 0.220.17 / 0.25
SD across setting means (recipes / field conditions)1.72 / 1.731.76 / 2.54

The SD across setting means was 6.5–10.5 times the within-setting SD. The 10 contrasts (F versus A4, A1, and three field interventions on both splits) kept their direction across seeds. Their largest SD was 0.39 points, and each seed-42 contrast was within 0.39 points of its three-seed mean.

Table 4. Seed-42 Test differences relative to the largest observed SD across seeds

ComparisonDifference (points)Approx. multiple
F − PRED control (matched per-field random dropout)+2.205.6
F true age − random age+1.844.7
F natural − training-type errors+2.005.1
Training-type errors − full masking+3.649.3
F natural − corruptions outside training+4.8412.3
F true age − hidden age+6.7617.2

The paper establishes the full recipe's closed-loop gains in the evaluated settings. The new results show that the recipe ranking and online guidance gains hold across all three tested seeds.

10. Three-seed extensions in the final version

The remaining experiments may extend beyond the rebuttal period. As promised, the final version will report mean ± sample SD over three training seeds for all ablations. For the comparisons below, we will report within-seed differences from F with natural guidance and also compare both error injections with full masking.

  • Training recipes. Add two seeds each for the PRED control with per-field random dropout, A2 (no numerical age), and A3 (no extra age jitter), using the other settings from seed 42.
  • Inference conditions. On the two additional F checkpoints, repeat both error injections, both age interventions (on Test), and memory removal.

We thank the area chair and reviewers again. We will document the controls and their evaluation settings in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.