- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the mismatch between semantic guidance used to train vision-language-action policies and the guidance available during deployment, where predictions may be incorrect, missing, or temporally stale. It introduces a Perceiver–Worker framework: the Perceiver maintains task memory and predicts subtask descriptions, affordance regions, and image-space trajectories, while a flow-matching Worker conditions on current observations and the latest guidance, accompanied by field-specific physical ages. The Worker is trained using a mixture of annotated, predicted, perturbed, and missing guidance, with temporal sampling that preserves the alignment between current observations and demonstrated actions. An asynchronous cache allows guidance fields to refresh independently. The paper also derives a squared flow-matching risk decomposition and a transfer bound involving training error, input coverage, and conditional drift. Evaluation includes LIBERO, a training/held-out task split of RoboTwin 2.0, controlled guidance interventions on fixed demonstration states, and real-robot desktop-cleaning experiments with several scene variations.
Critical strengths
- The paper identifies a concrete deployment problem and develops a coherent, inspectable solution. The distinction between incorrect, missing, and stale guidance is useful for reasoning-conditioned robot policies. The proposed interface makes the source age of each guidance field explicit, preserves access to current observations, and allows action generation to continue while semantic predictions are refreshed. The training procedure is consistent with this design: offline predictions use causal observation histories, and temporal perturbations change the guidance while keeping the current action supervision aligned (Sections 4.2–4.4 and Appendix F.3). The analysis provides a useful conceptual account of when guidance can help or hurt. Equation (1) separates guidance information gain from the learned policy’s excess error, and Proposition 1 appears correct under its stated assumptions. The authors also correctly acknowledge that deterministic guidance can help a finite learned model without adding information to the Bayes-optimal predictor. Together, these elements make the work informative for designing practical semantic interfaces.
- The main simulation results provide credible evidence that the combined recipe improves closed-loop performance. The strongest evidence is the authors’ own simulation evaluation. Their observation-only π0.5 control follows the same continued-pretraining and downstream training stages, and the RoboTwin fine-tuning ablations share the general Worker checkpoint, Perceiver, data, and training budget (Appendix C). The method improves LIBERO mean success from 97.1% to 99.0%, including a gain from 93.5% to 97.8% on LIBERO-Long, and improves held-out RoboTwin success from 19.48% to 25.04% (Tables 3–4). The evaluation is substantial, with 500 rollouts per LIBERO task and 100 rollouts per RoboTwin task. Table 5 further supports the usefulness of the overall training recipe through closed-loop ablations. The real-robot experiments provide additional evidence of feasibility across changed objects, layouts, and partially completed tasks, although their sample size and baseline matching limit the strength of the conclusions drawn from them.
Critical weaknesses
- The controlled robustness evidence does not yet establish that the policy preserves useful guidance dependence under held-out, closed-loop deployment shifts. Table 6 is a useful sensitivity diagnostic, but Appendix E specifies that it uses 75 and 77 selected states from the Worker’s training demonstrations. The ERR intervention also reuses the corruption operators used during training. In these tests, all three guidance fields receive the same status, whereas the motivating deployment problem includes combinations of valid, incorrect, missing, and differently aged fields. Moreover, the full model has nearly identical action errors with correct, erroneous, and missing guidance in Table 6(a): 1.96, 1.97, and 1.99, respectively, in units of (10-2) rad. This demonstrates low sensitivity on those states, but leaves open how much useful online guidance the policy retains. It does not establish that the model ignores guidance; the relevant same-checkpoint, closed-loop comparison is missing. The paper does report held-out closed-loop success in Tables 4–5, which is valuable. However, those results do not isolate guidance usefulness or robustness under controlled shifts. A focused evaluation on held-out tasks, comparing the same full model with natural guidance, masked guidance, and mixed-field failures, would directly connect the proposed mechanism to deployment performance.
- The incremental contribution of mixed-source training is not fully isolated, and some supporting comparisons have different training histories. The GT-only and PRED-only ablations change more than the guidance-source mixture. Appendix C states that A1 and A4 also disable random emptying, while the full recipe includes artificial field dropout. Consequently, the comparisons establish the benefit of a package of changes, but do not isolate the contribution of GT/NOISE/ERR mixing beyond predicted guidance with the same dropout, age inputs, and temporal jitter. A matched PRED-plus-dropout control is particularly important given the related work on reasoning dropout and delayed-input training already acknowledged by the authors. The closest existing comparison is F versus A4, with held-out success of 25.04% versus 23.56%. The reported downstream setup specifies one training seed, and uncertainty for this difference is not reported. Paired task-level results and repeated runs for this pivotal comparison would clarify its reliability. Training-history matching also requires clarification outside the main simulation comparison. Appendix E explicitly states that the observation-only reference in Table 6 uses a different initialization and learning rate. For the real robot, Appendix D describes the proposed model as starting from the general Worker and the baseline as starting from pretrained π0.5. These details limit attribution of the supporting gains specifically to the proposed guidance-training recipe.
Questions for authors
- Targeted evidence: useful guidance under deployment shift. Can the same trained full model be evaluated on a representative subset of held-out tasks with natural Perceiver guidance, all guidance masked, and selected mixed-field failures or delays? Including naturally occurring predictor mistakes or a corruption type excluded from training would help establish whether the policy retains a measurable benefit from informative guidance while remaining robust to failures.
- Targeted evidence: source mixing beyond dropout. What is the performance of a PRED-only variant that retains the full model’s field-dropout probability, temporal jitter, age inputs, and training budget? How consistent is the full model’s advantage across held-out tasks and training seeds, especially given the 1.48 percentage-point gap between F and A4?
- Clarification and targeted evidence: the role of numerical age. How should the large A2 degradation in Table 5—25.04% to 17.48% held-out success—be reconciled with Appendix E’s finding that replacing a stale field’s true age with 0.2 s changes action error on the tested subtask states by at most 0.01? These interventions differ, so the findings need not conflict. However, same-checkpoint closed-loop results with true, hidden, or shuffled age tags would clarify whether numerical age controls reliance at inference or mainly affects training.
- Clarification: baseline initialization and continued pretraining. Did the real-robot π0.5 baseline receive the same continued-pretraining data and budget as the general Worker before the 6,000 adaptation updates? Please provide the checkpoint lineage and optimization settings for the main simulation baseline, the Table 6 reference, and the real-robot baseline. If their histories differ, which comparisons should be interpreted as tests of the overall system, and which isolate the guidance-training procedure?
- Clarification: measured training–deployment alignment. How were the source probabilities and delay ranges selected, and what are the effective per-field age, missingness, and failure distributions during deployment? Appendix D reports ages from one successful Base rollout sampled every 10 seconds; broader measurements would better support the alignment claim. Please distinguish actual PRED ages from nominal lookback ranges, and report the Worker query cadence and Perceiver/Worker latency statistics on the shared GPU.
4 comments
Reply to Reviewer 8G3t (2/4)
R2: The role of source mixing beyond dropout (W2, Q2)
Thank you for suggesting a PRED-only control that keeps the same dropout to isolate the contribution of source mixing.
2.1 Matched control
In the paper, A4 also disables random dropout, so the comparison between F and A4 indeed cannot separate this contribution.
Following your suggestion, we trained PRED + dropout: guidance comes only from PRED and every other setting matches F, including initialization, Perceiver, data, the 5,000-step training budget, the per-field dropout probability of 0.20, temporal jitter, and numerical age. The control differs from F only in the GT, NOISE, and ERR sources.
2.2 Closed-loop results
| Recipe | Sources | Dropout per field | Train success (%) | Test success (%) |
|---|---|---|---|---|
| F (paper Table 4) | GT + NOISE + PRED + ERR | 0.20 | 89.12 | 25.04 |
| PRED + dropout control (newly trained here) | PRED only | 0.20 | 87.16 | 22.84 |
| A4 (paper Table 5) | PRED only | 0 | 87.48 | 23.56 |
Under the same dropout rate, temporal jitter, and age input, F's Test success is 2.20 percentage points above the PRED + dropout control (1.96 on Train), so source mixing gives an additional gain beyond dropout in this matched comparison. The A4 row differs from F in both sources and dropout, which is why the matched control was needed.
The comparisons above use training seed 42. We commit to reporting the mean and standard deviation over three training seeds for all ablation experiments in the final version (see the reply to all reviewers).
R3: The role of numerical age (Q3)
Thank you for pointing out that the large A2 degradation and the small change in Appendix E need to be reconciled, and for suggesting a comparison of age tags on the same checkpoint. We first explain how the two differ and then report the added closed-loop test. A2 and the Appendix E test apply different interventions and measure different quantities.
3.1 Experimental settings
A2 replaces the numerical age with present in both training and deployment, so it changes the trained model and the prompt format together and is measured by closed-loop success.
Appendix E fixes the trained models and the demonstration state: on states just after a subtask switch, only the subtask is made stale while the box and trajectory stay fresh, and the subtask's true age is replaced by 0.2 s; for F and A3, two separately trained and frozen models, the action error changes by at most 0.01 (×10⁻² rad). This comparison measures the change in error relative to demonstration actions; the predicted actions themselves may still change.
Table 6(b) is a separate protocol in which all three fields are stale together. The original manuscript did not distinguish these settings, and we apologize for it; the revision states them in Sections 5.1.2 and 5.1.3 of the paper and in Appendix E.
To test the role of age at inference on the same checkpoint, we fix F's weights and change only the age text in the Worker input, while the age-cap check and scheduling still use the true age. The two variants are:
(1) hidden age: the age of each present field is written as the word present, which changes the representation;
(2) random age: at each read, the age of each present field is drawn independently and uniformly from 0.1–2.0 s, which keeps the numerical format and perturbs only the value.
3.2 Closed-loop results
| Age the Worker sees (F, Test) | Test success (%) |
|---|---|
| True age (paper Table 4) | 25.04 |
Hidden age (present) | 18.28 |
| Random age (uniform 0.1–2.0 s) | 23.20 |
Test success under true age is 1.84 percentage points above random age, so the age value itself affects closed-loop performance of this checkpoint. Under hidden age success falls to 18.28%, a condition that changes the age information and the prompt format together.
Reply to Reviewer 8G3t (3/4)
R4: Baseline initialization and continued pretraining (Q4)
Thank you for asking us to state the lineage and optimization settings of each baseline, and which comparisons test the whole system versus the training recipe.
Comparisons among F, A1–A4, and the PRED + dropout control isolate the guidance training recipe: these models share one general Worker checkpoint and differ only in downstream guidance training. Comparisons with observation-only π0.5 (the RoboTwin main table, the Table 6 reference, and the real-robot baseline) test the whole system, because the baseline's continued pretraining has no guidance.
| Model | Continued pretraining (DROID + AgiBot World) | Downstream training | Downstream optimization (AdamW, cosine decay) | What this comparison tests |
|---|---|---|---|---|
| F, A1–A4, PRED + dropout control | with guidance, sharing one general Worker | RoboTwin training tasks, 5,000 steps | peak learning rate 5×10⁻⁵, 250 warmup steps, decay to 2.5×10⁻⁷ | the guidance training recipe itself |
| RoboTwin observation-only π0.5 (revised Table 4, also the reference for Table 6) | without guidance | RoboTwin training tasks, 5,000 steps | peak learning rate 5×10⁻⁵, 250 warmup steps, decay to 2.5×10⁻⁷ | the whole system |
| Real-robot GoT-VLA | with guidance, general Worker | 866 real-robot demonstrations, 30,000-step schedule, evaluated at step 6,000 | peak learning rate 5×10⁻⁵, 1,500 warmup steps, decay to 2.5×10⁻⁷ | the whole system |
| Real-robot observation-only π0.5 | without guidance | 866 real-robot demonstrations, 30,000-step schedule, evaluated at step 6,000 | peak learning rate 5×10⁻⁵, 1,500 warmup steps, decay to 2.5×10⁻⁷ | the whole system |
Every observation-only π0.5 baseline starts from official π0.5, undergoes continued pretraining on DROID and AgiBot World without guidance, and is then fine-tuned downstream; our method performs continued pretraining with guidance on the same data. "Starting from pretrained π0.5" in Appendix D and "different initialization" in Appendix E both refer to this guidance-free continued-pretraining checkpoint. The initial version did not state the baseline lineage clearly, and we apologize for it; the revision states it in the Table 6 caption and in Appendices D and E.
The "different learning rate" mentioned in the initial Appendix E refers to the RoboTwin observation-only baseline using the fine-tuning learning rate recommended by the official π0.5 release at that time. We retrained and re-evaluated that baseline with the same downstream optimization settings as F: Train success 77.44%, Test 19.56%, versus 77.28% and 19.48% in the initial Table 4. The revision adopts these results (see the reply to all reviewers) and the table above lists this setting; the observation-only references in Table 6 are recomputed accordingly, with the Missing column changing from 3.10, 2.27 to 2.95, 2.21, and the conclusion unchanged.
On the real robot the two models use identical optimization settings. The training loss had largely stabilized by step 6,000; given our compute constraints, we evaluated the checkpoint at that step. The real-robot baseline uses the same schedule and is likewise evaluated at step 6,000. The RoboTwin downstream optimization setting (including the observation-only π0.5 baseline) is given in the "Downstream optimization" paragraph of Appendix F.5, and the real-robot schedule and evaluated checkpoint are given in Appendix D.
Reply to Reviewer 8G3t (4/4)
R5: Source weights and the guidance distribution at deployment (Q5)
Thank you for asking us to distinguish the nominal training lookback from the actual PRED ages and to report measured deployment statistics. We explain the training settings below and compiled a real-robot deployment record.
The source weights, field dropout rates, and training lookback ranges of each benchmark were set empirically before the held-out evaluation and then held fixed; the values are in Appendix F.5, and R2 tests the recipe under these settings.
The nominal lookback is a training-time sampling setting. For each field, training samples a historical time in physical time within the benchmark's range (0.12–0.66 s on RoboTwin, 0.1–1.0 s for the real-robot Worker). Annotated and perturbed fields take this sampled age, whereas a PRED field carries the source time of the selected prediction record, so its age can be older than the sampled lookback (Appendix F.5).
Deployment involves no lookback sampling: each field's age is computed from its true source time and the time it is read.
For the distribution at deployment, we compiled a real-robot deployment record: the Perceiver and the Worker share one RTX 5090, a client queried the Worker at the deployment cadence of one query per 25 executed steps, and the record covers the responses of the per-field request loops and the 53 Worker reads over 60 consecutive seconds.
| Loop | Completed (rate) | Latency (s) | Age (s) | Used / missing |
|---|---|---|---|---|
| Memory | 5 (one per window) | 0.90 / 1.12 | — | — |
| Subtask | 112 (1.87 Hz) | 0.47 / 0.70 | 0.70 / 0.87 / 1.43 | 53 / 0 |
| Affordance | 116 (1.93 Hz) | 0.52 / 0.74 | 0.97 / 1.30 / 1.67 | 52 / 1 |
| Trajectory | 116 (1.93 Hz) | 0.86 / 1.18 | 0.97 / 1.30 / 1.67 | 51 / 2 |
| Worker | 53 queries (0.88 Hz) | 0.27 / 0.66 | — | — |
Completed is the number of responses that loop returned during the 60 s recording, and the rate is that number divided by 60 s. Latency is the time from request submission to return (p50 / max).
The last two columns are computed over the 53 Worker queries rather than over the whole recording: age is the value of that field at those query times (p50 / p90 / max), and used / missing counts the queries in which the field was provided to the Worker or was blanked, so the two entries in each row sum to 53. Memory is not an input to the Worker.
In this recording, the subtask (1.87 Hz) and the two geometry fields (1.93 Hz) refreshed faster than the Worker queried (0.88 Hz), while memory produced one summary per completed 10 s window.
Across these 53 reads, the median age was 0.70 s for subtask and 0.97 s for affordance and trajectory, with a maximum of 1.67 s; the three fields were missing in 0, 1, and 2 reads.
This table reports age, missingness, refresh rate, and latency; it does not measure whether field contents are correct. "Used" means the field was present, not that it was correct, and "missing" does not cover all failures.
Online states have no frame-level semantic labels, so we have not measured an online error rate; the Perceiver's prediction-error statistics on training and held-out demonstrations are in our reply to Reviewer aYEZ, and they come from an evaluation on demonstrations. The revision adds this table to Appendix F.6 and states in Appendix D that the earlier age description comes from one successful trial.
Due to the time and computational resource constraints during the rebuttal period, we prioritized covering a broader range of experimental settings rather than running multiple seeds for each individual experiment. Nevertheless, we commit to reporting the mean and standard deviation across three independent training seeds for all simulation experiments in the final version.
Updated on November 18, 20:40 AoE: We have updated the 3-seed experiments (including the means and standard deviations) for some experimental settings. The results are presented on page 3 of the Common Response.
Reply to Reviewer 8G3t (1/4)
Thank you for your careful review and for recognizing our treatment of a practical deployment problem and the consistency between the semantic interface and training design. We also appreciate your recognition of the conceptual value of the analysis, the evidence from the main simulation experiments for the combined recipe, and the substantial evaluation effort. Below, we address your questions in turn.
R1: The role of guidance under deployment variation (W1, Q1)
Thank you for suggesting that the same full model be compared on held-out tasks under natural guidance, masked guidance, and mixed-field failures. Following this suggestion, we fixed the F checkpoint and ran closed-loop comparisons, with the all-masked result as the direct evidence of the closed-loop benefit of guidance.
The natural online baseline already retains the Perceiver's prediction errors, missing fields, and the ages produced by asynchronous execution, with no additional artificial injection; the interventions below change this input in a controlled way. F is trained with ERR corruptions and random field dropout (Appendix F.5); the comparison below tests what F does with guidance at inference.
1.1 Setup
We fix the F checkpoint and evaluate on the 25 training tasks (Train) and 25 held-out tasks (Test) of RoboTwin 2.0, 100 scenes per task, 2,500 rollouts per cell, seed 42. The three inference-time interventions are:
All guidance masked: the Worker input receives no guidance.
Additional training-style errors: at each read, each present subtask, affordance, and trajectory field is selected for the ERR operators of Appendix E with probability 12.5%, 8%, and 8%, taken from the ERR source weights of F's training recipe before dropout. These are selection probabilities applied on top of PRED's own natural errors, not the realized error rate of the input.
Corruptions not used in training: each present field is replaced with probability 30%; the subtask is replaced by another task's subtask, and the box and trajectory are mirrored horizontally.
Injection is independent per field and applied on top of the natural Perceiver output: fields that are not replaced keep their natural output, and fields that were already missing stay missing. These per-field interventions allow combinations of correct, wrong, and missing fields to occur; these closed-loop tests keep the natural field ages, and the fixed-state 3 s staleness test is discussed in R3.
1.2 Closed-loop results
The drop is the decrease in success relative to natural online guidance, in percentage points (pp).
Masking all guidance lowers F's Train / Test success by 3.64 / 5.64 percentage points, so the same checkpoint obtains a measurable closed-loop benefit from online guidance. In this evaluation the drop is larger on the held-out tasks, where Test success falls from 25.04% to 19.40%.
Under the two tested error interventions, Train / Test success falls by 1.80 / 2.00 percentage points for training-style errors and by 2.08 / 4.84 percentage points for corruptions not used in training; in all four cells success remains above that of full masking. This result applies to the intervention types and rates tested here.