acceptodds
Desk rejected at ICLR 2027

Q-Guided Action-Transport Editing for Real-World Reinforcement Learning of Vision-Language-Action Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Flow-based vision-language-action (VLA) policies provide expressive priors for robot manipulation. Adapting them with online reinforcement learning remains difficult because each action is produced by a multi-step sampler. Backpropagating through the complete transport is costly, while an output-space correction cannot use the remaining generative dynamics to refine its edit. We introduce Q-Transport, an off-policy adaptation framework that inserts a stochastic local edit at an interior flow state. A conservative critic evaluates the resulting action, and its gradient reaches the edit policy through only the remaining frozen Euler steps. Q-Transport uses a fixed interior point produced by a stopped frozen prefix. Aggressive Q-Transport instead samples randomized interior points and directly corrupts completed actions, learning across different transport lengths. Both variants share an objective that combines action value, entropy, and optional prior-score regularization, and both select between paired base and edited candidates at inference. The edit location also unifies lightweight adaptation strategies: output-space residual methods act at the clean endpoint with an identity tail, initial-noise methods act at the noise endpoint before the complete sampler, and Q-Transport uses partial pretrained transport between these endpoints.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision Language Model, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VfFvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Q-Transport introduces a mechanism for adapting pretrained flow-based VLA policies by learning critic-guided edits at intermediate flow states rather than modifying only the source noise or final generated action. The method performs well on the paper's five real-robot insertion/manipulation tasks, achieving 83.2% mean success rate versus 70.0% for the strongest competitor, and the real-world evaluation is a major strength. At the same time, the evidence for broad VLA adaptation is limited because the tasks occupy a relatively narrow contact-rich insertion regime, the implementation is computationally heavy, and success drops to only 18% under held-out initial configurations.

Critical strengths

Interesting and technically clean adaptation interface. Editing partially formed flow states exploits the pretrained model as a nonlinear transport operator rather than simply treating it as a frozen action generator.

Strong real-world results. Five physical manipulation tasks are evaluated, and Q-Transport outperforms the compared methods on every task by roughly 10–16 percentage points, averaging +13.2 points over the strongest competitor.

Practical ablations are useful. The paper demonstrates that critic-guided editing provides most of the improvement, quantifies the contribution of base-policy consolidation, and identifies four flow steps as a sensible computation/performance knee.

Critical weaknesses

Generalization remains poor. On held-out initial configurations, all systems degrade sharply and Q-Transport achieves only 18% success, despite outperforming baselines. This substantially qualifies the claim of robust task adaptation.

Computational demands are substantial. Training the π0.5-based methods uses eight A100 80GB GPUs plus another A100 for rollout collection, while inference evaluates multiple base and edited candidates through repeated flow evaluations. This makes the method less obviously attractive when the motivation includes efficient real-world adaptation.

The experimental scope is narrower than the VLA framing suggests. Flower insertion, cable insertion, charger insertion, and two USB insertion variants are all contact-rich manipulation/insertion problems with comparatively small pose variation. More semantically diverse and longer-horizon VLA tasks would be needed to support broader claims.

Questions for authors

See Weaknesses above.

Additionally:

How does this compare to continual learning results such as this one: CLARE: Continual Learning for Vision-Language-Action Models via Autonomous Adapter Routing and Expansion (2026)?

Can you further evaluate your method in broader simulation experiments to support the claims around the potential and the limitations of this work?

2 comments
AuthorsOpenReview ↗

Response to Reviewer VfFv (1/2)

We thank you for your constructive review. Revisions to the main manuscript are highlighted in orange. We add four supplementary real-robot tasks and four non-insertion simulation tasks, and below we clarify the precision-manipulation setting, SFT initialization, additional results, computational cost, and connection to continual learning.

W1 & Q2. Generalization and Scope

Why is SFT performance low?

We deliberately selected difficult, high-precision manipulation tasks for which demonstration-only training is difficult to bring to reliable autonomous convergence under the available data budget. Small errors in alignment or contact can cause task failure even when the overall motion resembles the demonstrations. With 25 human-demonstration episodes per task, offline SFT achieves a 2.0% mean success rate on the five original tasks. We therefore first use human-in-the-loop online BC to establish a stronger common initialization, reaching 37.2% mean success. All compared online adaptation methods start from this task-specific BC checkpoint; QT then reaches 83.2%, compared with 70.0% for EXPO-FT (Section 4.1; Table 1). These results show the benefit of online adaptation beyond the shared BC initialization.

Broader real-robot task coverage

Appendix B.4 adds bottle placement in an open drawer, cup placement on a coaster, bottle-opener hanging, and bimanual sweeping. Each method is evaluated over 50 autonomous trials per task, with training seed 42. Table 7 reports the following success rates (%):

TaskQTHG-DAggerEXPO-FT
Bottle in open drawer666662
Cup on coaster908886
Opener on hook765262
Bimanual sweeping545650
Four-task mean71.565.565.0

QT achieves the highest four-task mean, 71.5%, compared with 65.5% for HG-DAgger and 65.0% for EXPO-FT. These experiments cover four additional task families on the same robot platform. The drawer task evaluates bottle placement in an already-open drawer, with QT refining the placement phase.

Broader simulation evaluation

We added RoboTwin 2.0 (SAPIEN) experiments on stamp_seal, place_a2b_left, stack_blocks_two, and hanging_mug, covering precision placement, relative placement, stacking, and hanging. Appendix D specifies the same π0.5-based observation/action interface and task-specific SFT initialization for QT, EXPO, DSRL, and SAC-Flow. The protocol uses three training seeds per task and method (48 RL units), 100 paired evaluation seeds per task, and 11 checkpoints. The online training budget is 1,020 updates and 340 episodes (Table 12). Table 13 reports the following four-task mean success rates at step-1020:

MethodSuccess Rate (%)
EXPO65.0
QT71.0
DSRL51.3
SAC-Flow40.0

At step-1020, QT achieves 71.0% mean success, compared with 65.0% for EXPO, 51.3% for DSRL, and 40.0% for SAC-Flow. QT improves on EXPO by 6.0 percentage points. The paired SFT baseline for this RL protocol averages 38.8%; its stamp_seal baseline is 20% (Table 13).

Evaluation setting

These additions extend the evaluation beyond the original insertion tasks. The real-robot experiments measure adaptation to each task’s nominal setup; success on held-out initial configurations is 18% (Section 4.4). The simulation evaluates precision-critical key stages from scripted initial states, with fixed scene appearance and expert-feasible seeds for one selected arm per task.

AuthorsOpenReview ↗

Response to Reviewer VfFv (2/2)

W2. Computational Demands and Efficiency

The real-robot π0.5-based setup uses eight A100 80 GB GPUs for training and a separate A100 80 GB for rollout collection. Appendix A.5 and Table 4 separate approximately seven hours of shared data collection and SFT initialization from approximately two hours of online adaptation per task. The total budget is approximately nine hours when the shared initialization is charged to each method. QT and EXPO-FT both use 16 candidates for action selection and TD backups, whereas DSRL, SAC-Flow and HG-DAgger use one. Each QT edited candidate applies three successive corrections. During editor training, the value gradient passes through one frozen x-prediction per update, giving the short gradient path shown in Figure 5.

Table 4 reports range-derived latency estimates as midpoint ± half-width. The current configuration is estimated at 390 ± 80 ms per decision for QT and 375 ± 75 ms for EXPO-FT; after prefix de-duplication, the estimates are 120 ± 30 ms and 110 ± 30 ms, respectively. The table accounts for candidate counts and method-specific update quotas alongside these latency estimates.

Q1. Relation to CLARE and Continual Learning

CLARE [1] studies sequential task acquisition with adapter expansion and autonomous routing, retaining earlier skills while learning each new task. Q-Transport studies reward-guided adaptation within a task: it edits intermediate action states and uses replay to improve the task-conditioned flow policy. Each task is adapted separately from its task-specific initialization.

The two evaluation protocols measure different aspects of adaptation. CLARE evaluates new-task acquisition and retention across a task sequence. Our simulation evaluates learning progress and stability across checkpoints within each task; the FWT/NBT/AUC quantities in Appendix D are defined for that within-task protocol.

[1] Römer, Zhang, Li, and Schoellig. CLARE: Continual Learning for Vision-Language-Action Models via Autonomous Adapter Routing and Expansion. IEEE Robotics and Automation Letters, 2026.

Reviewer_PSy9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Generative Control Policies based on diffusion and flow-matching models have been augmented in various ways to not only imitate the data distribution described by demonstrations but to also follow other signals (typically: rewards) and optimize auxiliary objectives. The aim of these approaches is to adapt a given pretrained model so as to make it better according to another metric. In contrast with previous methods which act on either the initial noise or the action (final stage of denoising), the authors instead exploit an arguably underused interface: the intermediary, “noisy” actions corresponding to the non-final flow stages. Their method, Q-Transport, operates on this interface. It edit intermediate-stage actions then predict clean actions from those edits in one velocity network evaluation. The editor is trained to maximize the soft action-values of the predicted clean actions after edit. The soft action-value function is trained on both edited and non-edited flow path, doubling the number of action candidates that the target action-value uses. Training (for all three networks, counting the Q ensemble as one) is done off-policy, with a replay buffer. The underlying VLA continues to evolve through behavior cloning.

Critical strengths
  • The introduction is nice and clear. Comment: I fail to see the logical link to the bit “even though their mapping to the clean action still depends on the pretrained velocity field”, L49-50.
  • The gradient flow (computational graph) discussion in the paragraph named “Optimizing the edit policy” is key, especially considering the related works performing backpropagation from Q values through entire flow chains. It is central to the appeal of the method in the landscape, and I think the authors should put this section is better emphasis, maybe dedicating it a diagram, with the various gradient flows.
  • Equation 4 illustrates well how the edit shapes the geometry.
  • The method involves techniques targeted at increasing robustness and reducing epistemic uncertainty: the executed policy is the composition of stochastic candidate generation and critic-based candidate selection. In particular, the latter involves an ensemble of 10 Q functions, and the former involves: 8 base action candidates, each using four Euler steps, and 8 edited action candidates, each using three successive corrections.
  • Experiments: Q-Transport provides a consistent improvement across tasks.
  • It is interesting to see that removing base-policy consolidation reduces average success from 78.7% to 67.3% across three tasks.
Critical weaknesses
  • That might just be a preference, but I would encourage the authors to put the references as close as possible to the referred item, while preserving legibility to the best of your ability. This would make Section 2 easier to parse, as opposed to having a 5+ references stacked at the end of every sentence. Having the methods’ names there would be a plus too.
  • The Related Work is a long catalog of dense sentences intertwined with stacks of references; after a page, the goal is to relate as tightly as possible the work the authors proposes to the other neighboring works in the research landscape. Less is more in that case; if the Related Work section would work in pretty much any GCP paper, then it does not play the role it is supposed to play. I therefore encourage the authors to lighten and adapt the Related Work.
  • End of page 3: it might be useful for the reader to be more explicit about what “decision index” refers to.
  • Figure 1: the caption contains “a tilde” while the image contains no “a tilde” but contains “a hat”, absent from the caption. The authors should make it clearer that the “a hat” is the clean action predictor, and “a tilde” is the clean action, before sending the reader to Fig 1.a.
  • The way and especially the order in which the various elements composing the method are exposed is perplexing. I would encourage the authors to revise how the method is presented, especially how the components are introduces. Figure 1.a is unclear and could use a major revision as well.
  • The algorithm could be split into two, placed side-by-side, separating sampling from online updates.
  • The paper does not specify the number of independent training seeds.
  • Mechanism isolation is insufficient. The paper describes ablations without learned edits and with a single correction, but their numerical results appear to be missing. These are arguably the most important comparisons for understanding the contribution.
Questions for authors
  • L178-179: “while the editor changes the state that the transport receives”. Is the “transport” what the authors name the operation operated by the editor, i.e. an internal-state edit? If that is the case, what is “transport editing” (title of Figure 1.a)?
  • Am I correct in understanding that: a) at training time, the editor only receives gradients for one randomly chosen noise level; b) At inference, three successive edits are composed. The training objective does not directly optimize the complete three-edit 0.75,0.5,0.25\\{0.75,0.5,0.25\\} sampling procedure.
  • Could the authors justify why (at inference/sampling) a) they consecutively x-predict a clean action after edit and then re-corrupt the action and do the operation all over again b) at a noise level closer to the clean action every time?
  • The edit’s objective is not directly the objective of the deployed policy. The method relies on independently trained local corrections working well when composed. Wouldn’t also having the consecutive three-edit spree in training not only align the objective with the executed policy but also increase robustness of the learned editor (and maybe even BC VLA)?
  • At inference/sampling, why does it start from unprocessed Gaussian noise rather than an intermediate state obtained by running the pretrained flow, given that the editor is trained on affine corruptions of executed actions? The way it is done, the distributions of intermediate states in training and sampling do not need to match.
  • Do the authors apply the edits at decreasing noise levels as a way to operate coarse-to-fine refinement?
2 comments
AuthorsOpenReview ↗

Response to Reviewer PSy9 (1/2)

Thank you for the careful review and for recognizing the real-robot results and gradient-path design; the corresponding manuscript revisions are highlighted in blue. We now report the requested mechanism ablations (Appendix C, Table 8). Below, we clarify the editing interface and training–sampling distinction, then summarize the presentation and reproducibility revisions.

S1 / Q1. What is transported, and what is edited?

We edit an intermediate action-space state so the pretrained flow shapes the correction before critic evaluation. The editor and frozen x-prediction map perform distinct operations (Section 3.2, Equations 3 and 5): Fψˉ(s;y,σ)=y−σvψˉ(s,y,σ),x^σ=xσ+Δx,a^=Fψˉ(s;x^σ,σ).F_{\bar\psi}(s;y,\sigma)=y-\sigma v_{\bar\psi}(s,y,\sigma),\qquad \hat x_\sigma=x_\sigma+\Delta x,\qquad \widehat a=F_{\bar\psi}(s;\hat x_\sigma,\sigma). "Transport editing" changes the map's input, with its parameters fixed during the editor update. Equation 4 shows that the local action change is shaped by I−σ∇xvI-\sigma\nabla_xv. An output residual acts after this map; an initial-noise perturbation passes through the complete sampler. Our interface retains the pre-trained action geometry between the correction and the critic.

S2. Where does the Q-gradient flow?

Figure 5 (Appendix A.2) separates forward computation from the Q-gradient path: critic action input → one frozen x-prediction → editor parameters. Gradients stop at the replay-derived affine state; no full Euler chain is unrolled. Flow parameters are frozen during this update, but input gradients pass through the flow. Successful-action behavior cloning updates the base policy separately (Equation 8; Section 3.3).

Q2. One sampled noise level in training versus three edits at inference

Training optimizes one sampled-level correction; inference composes the same shared editor across levels. Each update constructs an interior state directly from an executed replay action, samples σ\sigma uniformly from {0.25,0.50,0.75}\{0.25,0.50,0.75\}, and differentiates one x-prediction (Equation 6; Section 3.3). Inference reuses this noise-conditioned operator at 0.75\to0.50\to0.25, acting on the current estimate (Algorithm 1). This enables direct off-policy training without reconstructing generative trajectories, while sampling uses repeated refinement.

Q3. Why predict an endpoint, re-corrupt it, and repeat?

x-prediction converts a noisy-state edit into a complete action chunk scored by Q, allowing the task critic to train the editor directly (Equations 3, 5 and 6). Re-corruption anchors the next correction to this updated action, carrying earlier edits forward through the same local operation. The decreasing-noise schedule progressively retains more of the accumulated estimate, as explained in Q6.

Q4. Would training through all three edits improve alignment and robustness?

Three-edit unrolling would optimize the final candidate through the composed maps. We instead train each local correction directly against Q, using one differentiable flow evaluation per update (Equation 6; Figure 5). This keeps the off-policy update compact and lets executed replay actions provide training examples directly. The resulting noise-conditioned operator is then reusable across refinement levels.

Q5. Gaussian initialization and training–sampling state mismatch

Gaussian initialization gives the edited branch an independent proposal route without first following a base-flow trajectory. The task-conditioned frozen map still shapes every correction. The base branch supplies unedited alternatives, and the critic selects among the completed candidates. Training anchors local corrections to executed replay actions; sampling anchors each subsequent correction to the current estimate (Algorithm 1; Equation 5).

Q6. Is the decreasing-noise schedule intended as coarse-to-fine refinement?

The coarse-to-fine effect is progressive retention of the current estimate: as σ\sigma decreases from 0.750.75 to 0.250.25, its affine weight rises from 0.250.25 to 0.750.75, with less fresh noise. Table 8 supports the three-level choice: mean success across eight tasks is 77.75%, versus 70.50% for one correction at σ=0.5\sigma=0.5.

AuthorsOpenReview ↗

Response to Reviewer PSy9 (2/2)

W1. Citation placement and method names

Section 2.1 now places method names beside their citations, including ConRFT, VLA-RL, SimpleVLA-RL, πRL and FORCE. Section 2.4 similarly names EXPO, EXPO-FT and DSRL.

W2. Positioning relative to neighboring methods

Sections 2.3–2.4 distinguish adaptation through source noise (DSRL), completed actions (EXPO-FT), and interventions within generative policies. Section 3.2 specifies Q-Transport’s interior-state editing interface and the single-prediction gradient path detailed above.

W3. Meaning of the decision index Section 3.1 defines t as the environment-decision index, with one executed action chunk per decision. It differs from flow-step index k and is omitted in local flow equations.

W4. Clean-action predictor versus edited action

Figure 1(a), its caption, Equations 3–6 and Figure 5 now consistently use FψˉF_{\bar\psi} for the frozen predictor, x^σ\hat x_\sigma for its edited input, and a^\widehat a for the action output. The caption identifies the arrow from input to output as the frozen map.

W5. Order of the method presentation

Section 3.2 now introduces the frozen map, then the edit, followed by training and composition. Figure 1’s caption traces (a,ϵ)→xσ→x^σ→a^(a,\epsilon)\to x_\sigma\to\hat x_\sigma\to\widehat a and explains output reuse before fresh corruption. It refers separately to Figure 5 for the training gradient path.

W6. Separate sampling and online-update algorithms

Algorithms 1 and 2 now appear side by side on page 6, and Section 3.3 explains their connection. Algorithm 1 generates base and edited candidates; Algorithm 2 selects and executes an action, collects replay, and updates the critic, base policy and editor. The algorithm cross-references are also corrected.

W7. Number of independent training seeds

The real-robot results use one training seed (42; Appendix A.4), with 50 autonomous evaluation trials per task and configuration. The RoboTwin experiments use three training seeds per task and method (Appendix D, Table 9).

W8. Numerical results for no learned edits and a single correction

Appendix C, Table 8 now reports both ablations on eight real-robot tasks, with 50 autonomous trials per task and configuration. Below are the original two tasks and the unweighted mean over all eight:

SettingHard USBLAN cableEight-task mean
Without learned edits (3 levels)50% (25/50)54% (27/50)64.25%
Single correction (σ = 0.5)60% (30/50)64% (32/50)70.50%
Q-Transport (3 corrections)74% (37/50)78% (39/50)77.75%

The no-edit control sets Δx=0\Delta{x} = 0 and retains 16 candidates, with three successive rounds that each apply affine corruption followed by x-prediction. The single-correction control is trained and sampled only at σ=0.5\sigma = 0.5, also with 16 candidates. The variants share the BC initialization, interaction budget, critic schedule and successful-action base-policy updates (Section 4.2). Q-Transport improves the eight-task mean by 13.50 percentage points over no learned edits and 7.25 points over one correction.

Reviewer_hew6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  1. The paper studies online RL post-training of a pretrained flow-based VLA policy (π0.5) on real robots. The proposed Q-Transport learns a small stochastic editor that perturbs intermediate flow states and lets the frozen pretrained model map the edited state back to a complete action. A critic trains the editor and selects among candidate actions at inference.

  2. On five real-robot insertion tasks, Q-Transport achieves the best final success rate on every task, averaging 83.2% vs. 70.0% for the strongest baseline (EXPO-FT), and reduces late-stage human intervention.

Critical strengths

S1. The paper proposes a novel and effective way to adapt pretrained flow-based VLA policies with RL

Instead of editing the final action or the initial noise, the author uses a lightweight editor to perturb intermediate flow states, and utilizes a frozen pretrained model to map the edited state back to a complete action. This keeps the pretrained transport in the loop and lets value gradients reach the editor through a single frozen forward call, without differentiating through the sampler.

The idea works well in practice: on five real-robot tasks, the method achieves the best final success rate on every task (83.2% on average, +13.2 points over the strongest baseline).

Critical weaknesses

None

Questions for authors

Q1. Training efficiency.

Could the authors report consolidated efficiency numbers: total wall-clock training time per method per task, and per-decision inference latency for all methods? The methods differ substantially in candidate counts and per-episode update quotas, which makes a fair efficiency comparison difficult from the current reporting.

Q2. Task diversity.

All five evaluation tasks are insertion-type tasks on a single robot setup, and the two USB variants share the same scene. Have the authors considered transferring the method to other task families (e.g., pick-and-place, articulated or deformable objects, longer-horizon tasks)?

1 comment
AuthorsOpenReview ↗

Response to Reviewer hew6

We appreciate your careful review and valuable suggestions. Revisions to the main manuscript addressing your comments are marked in purple. In response to your feedback, we have consolidated the training-time, candidate-count, update-quota and inference-cost accounting, and supplemented our evaluation with four additional real-robot tasks and four non-insertion simulation tasks.

Q1. Training efficiency

Appendix A.5 and Table 4 separate the shared initialization from method-specific online adaptation. Per task, the shared stage takes approximately two hours of end-to-end robot data collection, including 20–30 minutes of HIL interaction, followed by five hours of SFT. Each adaptation method then runs for approximately two hours under the common interaction budget. The total budget is therefore approximately nine hours per method–task pair when shared initialization is included; the incremental adaptation stage is two hours. The table below reports the candidate count and the number of updates within each learner call. Latencies are range-derived estimates, expressed as midpoint ± half-width in milliseconds; the two values show the current implementation and the configuration after prefix de-duplication (Table 4).

MethodTotal budget/task (h)CandidatesCalls/episode × updates/callLatency estimate: current / dedup (ms)
Q-Transport9161 × (20Q + 1A + 1E)≈ 390 ± 80 / 120 ± 30
EXPO-FT9162 × (20Q + 1A + 1E)375 ± 75 / 110 ± 30
DSRL912 × (10Q + 1E)77.5 ± 22.5 / 77.5 ± 22.5
HG-DAgger915 × 1A65 ± 15 / 65 ± 15
SAC-flow913 × (20Q + 1E)77.5 ± 22.5 / 77.5 ± 22.5

Q, A and E denote critic, base-actor and editor updates. QT makes one learner call per episode, comprising 20 critic updates, one base-actor update and one editor update. EXPO-FT makes two calls with the same per-call quota. The remaining method-specific quotas are shown in the table. The online-BC reference uses three actor-only calls per episode and has an estimated decision latency of 65 ± 15 ms (Table 4).

Q2. Task diversity

We added four real-robot tasks beyond insertion: bottle placement in an open drawer, cup placement on a coaster, bottle-opener hanging, and bimanual sweeping. Figure 6 and Appendix B.4 describe the task procedures and success criteria. Table 7 reports final autonomous success over 50 trials per task with training seed 42:

TaskQTHG-DAggerEXPO-FT
Bottle in open drawer66%66%62%
Cup on coaster90%88%86%
Opener on hook76%52%62%
Bimanual sweeping54%56%50%

QT achieves a four-task mean success rate of 71.5%, compared with 65.5% for HG-DAgger and 65.0% for EXPO-FT. The tasks cover object placement, hanging, and coordinated bimanual sweeping. In the drawer setting, the drawer is already open and QT refines the precision-critical placement phase.

We also added RoboTwin 2.0 experiments on stamp_seal, place_a2b_left, stack_blocks_two, and hanging_mug, covering precision placement, relative placement, stacking and hanging (Appendix D, Tables 12–13). The protocol uses three training seeds per task and method and 100 paired evaluation seeds per task. At step-1020, after 340 online episodes, QT reaches 71.0% mean success, compared with 65.0% for EXPO, 51.3% for DSRL and 40.0% for SAC-Flow. Episodes start from task-specific key-stage states so that the online budget focuses on the precision-critical segment.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback: hew6 calls the approach "a novel and effective way" to adapt flow-based VLAs, VfFv highlights its "technically clean adaptation interface", and PSy9 notes "a consistent improvement across tasks". Their comments helped us clarify the design and strengthen the evaluation.

What this work studies. We focus on precision manipulation, where small errors in alignment or contact can prevent task completion. We study how online reward feedback can adapt a pretrained flow-based VLA to these tasks through targeted action corrections, while retaining its action prior and a short value-gradient path.

Contributions

  • Interior-state editing through the pretrained flow. Q-Transport learns a stochastic correction to an intermediate action state. The frozen x-prediction map transforms this correction into a complete action, so the pretrained velocity field shapes its effect (Section 3.2, Equations 3–5).
  • Replay-based local learning and composed sampling. The editor learns from executed actions through one differentiable x-prediction per update. At deployment, the shared noise-conditioned editor is reused across correction levels, and the critic selects among base and edited proposals. Successful actions separately improve the base policy through behavior cloning (Section 3.3).
  • Consistent gains on the original real-robot tasks. Q-Transport achieves the highest success rate on all five original tasks, averaging 83.2% versus 70.0% for EXPO-FT, and reduces late-stage human intervention (Table 1; Section 4.3).

Shared Questions

  • Do learned edits and repeated corrections contribute? Appendix C, Table 8 reports eight-task means of 64.25% without learned edits, 70.50% with one correction, and 77.75% for Q-Transport. All variants retain 16 candidates and share the interaction budget, critic schedule and base-policy updates. The no-edit control also retains the same three corruption–prediction rounds, isolating the benefit of learning the edit; the single-correction control evaluates the full schedule against a one-level alternative.
  • Does the evaluation extend beyond insertion? We added object placement, hanging and bimanual sweeping tasks. Across all nine real-robot tasks, Q-Transport averages 78.0% success versus 67.8% for EXPO-FT (Table 1). Four RoboTwin tasks further evaluate precision-critical key stages in placement, stacking, stamping and hanging: Q-Transport reaches 71.0% versus 65.0% for EXPO-FT, using three training seeds and 200 paired evaluation seeds per task (Appendix D, Tables 9–10).
  • How is computational cost accounted for? Table 4 separates approximately seven hours of shared data collection and SFT from two hours of online adaptation per task. It also consolidates candidate counts, learner-update quotas and range-derived inference-latency estimates for the compared methods (Appendix A.5).

Revisions to the PDF

  • Purple (hew6). Section 4.1 and Appendix A.5, Table 4: consolidated training-budget and inference-cost accounting.
  • Blue (PSy9). Sections 2–3, Figures 1 and 5, and Algorithms 1–2: clearer positioning, notation, gradient flow and sampling/update organization. Appendix A.4 specifies training seeds; Appendix C, Table 8 reports the requested mechanism ablations.
  • Orange (VfFv). Figure 2, Table 1 and Appendix B.4: additional real-robot tasks. Appendix D, Figure 8 and Tables 9–11: RoboTwin setup, results and learning-progress analyses.

Detailed answers are provided in the reviewer-specific replies below. Thank you again for your time and consideration.

AuthorsOpenReview ↗

Review and Reviewer-Author Discussion Summary (1/2)

Dear PCs, SACs, ACs, and Reviewers,

We thank the reviewers for their careful reading and constructive feedback. Their comments helped us clarify the editing interface, isolate its contributions, and broaden the evaluation.

Strength. We appreciate the reviewers' recognition of the following strengths.

  • An effective adaptation interface for flow-based VLAs. hew6 highlights editing intermediate flow states while retaining the pretrained model in action construction. VfFv describes the interface as technically clean, and PSy9 emphasizes the short gradient path and the geometry of the transported edit (hew6: Strength 1; VfFv: Strength 1; PSy9: Strengths).

  • Consistent gains on the original real-robot tasks. All three reviewers recognize the real-world results. On the five original tasks, Q-Transport reaches 83.2% mean success versus 70.0% for EXPO-FT, with lower late-stage human intervention (hew6: Summary and Strength 1; PSy9: Strengths; VfFv: Strength 2).

  • Useful component and computation analyses. PSy9 highlights the effect of base-policy consolidation, and VfFv notes the practical ablations and the trade-off in the number of flow steps (PSy9: Strengths; VfFv: Strength 3).

Concerns and Our Addressing. Below we group the main concerns raised during review and briefly summarize how we addressed them in the rebuttal and revised manuscript.

  • Concerns about experiment design.

    • (hew6: Question 2; VfFv: Weaknesses and Question 2) Real-robot task diversity beyond insertion.
      Our Addressing. We added bottle placement in an already-open drawer, cup placement on a coaster, bottle-opener hanging, and bimanual sweeping. QT averages 71.5% on these four tasks versus 65.5% for HG-DAgger and 65.0% for EXPO-FT. Across all nine real-robot tasks, QT averages 78.0% versus 67.8% for EXPO-FT (Section 4.1, Figure 2 and Table 1). The drawer task evaluates placement, with the drawer already open.

    • (VfFv: Question 2; hew6: Question 2) Broader simulation evaluation.
      Our Addressing. We added four RoboTwin 2.0 tasks covering stamping, relative placement, stacking and hanging. With three independent training seeds per method and task and 200 fixed, paired expert-feasible evaluation seeds per task, QT reaches 71.0% mean success after 340 online episodes, versus 65.0% for EXPO-FT, 51.3% for DSRL and 40.0% for SAC-flow (Appendix D, Figure 8 and Tables 9–10).

    • (VfFv: Weaknesses) Generalization and the scope of the evidence.
      Our Addressing. We distinguish within-task adaptation from broad generalization. Success on held-out real-robot initial configurations remains 18% (Section 4.4). The simulation starts at scripted precision-critical key stages in fixed-appearance scenes, with expert-infeasible seeds excluded before fixing the evaluation set.

    • (hew6: Question 1; VfFv: Weakness 2) Training and inference cost.
      Our Addressing. Appendix A.5 and Table 4 separate about seven hours of shared data collection and SFT from two hours of online adaptation per task. They report candidate counts and learner-update quotas: QT and EXPO-FT both use 16 candidates; QT uses one learner call per episode versus two for EXPO-FT. The reported range-derived QT latency estimates are 390 ± 80 ms currently and 120 ± 30 ms after prefix de-duplication; EXPO-FT is 375 ± 75 and 110 ± 30 ms, respectively (midpoint ± half-width). The real-robot setup uses eight training A100 80 GB GPUs and one for rollout collection.

AuthorsOpenReview ↗

Review and Reviewer-Author Discussion Summary (2/2)

  • Evidence for action-transport editing.

    • (PSy9: Weakness 8) Contribution of learned edits.
      Our Addressing. Appendix C, Table 8 now reports the no-edit control on eight real-robot tasks. It sets the edit to zero while retaining 16 candidates and the same three corruption–prediction rounds. The variants share BC initialization, interaction budget, critic schedule and base-policy updates. QT reaches 77.75% mean success versus 64.25% without learned edits, a 13.50-point gain at matched flow-evaluation count.

    • (PSy9: Weakness 8 and Question 6) Contribution of repeated corrections.
      Our Addressing. The single-correction control is trained and sampled only at noise level 0.5, retaining 16 candidates. Its eight-task mean is 70.50%, versus 77.75% for the three-level schedule, a 7.25-point gain (Section 4.2; Appendix C, Table 8). This tests the full schedule, including correction count and noise-level coverage, rather than isolating the ordering of the levels.

    • (PSy9: Questions 2 and 4) Local training versus composed deployment.
      Our Addressing. We clarify that each update trains one randomly sampled level through one x-prediction, while inference reuses the same noise-conditioned editor at 0.75 → 0.50 → 0.25. This supports direct learning from executed replay actions without reconstructing flow trajectories. The local objective does not backpropagate through the complete three-edit chain; the added schedule ablation evaluates composition empirically rather than establishing objective equivalence (Section 3.3, Equation 6; Algorithms 1–2).

  • Method design and reproducibility.

    • (PSy9: Strengths 1–2 and Question 1) Editing interface and gradient flow.
      Our Addressing. Section 3.2 separates the editor's action-state perturbation from the frozen x-prediction map that converts it to an action. Equation 4 describes how the velocity Jacobian shapes the local action change. Figure 5 and Appendix A.2 show the Q-gradient through the critic's action input and one frozen prediction to the editor, without unrolling the Euler sampler. Base-policy behavior cloning is updated separately (Equation 8).

    • (PSy9: Questions 3, 5 and 6) Re-corruption, initialization and decreasing noise.
      Our Addressing. Each prediction produces an action-space output; re-corruption carries that estimate into the next correction. Gaussian initialization supplies an independent edited proposal route alongside the base branch. As noise decreases from 0.75 to 0.25, the affine weight on the current estimate increases from 0.25 to 0.75. This explains the refinement rationale without asserting equality of training and sampling state distributions (Equation 5; Algorithm 1; PSy9 response, Q3–Q6).

    • (PSy9: Weaknesses 1–6) Positioning, notation and presentation.
      Our Addressing. Section 2 names neighboring methods beside their citations and distinguishes source-noise, completed-action and interior-state adaptation. Sections 3.1–3.2 define the decision index and introduce the frozen map before the edit. Figure 1 and Equations 3–6 distinguish the predictor, edited input and output action; Figure 5 shows training gradients. Algorithms 1–2 separately present candidate sampling and online updates.

    • (PSy9: Weakness 7) Independent training seeds versus evaluation trials.
      Our Addressing. Appendix A.4 specifies that reported real-robot results use one training seed, 42, with 50 autonomous evaluation trials per task and configuration. RoboTwin uses three independent training seeds per task and method, each evaluated on the same 200 fixed expert-feasible seeds; Table 10 reports their means (Appendix D, Table 9).

    • (VfFv: Question 1) Relation to CLARE and continual learning.
      Our Addressing. We distinguish CLARE's sequential task acquisition and retention from QT's reward-guided adaptation within each task. Each QT task starts from its own initialization. The FWT, NBT and AUC quantities in Appendix D.4 measure learning progress and stability within a task(VfFv response, Q1).

The added evaluations support the contributions of learned interior edits and repeated refinement, and extend the evidence beyond the original insertion tasks within the stated protocols. We hope this summary of the reviews and our responses helps inform the decision. We thank the reviewers for their constructive feedback and the Area Chair, Senior Area Chair, and Program Chairs for their time and consideration.

Sincerely,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.