acceptodds
Under review as a conference paper at ICLR 2027

Rerouting the Flow: Versatile Test-Time VLA Adaptation via Source Distribution Learning

Abstract

Vision-language-action (VLA) agents acquire broad manipulation capabilities from large-scale robot data, yet remain brittle under distribution shifts at deployment. Our diagnostic analyses show that failed rollouts can retain task-directed behavior and that denoising perturbations occasionally yield success, suggesting that some failures reflect unreliable access to pretrained capabilities rather than their absence. Motivated by this insight, we propose ReFlowTTA, a versatile test-time adaptation framework that learns the source distribution of initial noise of a flow-based VLA, shifting probability mass toward states associated with better execution outcomes. Task feedback and optional human corrections guide this shift, allowing accumulated experience to shape subsequent sampling and make successful behaviors more accessible. This turns adaptation across objects, scenes, and robot embodiments into a common problem of learning where generation should begin, without relearning how actions are produced. Across three simulation benchmarks and two real-world robot platforms, ReFlowTTA consistently outperforms representative test-time adaptation methods with limited deployment interactions. Further analyses show that adaptation stabilizes and that the gains depend on the direction of the learned source shift instead of on generic perturbations. We release the core implementation at an anonymous repository.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Test-Time Adaptation, Vision-Language-Action Agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_p1XAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReFlowTTA, a test-time adaptation framework for flow-based VLA models that optimizes a low-dimensional initial-noise distribution while keeping the pretrained policy frozen. Experiments across three simulation benchmarks and two real-world robot platforms report improved task success rates under various out-of-distribution conditions.

Critical strengths
  1. The paper presents an interesting perspective on VLA adaptation by modifying the source distribution of action generation rather than updating pretrained model parameters.

  2. The method provides a unified adaptation mechanism for different distribution shifts, with optional human corrections incorporated through inverse flow.

  3. The experimental evaluation is relatively comprehensive, covering multiple simulation benchmarks, cross-embodiment transfer, and real-world robotic experiments, with consistent improvements over the evaluated baselines.

Critical weaknesses
  1. Dependence on Dense Task Feedback. The method relies on dense chunk-level rewards, using privileged geometric information in simulation and an external reward model in real-world experiments. Its effectiveness under sparse, delayed, or noisy feedback remains unclear, limiting the practicality of the proposed test-time adaptation framework.

  2. Insufficient Mechanism Ablations. While the paper investigates source-space dimensionality and perturbation directionality, it lacks comparisons with simpler source-distribution optimization strategies under matched interaction budgets. It is therefore difficult to determine whether the gains specifically arise from the proposed PPO-based optimization.

  3. Limited Generalizability. All experiments rely on the same π0.5 backbone, leaving the applicability to other flow-based VLA architectures unverified. Moreover, the shared, state-independent source mean may limit adaptation to heterogeneous states or changing deployment conditions.

  4. Limited Analysis of Human Corrections. The proposed inverse-flow mechanism is an interesting component, but its reconstruction accuracy, sensitivity to numerical integration, and individual contribution to adaptation performance are not sufficiently evaluated.

Questions for authors
  1. How does ReFlowTTA perform with sparse binary rewards or noisy reward-model feedback instead of dense task-progress signals?

  2. How accurately does inverse flow recover source states from human corrections, and how sensitive are the resulting improvements to inversion error and the number of correction samples?

5 comments
AuthorsOpenReview ↗

Responses to Reviewer p1XA (1/3)

We thank Reviewer p1XA for the insightful feedback and valuable suggestions. We address each comment below and highlight the corresponding revisions in red in the updated manuscript. All references to lines, tables, and sections refer to this revised version.


W1: Its effectiveness under sparse, delayed, or noisy feedback remains unclear.

A1: We test three feedback settings:

i) Sparse: binary rewards, with 1 only for the chunk that completes the task and 0 otherwise;

ii) Delayed: rewards delivered Δ\Delta chunks later, remaining paired with their original chunks and used for updates only after arrival;

iii) Noisy: r~t=rt+ϵt\tilde r_t=r_t+\epsilon_t, where ϵt∼N(0,(ασr)2)\epsilon_t\sim\mathcal N(0,(\alpha\sigma_r)^2) and σr\sigma_r is the standard deviation of the unperturbed rewards within each rollout batch. Results on ManiSkill3 (§4.2) are as follows:

Feedback settingOnline SR@15 (%) ↑Final SR (%) ↑
Sparse binary reward41.745.2
Delayed, Δ=2\Delta=247.853.4
Delayed, Δ=4\Delta=447.553.1
Delayed, Δ=6\Delta=646.953.2
Noisy, α=5\alpha=5 %49.153.1
Noisy, α=7\alpha=7 %49.353.4
Noisy, α=10\alpha=10 %49.053.3
ReFlowTTA (original feedback)49.2$_{\pm 0.21}$53.6_{\pm 0.15}\

i) Sparse: Online and final SR decrease by 7.5% and 8.4%, respectively, while Final SR remains above Base VLA (45.2% vs. 33.8%), demonstrating that dense feedback is an important design choice for rapid adaptation.

ii) Delayed: Final performance remains stable, but postponed updates slightly slow online adaptation, with Online SR@15 dropping by up to 2.3%.

iii) Noisy: Online and final SR remain stable under reward noise, deviating by at most 0.2% and 0.5%, respectively.

We have added this comparison to Appendix D.4 and discussed the associated limitation in Appendix G (Lines 1355–1370, 1488–1490).

W2: it lacks comparisons with simpler source-distribution optimization strategies under matched interaction budgets.

A2: We adopt PPO to balance reward improvement and update stability under limited deployment feedback. Its clipped objective discourages overly aggressive updates to the source distribution. We compare two simpler strategies under the same setting:

i) REINFORCE [1], which updates the mean through reward-weighted log-likelihood maximization;

ii) a CEM-style update [2], which moves the mean toward the average of high-reward source samples. Results on ManiSkill3 (§4.2) are as follows:

Optimization strategyOnline SR@15 (%) ↑Final SR (%) ↑
REINFORCE45.949.2
CEM-style mean update42.546.4
Clipped PPO (ours)49.2$_{\pm 0.21}$53.6_{\pm 0.15}\

Compared with REINFORCE, clipped PPO improves Online SR@15 and Final SR by 3.3% and 4.4%, respectively; compared with the CEM-style update, the gains are 6.7% and 7.2%. These results show that clipped PPO is more effective for source-mean learning under the matched interaction budget.

We have added this comparison to Appendix D.6 (Lines 1386–1397).

W3-1: leaving the applicability to other flow-based VLA architectures unverified.

A3-1: Because ReFlowTTA updates only the source-noise mean while keeping the pretrained VLA frozen, its adaptation rule is not tied to a particular flow-based VLA backbone. To test its applicability to other flow-based VLA architectures, we evaluate it on GR00T N1.7 [3] and LingBot-VLA-4B [4], comparing the frozen base policy, a DSRL-based variant, and ReFlowTTA.

MethodGR00T N1.7: Online SR@15 (%) ↑GR00T N1.7: Final SR (%) ↑LingBot-VLA: Online SR@15 (%) ↑LingBot-VLA: Final SR (%) ↑
Base VLA–36.9–34.2
DSRL-based adaptation38.540.437.540.3
ReFlowTTA48.752.143.848.6

Compared with DSRL-based adaptation, ReFlowTTA improves Online SR@15 and Final SR by 10.2% and 11.7% on GR00T N1.7, and by 6.3% and 8.3% on LingBot-VLA. These results demonstrate the applicability of source-mean adaptation across different flow-based VLA backbones.

We have added this comparison to Appendix D.7 (Lines 1401–1414).

AuthorsOpenReview ↗

Responses to Reviewer p1XA (2/3)

W3-2: the shared, state-independent source mean may limit adaptation to heterogeneous states or changing deployment conditions.

A3-2: The shared mean pools feedback across states to enable efficient adaptation under a limited interaction budget. Since the frozen VLA conditions action generation on the current observation and robot state, the same source mean still allows state-dependent behaviors. We compare this design with a state-conditioned mean variant following DSRL’s conditioning architecture, under otherwise matched adaptation settings. Results on ManiSkill3 (§4.2) are as follows:

AgentOnline SR@15 (%) ↑Final SR (%) ↑
Base VLA–33.8
DSRL34.338.1
ReFlowTTA (state-conditioned mean)38.9$_{\pm 0.86}$42.6$_{\pm 0.94}$
ReFlowTTA (shared mean)49.2$_{\pm 0.21}$53.6_{\pm 0.15}\

The shared mean achieves higher success rates with lower variability across runs, supporting more effective and stable adaptation than the state-conditioned variant under the tested interaction budget.

We have added this comparison and analysis to Appendix D.1 (Lines 1315–1323).

W4: "reconstruction accuracy, sensitivity to numerical integration, and individual contribution to adaptation performance are not sufficiently evaluated."

A4: i) Reconstruction accuracy and inversion cost. We evaluate 400 correction chunks from the real-world experiments (10 per task, 20 tasks per platform), passing the recovered source noise through the actual four-step forward decoder and comparing the reconstructed actions with the original corrections. As a reference, we additionally evaluate a computationally more expensive fixed-point inversion method [5] for the same decoder, comparing both reconstruction accuracy and inversion time:

MethodMAE on PiPER (deg) ↓MAE on ARX X5 (deg) ↓Inversion time (ms) ↓
Fixed-point inversion2.22.41748.5
Ten-step reverse Euler (ours)2.73.1252.4

Ten-step reverse Euler yields MAEs of 2.7° and 3.1° on PiPER and ARX X5, respectively, compared with 2.2° and 2.4° for fixed-point inversion, while requiring 252.4 ms rather than 1748.5 ms (6.9× speedup). These results quantify the reconstruction accuracy and computational cost of the reverse-flow approximation.

We have added this analysis to Appendix D.3 (Lines 1340–1351).

ii) Sensitivity to numerical integration. Recovered source coordinates provide auxiliary supervision for shared-mean learning alongside execution feedback, without requiring exact reconstruction of every corrective action. We compare nn-step reverse Euler with fixed-point inversion [5] under the same human-correction budget on PiPER. Results are as follows:

Inversion settingReconstruction MAE (deg) ↓Online SR@15 (%) ↑Final SR (%) ↑
Without human corrections–51.658.6
Reverse Euler, n=3n=35.856.862.4
Reverse Euler, n=5n=53.557.062.5
Reverse Euler, n=10n=10 (ours)2.757.362.9
Fixed-point inversion [5]2.257.462.5

Across the tested inversion settings, online and final SR differ by at most 0.6% and 0.5%, respectively, indicating low sensitivity to numerical integration.

We have added this analysis to Appendix D.8 (Lines 1418–1428).

iii) Contribution of human corrections. Table 4 already shows that removing human corrections reduces Online SR@15 and Final SR on PiPER by 5.7% and 4.3%, respectively, demonstrating the additional gains from correction supervision (Lines 378–392, 413–417). We further evaluate sensitivity to correction quantity by capping the number of corrections at 40%, 60%, and 80% of the original count for each task. Results on PiPER are as follows:

Correction budget (% of original count)Online SR@15 (%) ↑Final SR (%) ↑
0%51.658.6
40%53.559.2
60%54.961.1
80%56.661.8
100% (original setting)57.362.9

Compared with the full correction budget, using 40% lowers Online SR@15 and Final SR by 3.8% and 3.7%, respectively; both metrics increase with the correction budget. This demonstrates the contribution of human corrections to adaptation efficiency and final performance.

We have added this analysis to Appendix D.9 (Lines 1431–1443).

AuthorsOpenReview ↗

Responses to Reviewer p1XA (3/3)

Q1: "How does ReFlowTTA perform with sparse binary rewards or noisy reward-model feedback instead of dense task-progress signals?"

A1: Please see W1.

Q2-1: "How accurately does inverse flow recover source states from human corrections?"

A2-1: Please see W4(i).

Q2-2: "how sensitive are the resulting improvements to inversion error and the number of correction samples?"

A2-2: Please see W4(ii) for sensitivity to inversion error and W4(iii) for sensitivity to the number of correction samples.

[1] Black et al., Training Diffusion Models with Reinforcement Learning. In ICLR 2024.

[2] Pinneri et al., Sample-efficient Cross-Entropy Method for Real-time Planning. In CoRL 2020.

[3] NVIDIA Isaac GR00T (N1.7) 2026.

[4] Wei Wu et al. A Pragmatic VLA Foundation Model. arXiv:2601.18692, 2026.

[5] Lu et al. UniSteer: Unified Noise Steering for Efficient Human-Guided VLA Adaptation. arXiv:2605.10821.

Reviewer_fvW1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReFlowTTA, a test-time adaptation method for flow-based VLA policies that leaves the pretrained model frozen and changes only the mean of its initial-noise distribution. The mean is learned in a fixed 64-dimensional subspace; an orthogonal Gaussian component preserves the full source covariance. Chunk-level execution feedback updates the source mean with a PPO-style objective, while optional human corrections are inverted through the frozen flow and projected into the same subspace for likelihood-based updates. The mean is retained across episodes under a given deployment condition.

The method is evaluated on ManiSkill3, LIBERO, and LIBERO-Plus, as well as on PiPER and ARX X5 robots. The paper also reports ablations of subspace dimensionality and source-shift direction, adaptation dynamics, and failure cases.

Critical strengths
  1. A simple and well-defined adaptation interface for a frozen VLA. ReFlowTTA changes only the mean of a low-dimensional Gaussian source distribution without modifying the pretrained flow model. Under the stated orthonormal-basis construction, Equation (3) preserves the full source covariance. The same interface can also incorporate optional human corrections through inverse flow and source-space likelihood optimization. This is a practical alternative to adapting the entire action model from a small deployment dataset.

  2. Encouraging results across several deployment shifts, supported by useful ablations. The method outperforms the tested baselines on ManiSkill3, cross-embodiment LIBERO, LIBERO-Plus, and both real-robot platforms. The dimensionality sweep in Table 5 and the source-direction controls in Table 6 are particularly helpful: reversing or randomizing the learned shift substantially reduces performance. The failure cases also make clear that source rerouting cannot recover a skill absent from the pretrained policy.

Critical weaknesses
  1. The actual source-mean optimization procedure is not sufficiently specified. Algorithm 1 shows a single optimizer step after collecting each batch. If the method takes only one gradient step starting from the same distribution that generated the batch, the likelihood ratio in Equation (6) is initially one, so PPO clipping has no effect on that update's gradient. It is therefore unclear how the clipping-parameter differences in Table 12 arise. Multiple optimization steps or data reuse could resolve this, but these are not stated. For the human-correction path, the value of the loss weight and the number of corrections are also missing. The paper uses ten Euler steps for inverse flow and four steps for forward generation, followed by low-dimensional projection, without measuring how faithfully corrected actions can be reconstructed.

  2. The comparisons do not fully establish equal feedback and resource budgets, and the uncertainty reporting is uneven. The simulation reward is specified in Appendix C.6, but it uses stage-dependent geometric progress, while real-world adaptation depends on RoboMeter's learned progress estimates. It is not clear which learning baselines have access to equivalent feedback or how the reward computation affects the practical cost of adaptation. Appendix C.7 also states that DSRL uses ten additional replay-buffer warm-start episodes beyond the reported adaptation budget, and the PDF baseline uses an autoregressive action-decoding implementation rather than the same flow head. These differences are disclosed, but complicate like-for-like comparisons. Tables 1–3 primarily give uncertainty for ReFlowTTA rather than the competing learning methods, and the real-world results in Table 4 lack run-to-run uncertainty. The real-robot cross-embodiment experiment additionally uses a fixed joint-coordinate alignment for ARX X5 (Appendix C.4), whose application across baselines should be made explicit.

Questions for authors
  1. Optimization and PPO clipping. How many optimizer steps are taken on each batch of four chunks, and is the batch reused? Which optimizer is used, and are rewards normalized or centered? If there is only one step from the behavior distribution, why does changing the clipping threshold affect the results in Table 12? A short description of the exact training loop would resolve this.

  2. Feedback and compute comparability. Do PDF and DSRL receive the same chunk-level dense rewards, including the geometric progress signals in simulation? Please report total deployment interactions, including DSRL's ten warm-start episodes, and the adaptation-time cost of reward estimation, model inference, and optimization. This would help substantiate the claim that ReFlowTTA is lightweight under comparable resources.

  3. Human-correction mechanism. What is the value of λ_H, how many corrections are provided per task or adaptation run, and what triggers them? Since corrected action chunks are inverted numerically and then projected to a 64-dimensional subspace, can the authors measure reconstruction error after mapping them back through the forward flow, or provide an ablation that isolates the effect of this projection?

  4. Uncertainty and cross-embodiment evaluation. Can the authors report variability for the learning baselines and repeated adaptation runs in the real-world experiments? Is the joint-coordinate alignment in Equations (19)–(21) applied identically to all methods on ARX X5, and how are its fixed offsets selected? These details would make the robustness and transfer claims easier to assess.

5 comments
AuthorsOpenReview ↗

Responses to Reviewer fvW1 (1/3)

We appreciate Reviewer fvW1’s thoughtful comments and helpful suggestions. Our point-by-point responses follow, and the associated manuscript changes are marked in red. References to lines, tables, and sections throughout these responses are based on the revised manuscript.


W1-1: "Algorithm 1 shows a single optimizer step after collecting each batch... It is therefore unclear how the clipping-parameter differences in Table 12 arise."

A1-1: We apologize for omitting this detail. We perform two optimizer steps on the same rollout batch to update μϕ\mu_\phi. Clipping can therefore affect the second update, explaining the differences across clipping thresholds in Table 12.

We have clarified the two-step update procedure in §3.2 and Algorithm 1 (Lines 243–245, 821–836).

W1-2: "the value of the loss weight and the number of corrections are also missing."

A1-2: We thank the reviewer for highlighting these missing details. The table below reports the average correction usage per task over 15 real-world adaptation episodes, together with the supervision weight λH\lambda_H:

Per task (15-episode adaptation)PiPERARX X5
Corrections / total chunks63.4 / 270.3 (23.5%)91.7 / 289.6 (31.7%)
Supervision weight λH\lambda_H0.30.4

We have added these details to Appendix C.4 (Lines 1188–1192).

W1-3: "...without measuring how faithfully corrected actions can be reconstructed."

A1-3: We evaluate action reconstruction on 400 correction chunks from the real-world experiments (10 per task, 20 tasks per platform), passing the recovered source noise through the actual four-step forward decoder and comparing the reconstructed actions with the original corrections. As a reference, we also evaluate fixed-point inversion [1] of the same decoder and report inversion time for both methods:

MethodMAE on PiPER (deg) ↓MAE on ARX X5 (deg) ↓Inversion time (ms) ↓
Fixed-point inversion2.22.41748.5
Ten-step reverse Euler (ours)2.73.1252.4

Ten-step reverse Euler yields reconstruction MAEs of 2.7° and 3.1°, exceeding the fixed-point reference by 0.5° and 0.7°, respectively, while reducing inversion time from 1748.5 to 252.4 ms (6.9× speedup). This accuracy–cost trade-off supports its use as a practical approximation for recovering correction targets.

We have added this reconstruction error and inversion time analysis to Appendix D.3 (Lines 1340–1351).

W2-1: "It is not clear which learning baselines have access to equivalent feedback..."

A2-1: All learning-based baselines receive the same reward information.

i) For PDF, we aggregate the same chunk-level dense rewards into the scalar delayed feedback rr used in its adaptation loss.

ii) DSRL receives the same dense reward for each chunk-level transition stored in its replay buffer.

We have clarified these details in Appendix C.7 (Lines 1277–1278, 1284–1286).

W2-2: "...how the reward computation affects the practical cost of adaptation."

A2-2: i) Simulation: Rewards are computed from distance-based geometric progress (Lines 1236–1241, 1256–1259), taking only microseconds per chunk and requiring no additional GPU memory.

ii) Real-world: Rewards are derived from task-progress estimates provided by RoboMeter (Lines 1261–1266). The average inference time per estimate and GPU memory usage are reported below:

Average time per estimate (ms)Average GPU memory usage (GB)
168.28.89

We have added these reward-computation costs to Appendix C.6 (Lines 1268–1270).

AuthorsOpenReview ↗

Responses to Reviewer fvW1 (2/3)

W2-3: Some differences between baseline protocols complicate like-for-like comparison.

A2-3: i) DSRL: We follow DSRL’s original protocol, which gives it ten additional warm-start episodes before the 15-episode adaptation (Lines 1274–1277). To assess their effect, we also evaluate DSRL without warm-start episodes:

MethodOnline SR@15 (%) ↑Final SR (%) ↑
DSRL with warm-start34.338.1
DSRL without warm-start33.935.4
ReFlowTTA49.2$_{\pm 0.21}$53.6_{\pm 0.15}\

Removing warm-start episodes lowers DSRL’s Final SR by 2.7%, while ReFlowTTA performs better without these additional interactions.

ii) PDF: PDF and ReFlowTTA use different action representations, so a strictly architecture-matched comparison is not possible. We therefore minimize this difference by matching the pretrained backbone, task-specific training data, and evaluation protocol (Lines 1279–1284). Matching the backbone while retaining method-specific action heads is a common modular comparison strategy in VLA research [2].

We have added the DSRL warm-start comparison to Appendix D.5 (Lines 1374–1382).

W2-4: Uncertainty reporting is uneven across Tables 1–4.

A2-4: We have added run-to-run standard deviations for the simulation results in Tables 1–3 and for real-world Final SR in Table 4. For real-world Online SR@15, some evaluation records were not retained, and repeating these physical-robot evaluations is extremely time-consuming. We therefore could not complete the repeated evaluations within the rebuttal period.

W2-5: ...a fixed joint-coordinate alignment for ARX X5 (Appendix C.4), whose application across baselines should be made explicit.

A2-5: The joint-coordinate alignment is not a ReFlowTTA-specific component; it is a coordinate transformation shared by all methods in the ARX X5 evaluation.

We have clarified the joint-coordinate alignment protocol in Appendix C.4 (Lines 1110–1112).

Q1-1: "How many optimizer steps are taken on each batch of four chunks, and is the batch reused?"

A1-1: Please see W1-1.

Q1-2: "Which optimizer is used, and are rewards normalized or centered?"

A1-2: i) Optimizer: We use AdamW. ii) Reward processing: We use the chunk-level reward directly as the advantage estimate, A^t=rt\hat A_t = r_t, without additional normalization or mean-centering.

We have clarified the optimizer and reward processing in Appendix C.5 (Lines 1207–1209).

Q1-3: "why does changing the clipping threshold affect the results in Table 12?"

A1-3: Please see W1-1.

Q2-1: "Do PDF and DSRL receive the same chunk-level dense rewards, including the geometric progress signals in simulation?"

A2-1: Please see W2-1.

Q2-2: "Please report total deployment interactions, including DSRL's ten warm-start episodes, and the adaptation-time cost of reward estimation, model inference, and optimization."

A2-2: i) Total interactions: PDF and ReFlowTTA each use a budget of 15 deployment episodes, while DSRL uses 25 episodes, including ten additional warm-start episodes. The table below reports the total action chunks across 20 tasks per platform; the DSRL entry includes both adaptation and warm-start chunks.

MethodPiPER: total chunksARX X5: total chunks
PDF59096173
DSRL5317 + 3611 (warm-start)5634 + 3845 (warm-start)
ReFlowTTA54065792

ii) Reward estimation time: Please see W2-2. iii) Model inference and optimization time: We measure the average inference time per action chunk and optimization time per update for all three methods:

MethodInference time (ms/chunk)Optimization time (ms/update)
PDF794.54.9
DSRL52.31653.4
ReFlowTTA40.32.2

ReFlowTTA takes 2.2 ms/update, only 0.13% of DSRL’s update time and 45% of PDF’s. PDF trades inference speed for fast updates by caching features and logits during the forward pass, whereas ReFlowTTA introduces no additional inference-time computation. ReFlowTTA's update also takes only 5.5% of its own per-chunk inference time. These results show that ReFlowTTA adapts lightly by updating only the low-dimensional source mean (Lines 1296–1300).

We have added these interaction counts and timing results to Appendix C.7 (Table 13; Lines 1242–1254, 1292–1300).

AuthorsOpenReview ↗

Responses to Reviewer fvW1 (3/3)

Q3-1: "What is the value of λ_H, how many corrections are provided per task or adaptation run, and what triggers them?"

A3-1: i) λH\lambda_H and correction counts: Please see W1-2.

ii) Intervention trigger: Human intervention is initiated when, after a chunk ends, the robot has clearly deviated from the intended task execution. We reset it to its pre-chunk state and execute a corrective action chunk.

We have clarified the intervention protocol in Appendix C.4 (Lines 1166–1168).

Q3-2: "can the authors measure reconstruction error after mapping them back through the forward flow, or provide an ablation that isolates the effect of this projection?"

A3-2: i) Reconstruction error: Please see W1-3.

ii) Effect of projection: Under our source-distribution parameterization, projected supervision is fully equivalent to full-space supervision for source-mean learning. Since B⊤B=IB^\top B=I and mtH=B⊤vec(ηtH)m_t^H=B^\top\mathrm{vec}(\eta_t^H) (Eq. 9), we have ∥∥vec(ηtH)−Bμϕ∥∥2=∥∥mtH−μϕ∥∥2+∥∥(I−BB⊤)vec(ηtH)∥∥2\|\|\mathrm{vec}(\eta_t^H)-B\mu_\phi\|\|^2=\|\|m_t^H-\mu_\phi\|\|^2+\|\|(I-BB^\top)\mathrm{vec}(\eta_t^H)\|\|^2. The second term is independent of μϕ\mu_\phi, so the full-space and projected negative log-likelihoods differ only by a constant and yield exactly the same gradient. Thus, projection does not affect the correction supervision signal for learning the source mean.

We have added this analysis to Appendix C.4 (Lines 1180–1187).

Q4-1: "Can the authors report variability for the learning baselines and repeated adaptation runs in the real-world experiments?"

A4-1: Please see W2-4.

Q4-2: "Is the joint-coordinate alignment in Equations (19)–(21) applied identically to all methods on ARX X5, and how are its fixed offsets selected?"

A4-2: i) Application across methods: Please see W2-5.

ii) Fixed-offset selection: Our scope is to improve OOD task execution by making the frozen VLA’s existing capabilities more reliably accessible (Lines 1447–1450). Since the base policy is fine-tuned on PiPER, opposite joint-coordinate conventions and large offsets on ARX X5 can hinder access to its learned action-generation regime. We therefore apply fixed joint-coordinate alignment, selecting offsets based on approximately corresponding initial configurations (Lines 1093–1109). Substantial physical differences remain in joint axes, geometry, and kinematics (Lines 1113–1115): Fig. 8 illustrates the resulting grasp failures and ReFlowTTA’s recovery, while Table 4 shows that ReFlowTTA outperforms all baselines on ARX X5 under the same alignment.

[1] Lu et al. UniSteer: Unified Noise Steering for Efficient Human-Guided VLA Adaptation. arXiv:2605.10821.

[2] StarVLA Community. StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing. arXiv:2604.05014.

Reviewer_CqKSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to use limited deployment interaction to improve a pretrained vision-language-action model’s performance when the objects, scenes, or robot embodiment differ from the training distribution, while keeping the VLA parameters frozen. The authors propose ReFlowTTA, which limits adaptation to the low-dimensional mean of the initial noise distribution used for action generation, while keeping the full source distribution’s covariance equal to the identity. The mean is updated using action-block-level rewards and value-free PPO, and is retained across steps and episodes under the same deployment condition. Optional human corrections are converted into noise-space supervision through inverse flow integration. The authors evaluate the method on ManiSkill3, LIBERO, LIBERO-Plus, and two real robot platforms, reporting sustained improvements over the base policy and the adaptation methods included in the comparisons. Dimension ablations, displacement-direction ablations, and analyses of the adaptation process provide further support for the effectiveness of low-dimensional mean learning under limited feedback. The paper’s main contribution is a highly parameter-efficient implementation of the existing noise-space adaptation paradigm, supported by broad empirical evaluation.

Critical strengths

A highly parameter-efficient adaptation interface delivers consistent and practically meaningful improvements under diverse deployment shifts. The method updates only the default 64-dimensional source-noise mean while keeping the VLA frozen, yet outperforms the reported comparison methods on three simulation benchmarks and two real-world platforms. For example, the final success rate on ManiSkill3 rises from 33.8% for the base policy to 53.6%. Without human corrections, the gains over the strongest reported baselines on the real PiPER and ARX X5 platforms are 14.3 and 17.8 percentage points, respectively (Tables 1 and 4). These results support the practical value of low-complexity noise-distribution adaptation and are the main reason I lean toward acceptance. The core ablations provide evidence for the design beyond performance comparisons. Table 5 shows that a subspace that is too small yields almost no improvement, while adaptation in the full space also performs worse than adaptation in a moderately sized low-dimensional space. This supports a trade-off between adaptation flexibility and learning difficulty under limited feedback. Table 6 shows that retaining the learned direction preserves some of the gains, while randomizing or reversing it substantially reduces performance. This indicates that the gains depend on structure learned from deployment feedback and cannot be attributed merely to adding random perturbations. Together, these analyses make the empirical contribution more complete.

Critical weaknesses

The relationship to the closest prior work is not sufficiently discussed, which affects both novelty positioning and attribution of the design’s benefits. DSRL [1] already adapts frozen diffusion/flow policies through noise-space reinforcement learning, and Appendix C.5 also includes a design for reducing the dimensionality of noise learning. UniSteer [2] already inverts human corrections into noise-space supervision and combines this with reinforcement learning to update the same noise policy. FlowDAgger [3] likewise adapts frozen generative policies through action inversion. The paper does not discuss the latter two works. The paper’s clearest contribution should therefore be framed around the specific combination of a fixed random subspace, preservation of the full covariance, and a mean shared across episodes. The current comparison with DSRL changes the policy parameterization, state conditioning, and optimization method at the same time, so it does not isolate the contribution of the shared low-dimensional mean. This limits the case for the paper as a new perspective, though it does not negate the empirical value of its specific design. Some mechanistic interpretations and claims about human-feedback efficiency lack corresponding evidence. The two-task perturbation experiment in Appendix B.2 supports the possibility that the frozen model can occasionally produce successful behavior. However, it changes both the initial noise and the intermediate denoising process, so it does not directly establish the mechanism of the mean-adaptation interface proposed in this paper. Nor is it sufficient to attribute failures generally to difficulty accessing existing capabilities. Although Table 4 reports gains from the human-feedback pathway, the reverse integration used in Appendix C.4 is not the exact discrete inverse of the actual four-step forward decoder. The paper does not report the action reconstruction error after decoding the inverted noise again; the number of corrections, intervention duration, and supervision weight are also unclear. In addition, dense progress rewards are an important assumption behind the main results. These gaps reduce confidence in the mechanistic interpretation, feedback efficiency, and scope of applicability, and should be distinguished from the main performance results.

  • [1] Wagenmaker et al. Steering Your Diffusion Policy with Latent Space Reinforcement Learning. CoRL 2025, arXiv:2506.15799.
  • [2] Lu et al. UniSteer: Unified Noise Steering for Efficient Human-Guided VLA Adaptation. arXiv:2605.10821.
  • [3] Murray et al. FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space. arXiv:2607.08877.
Questions for authors

Contribution positioning and attribution of design choices: Please clarify ReFlowTTA’s independent contribution relative to DSRL, UniSteer, and FlowDAgger. Are there results comparing a shared mean with a state-conditioned mean under the same reward, optimization method, feedback budget, and covariance setting? Such results would help determine whether the gains primarily come from low-dimensional parameterization, sharing across states, or differences in the optimization method. Reward and evaluation protocol: How are the approach/placement rewards in Appendix C.6 used across different skills and long-horizon tasks in LIBERO, and how are stage transitions determined? Do all learning-based baselines receive the same reward information? Please also clarify the definition of a deployment condition, the scope over which the mean is reset, and whether independent evaluation episodes are completely excluded from adaptation updates. These details matter for interpreting the claims about limited interaction budgets and cross-condition adaptation. Human-correction budget and inversion quality: Please report the number of correction action blocks actually used per task, intervention duration, trigger rules, and the setting of λH\lambda_H. Have you measured the action error after the inverted noise is passed through the actual four-step forward decoder, as well as the effect of low-dimensional projection on correction supervision? These results would directly inform the assessment of the reliability and efficiency of the human-feedback pathway.

3 comments
AuthorsOpenReview ↗

Responses to Reviewer CqKS (1/3)

We thank Reviewer CqKS for the careful review and constructive feedback. We respond to each point below, with the corresponding revisions highlighted in red in the revised manuscript. All line, table, and section references refer to the revised version.


W1-1: "The paper’s clearest contribution should therefore be framed around..."

A1-1: We thank the reviewer for highlighting these design choices. ReFlowTTA provides a lightweight, persistent source-distribution learning interface for adapting frozen VLAs from limited deployment feedback. Specifically, it confines the learnable source mean to a fixed random subspace while preserving the full source covariance, reducing the number of adaptation parameters while retaining full-dimensional source sampling. The mean is shared across states and retained across steps and episodes within the same deployment condition, allowing accumulated chunk-level feedback to continually guide subsequent action generation.

We have clarified this contribution in the Introduction of the revised manuscript (Lines 079–082).

W1-2: "The relationship to the closest prior work is not sufficiently discussed (e.g., DSRL, UniSteer, and FlowDAgger)..."

A1-2: i) ReFlowTTA differs from DSRL [1] in three key aspects:

a) Where and Why Dimensionality Is Reduced. DSRL reduces the dimensionality of the sampled noise itself to make its training tractable. ReFlowTTA samples full-dimensional noise and confines only the learnable mean to a fixed 64-dimensional subspace, reducing the learning burden under a limited deployment interaction budget (Lines 143–145, 200–209, 469–483).

b) State Conditioning and Persistence. DSRL learns a state-conditioned noise policy (Lines 143–145); ReFlowTTA learns one mean shared across states and retained across steps and episodes (Lines 146–150).

c) Optimization. DSRL trains its noise policy with SAC, a replay buffer, and warm-start rollouts (Lines 143–145, 1274–1277). ReFlowTTA adds no network, critic, or replay buffer, updating only the 64-dimensional mean through a value-free chunk-level objective (Lines 146–150, 226–228, 1196–1199).

ii) UniSteer [2] and FlowDAgger [3] are concurrent work. We apologize for this omission. Both methods train a noise-predicting policy network using human interventions. ReFlowTTA instead updates only a shared source mean, with inverse flow serving as an optional pathway for incorporating human corrections (Lines 145–150, 250–278).

We have added the above comparisons to the revised manuscript (Lines 142–150).

W2: "...it does not isolate the contribution of the shared low-dimensional mean."

A2: To isolate the contribution of mean sharing, we compare ReFlowTTA with a state-conditioned mean variant using DSRL’s conditioning architecture in the same fixed source subspace, with all other adaptation settings matched. Results on ManiSkill3 (§4.2) are as follows:

AgentOnline SR@15 (%) ↑Final SR (%) ↑
Base VLA–33.8
DSRL34.338.1
ReFlowTTA (state-conditioned mean)38.9$_{\pm 0.86}$42.6$_{\pm 0.94}$
ReFlowTTA (shared mean)49.2$_{\pm 0.21}$53.6_{\pm 0.15}\

The shared mean outperforms the state-conditioned variant by 10.3% in Online SR@15 and 11.0% in Final SR, with lower variability across runs. These results support the effectiveness of a shared source mean under the tested deployment budget.

We have added this comparison and analysis to Appendix D.1 (Lines 1315–1323).

W3-1: "...perturbation experiment in Appendix B.2... it does not directly establish the mechanism of the mean-adaptation interface..."

A3-1: i) Purpose of the diagnostic. §B.2 shows that perturbations can occasionally yield successful rollouts without updating the VLA, suggesting that successful behaviors remain accessible under its frozen generative dynamics (Lines 958–963). This observation motivates adapting the source distribution to make such behaviors more accessible, but is not intended to establish the mechanism of learned mean updates.

ii) Additional experiment. To examine whether successful rollouts can also be obtained by changing the source noise alone, we add a comparison with perturbations applied only to the initial noise:

Perturbation scopeTask A SR (%) ↑Task B SR (%) ↑
No perturbation (τ=0\tau=0)0.00.0
Initial noise only (τ=0.6\tau=0.6)4.04.0
Initial + denoising (M=2M=2, τ=0.6\tau=0.6)8.06.0

Initial-noise perturbations alone yield successful rollouts on both tasks (4.0% each), showing that successful behaviors can also be accessed without perturbing intermediate denoising steps.

We have added this comparison and analysis to Appendix D.2 (Lines 1327–1336).

AuthorsOpenReview ↗

Responses to Reviewer CqKS (2/3)

W3-2: "Nor is it sufficient to attribute failures generally to difficulty accessing existing capabilities."

A3-2: We apologize for the misunderstanding. The scope of this paper is to make successful behaviors already attainable with the frozen pretrained VLA more reliably accessible through source-distribution adaptation (Lines 013–022, 043–051). We do not claim that OOD failures generally arise from difficulty accessing existing capabilities. Appendix E discusses the limitations when the required behaviors are absent or insufficiently represented in the frozen VLA (Lines 1447–1457, 1465–1475).

We have revised the wording to clarify this scope (Lines 909–912, 958-963).

W4-1: "...the reverse integration is not the exact discrete inverse of the actual four-step forward decoder and the paper does not report the action reconstruction error..."

A4-1: We evaluate 400 correction chunks from the real-world experiments (10 per task, 20 tasks per platform), measuring the action reconstruction error after passing the recovered source noise through the actual four-step forward decoder. As a reference, we also evaluate fixed-point inversion of this decoder, following UniSteer [2], and report inversion time for both methods:

MethodMAE on PiPER (deg) ↓MAE on ARX X5 (deg) ↓Inversion time (ms) ↓
Fixed-point inversion2.22.41748.5
Ten-step reverse Euler (ours)2.73.1252.4

Ten-step reverse Euler yields MAEs of 2.7° and 3.1°, exceeding the fixed-point reference by 0.5° and 0.7° on PiPER and ARX X5, respectively, while reducing inversion time from 1748.5 to 252.4 ms (6.9× speedup). This accuracy–cost trade-off supports its use as a practical approximation for recovering correction targets.

We have added this reconstruction error and inversion time analysis to Appendix D.3 (Lines 1340–1351).

W4-2: "... the number of corrections, intervention duration, and supervision weight are also unclear."

A4-2: We apologize for omitting these details. The table below reports the average number of corrections and total intervention time per task over the 15-episode adaptation period, together with the supervision weight λH\lambda_H.

We have added these details to Appendix C.4 (Lines 1188–1192):

Per task (15-episode adaptation)PiPERARX X5
Corrections / total chunks63.4 / 270.3 (23.5%)91.7 / 289.6 (31.7%)
Human intervention time (s)164.5217.9
Supervision weight λH\lambda_H0.30.4

W5: "Dense progress rewards are an important assumption behind the main results"

A5: We agree that dense chunk-level feedback is important for ReFlowTTA's rapid adaptation under the tested interaction budget. To quantify this dependence, we repeated the ManiSkill3 experiments (§4.2) with a sparse binary reward (1 only for the chunk that completes the task, and 0 otherwise).

Feedback signalOnline SR@15 (%) ↑Final SR (%) ↑
Base VLA–33.8
DSRL (dense reward)34.338.1
ReFlowTTA (sparse reward)41.745.2
ReFlowTTA (dense reward)49.2$_{\pm 0.21}$53.6_{\pm 0.15}\

Replacing dense feedback with sparse binary rewards leads to absolute drops of 7.5% in Online SR@15 and 8.4% in Final SR, demonstrating that using dense chunk-level feedback is an important design choice for rapid adaptation under a limited interaction budget.

We have added this comparison to Appendix D.4 and discussed the associated limitation in Appendix G (Lines 1355–1370, 1488–1490).

Q1-1: "Please clarify ReFlowTTA’s independent contribution relative to DSRL, UniSteer, and FlowDAgger."

A1-1: Please see W1-2.

Q1-2: "Are there results comparing a shared mean with a state-conditioned mean under..."

A1-2: Please see W2.

AuthorsOpenReview ↗

Responses to Reviewer CqKS (3/3)

Q2-1: How are the approach/placement rewards in Appendix C.6 used across different skills and long-horizon tasks in LIBERO, and how are stage transitions determined?

A2-1: We apologize for the unclear description.

i) Reward formulation. All tasks (including grasping and other skill tasks) use the same distance-based potential, Φs=1−tanh⁡(10ds)\Phi_s=1-\tanh(10d_s), where dsd_s is the distance between the end effector and the interaction object, or between the manipulated object and its target. LIBERO-Long tasks are compositions of subtasks and reuse the corresponding reward designs.

ii) Stage transitions. Execution advances to the next stage when the current stage's distance dsd_s falls below a predefined threshold.

We have added these details to Appendix C.6 (Lines 1236–1241, 1256–1259).

Q2-2: "Do all learning-based baselines receive the same reward information?"

A2-2: All learning-based baselines receive the same reward information:

i) PDF accumulates the same chunk-level dense rewards into the scalar delayed feedback rr used in its adaptation loss.

ii) DSRL receives the same dense reward for each chunk-level transition stored in its replay buffer.

We have added these details to Appendix C.7 (Lines 1277–1278, 1284–1286).

Q2-3: "Please also clarify the definition of a deployment condition, the scope over which the mean is reset..."

A2-3: We apologize for the lack of clarity.

i) A deployment condition refers to a task under a specified OOD environment and robot embodiment.

ii) For each independent adaptation run under that condition, the mean is initialized to zero once, then retained and updated across steps and episodes, without resetting between episodes (Algorithm 1, Lines 839–841; Lines 245–246). Adaptation starts afresh for a new deployment condition.

We have clarified the deployment condition and mean initialization and persistence in Appendix A (Algorithm 1, Lines 839–841).

Q2-4: "...whether independent evaluation episodes are completely excluded from adaptation updates."

A2-4: We apologize for not making this protocol explicit. Independent evaluation rollouts are fully excluded from adaptation: their data are never added to the adaptation buffers or used for parameter updates. All adaptation updates use only feedback collected during the 15 deployment episodes (20 on LIBERO-Plus) under the corresponding condition.

We have clarified this protocol in §4.1 (Line 306).

Q3-1: "Please report the number of correction action blocks actually used per task, intervention duration, trigger rules, and the setting of λH\lambda_H."

A3-1: i) For correction counts, intervention duration, and λH\lambda_H, please see W4-2.

ii) Interventions are human-initiated. After a chunk ends, if the robot has clearly deviated from the intended task execution, we reset it to the state before that chunk and execute a corrective action chunk.

We have added the intervention protocol to Appendix C.4 (Lines 1166–1168).

Q3-2: "Have you measured the action error after the inverted noise is passed through the actual four-step forward decoder..."

A3-2: Please see W4-1.

Q3-3: "...the effect of low-dimensional projection on correction supervision?"

A3-3: Under our source-distribution parameterization, projected supervision is fully equivalent to full-space supervision for source-mean learning. Since B⊤B=IB^\top B=I and mtH=B⊤vec(ηtH)m_t^H=B^\top\mathrm{vec}(\eta_t^H) (Eq. 9), we have ∥∥vec(ηtH)−Bμϕ∥∥2=∥∥mtH−μϕ∥∥2+∥∥(I−BB⊤)vec(ηtH)∥∥2\|\|\mathrm{vec}(\eta_t^H)-B\mu_\phi\|\|^2=\|\|m_t^H-\mu_\phi\|\|^2+\|\|(I-BB^\top)\mathrm{vec}(\eta_t^H)\|\|^2. The second term is independent of μϕ\mu_\phi, so the negative log-likelihood in Eq. 10 differs from its full-space counterpart only by a constant and yields exactly the same gradient. Thus, projection preserves all correction information relevant to updating the mean.

We thank the reviewer for raising this point and have added this analysis to Appendix C.4 (Lines 1180–1187).

[1] Wagenmaker et al. Steering Your Diffusion Policy with Latent Space Reinforcement Learning. In CoRL 2025, arXiv:2506.15799.

[2] Lu et al. UniSteer: Unified Noise Steering for Efficient Human-Guided VLA Adaptation. arXiv:2605.10821.

[3] Murray et al. FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space. arXiv:2607.08877.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Responses & Paper Revisions

We sincerely thank all reviewers for their careful reviews and constructive feedback. We have revised the paper to address their concerns. The main updates are summarized below.

Contribution and Mechanism Clarifications

  1. Clarified our contribution and distinctions from related work (Secs. 1–2).
  2. Added controlled comparisons of different source-mean parameterizations and optimization strategies (Appendices D.1, D.6).
  3. Added experiments testing whether initial-noise perturbations alone can produce successful rollouts (Appendix D.2).
  4. Clarified the scope of our mechanistic interpretation (Appendices B.1–B.2).

Inverse Flow and Human Corrections

  1. Evaluated inverse-flow reconstruction accuracy and computational cost (Appendix D.3).
  2. Added analysis of adaptation sensitivity to numerical inversion (Appendix D.8).
  3. Clarified the effect of low-dimensional projection on correction supervision (Appendix C.4).
  4. Specified intervention triggers, correction counts, intervention time, and supervision weights (Appendix C.4).
  5. Evaluated the effect of the number of human corrections on adaptation performance (Appendix D.9).

Feedback Robustness and Generalization

  1. Evaluated adaptation sensitivity to sparse, delayed, and noisy rewards (Appendix D.4).
  2. Discussed the dependence on feedback density (Appendix G).
  3. Extended evaluation to additional flow-based VLA backbones (Appendix D.7).
  4. Clarified that new deployment conditions require independent adaptation (Appendices A, G).

Implementation and Evaluation

  1. Clarified the source-mean update procedure, including batch reuse, optimizer settings, and reward processing (Sec. 3.2, Algorithm 1, Appendix C.5).
  2. Clarified the adaptation and evaluation protocols, including mean persistence and independent evaluation (Sec. 4.1, Appendix A).
  3. Clarified reward design and the consistency of feedback and coordinate alignment across methods (Appendices C.4, C.6–C.7).
  4. Clarified baseline implementation differences and the settings matched for comparison (Appendix C.7).
  5. Reported deployment interaction budgets and computational costs (Appendices C.6–C.7).
  6. Evaluated the effect of additional warm-start interactions for a learning baseline (Appendix D.5).
  7. Added run-to-run standard deviations for simulation results and real-world Final SR (Tables 1–4).

Revisions are highlighted in red in the revised manuscript. Detailed responses to individual comments are provided in the reviewer-specific responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.