acceptodds
Under review as a conference paper at ICLR 2027

DRIFT: A Residual Flow Adapter for Decoding Continuous Outputs in Vision-Language Models

Abstract

Many modern vision-language models (VLMs) build on autoregressive decoding of discrete tokens. While text-based output interfaces enable scalable pretraining and strong zero-shot generalization across diverse tasks, they are poorly suited for problems that require precise continuous outputs, such as localizing temporal boundaries of events or generating robotic control actions. To address this challenge, we propose DRIFT, a general framework for adapting pretrained VLMs to continuous decoding tasks. DRIFT combines a base predictor, which provides a coarse estimate of the target output, with a generative refinement module based on flow matching that iteratively improves the prediction. This residual formulation transforms the generative modeling problem from learning a global output distribution to modeling a localized residual distribution around a strong prior, substantially simplifying optimization. We evaluate DRIFT on both perception and planning tasks, including visual grounding and robotic control. Across multiple tasks and architectures spanning MLLMs, VLAs, and WAMs, DRIFT consistently outperforms a strong set of regression- and generative-based solutions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Language Model, Flow decoder, Video Temporal Grounding, Vision-Language-Action, Spatial Grounding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_amGDOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DRIFT, a residual flow matching framework for adapting pretrained vision-language models to continuous output prediction. DRIFT first obtains a coarse prediction from a base predictor and then uses a flow-based module to refine it, with a learnable skip connection and a joint training strategy. The authors evaluate the method on robotic manipulation, temporal and spatial grounding, and world action models, showing consistent improvements over several baselines.

Critical strengths
  1. The paper presents a simple and intuitive idea of combining a coarse prediction with flow-based refinement for continuous output generation. The joint training strategy also seems useful, as it improves the base predictor itself rather than only relying on the refinement module.
  2. The method is evaluated across multiple tasks, including robotic manipulation and visual grounding, with consistent improvements over regression and generative baselines. I think the broad applicability and practical improvements are meaningful, although the gains over direct flow matching are relatively modest.
Critical weaknesses
  1. My main concern is the limited novelty of the proposed method. The idea of using an informative prior for generative refinement has been explored in previous works, such as BRIDGeR (RSS 2024) [1], which also starts from a source policy rather than Gaussian noise. The relative idea is even more broader in visual generation such as Residual Denoising Diffusion Models (CVPR 2024) [2]. I could not find this work discussed or compared in the paper, and I think the authors should better clarify what is different from existing approaches.

  2. The improvements over direct flow matching seem relatively modest, and it is not entirely clear where the gains come from. In Table 5(b), jointly training the MLP with the flow refiner improves its performance from 88.2% to 93.0%. However, it is unclear whether this improvement comes from the proposed joint training strategy or simply from the additional model capacity and training compute. I think a more controlled comparison with a standalone MLP / flow-matching trained under a matched parameter and FLOPs budget would help clarify this.

[1] Behavioral Refinement via Interpolant-based Policy Diffusion, RSS 2024. [2] Residual Denoising Diffusion Models, CVPR 2024.

Questions for authors

Same as weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer amGD

We thank the reviewer for the constructive feedback. Throughout our responses, Tables/Figures 1, 2, etc. refer to those in the original paper, while Tables A, B, etc. refer to those in the rebuttal. We address the reviewer's questions below.

W1: Novelty:

We thank the reviewers for highlighting these relevant precedents. DRIFT's primary contribution is a general framework for adapting pretrained vision-language models to produce precise continuous outputs. It supports diverse base predictors, ranging from discrete tokenization to simple MLPs (Table 1), and applies to multimodal large language models (Tables 2–3), vision-language-action models (Table 1), and world-action models (Table 4). We further demonstrate its applicability across vision-language-action tasks (Table 1), temporal video grounding (Table 2), and spatial grounding (Table 3). Across these settings, DRIFT consistently outperforms the corresponding baselines, demonstrating its effectiveness and generalizability.

We acknowledge that DRIFT shares structural similarities with prior methods leveraging informative-source transport. DRIFT’s structural contribution is how it connects a pretrained VLM’s coarse prediction to a continuous decoder. The coarse prediction can come from the VLM’s existing token decoder or from a regression head trained on its hidden states. DRIFT uses that prediction twice: it centers the initial Gaussian in Eq. 5, and a learned gate in Eq. 4 blends it directly into the predicted endpoint at every refinement step. We will incorporate a discussion into the related work section and clarify the distinctions.

W2: Contribution of the performance improvement:

VLA. We create a new baseline Flow (Enlarged) by extending its feedforward layer from 3072 to 3328 to closely match DRIFT's decoder parameter count (33.96M vs. 33.97M). The result is shown in Table A below. The Flow (Enlarged) achieves a 93.9% success rate on the Libero-Long with 6.37 GFlops, which is mostly the same as the normal Flow (94.2%, 6.06 GFlops), and underperforms DRIFT (96.4%, 6.08 GFlops). This additional ablation demonstrates the improvement of DRIFT by refining the residual flow instead of additional parameters or an additional FLOPs budget.

Table A: Results with the prior predicting the average mode.

DecoderLibero-Long (%)#paramslatency (ms)GFlops
MLP88.21.58M28.40.02
Flow94.232.39M31.16.06
Flow (Enlarged)93.933.96M31.16.37
DRIFT96.433.97M (+4.8%)31.4 (+1%)6.08

TVG. The same holds on temporal video grounding (see our response to Reviewer evDc, Q3, and the TVG decoders table): the Flow and Diffusion decoders use the same refiner network as DRIFT (107.1M parameters), and DRIFT adds only its base predictor (9.5M parameters, 0.5 ms per query), yet it improves over them by 13.2 and 7.1 mIoU on Charades-STA. Capacity alone does not explain this: the Flow decoder is 6.3 mIoU below the much smaller MLP decoder (23.6M parameters).

Reviewer_evDcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses continuous-output prediction (e.g., robot actions, temporal boundaries, bounding boxes) with pretrained VLMs, whose native interface is discrete token decoding. The authors propose DRIFT, a decoding adapter with two stages. A base predictor (an MLP regressor or the model’s existing tokenizer) produces a coarse estimate. A flow-matching refiner then transports a Gaussian centered at that estimate, N(g(z),σ2I)N(g(z), \sigma^2I) with learnable σ\sigma, to the target distribution, and its output is combined with the base prediction through a gated skip connection. DRIFT is evaluated on VLA manipulation (Libero, Simpler), temporal video grounding (Charades-STA, ActivityNet Captions), spatial grounding (RefCOCO) and a world action model (FastWAM), and improves over the respective baselines in these settings.

Critical strengths
  1. The paper is well written and easy to follow. The method is clearly described, and the results are presented in a readable way.
  2. The evaluation is broad, covering robotic manipulation, temporal and spatial grounding, and world action models, which supports the claimed generality of the adapter. DRIFT is a lightweight plug-in that works with different base predictors (MLP, tokenizer, an existing flow decoder) and adds little inference overhead (e.g., 28.4 → 31.4 ms on VLA, Tab. 6b).
  3. The design choices (interpolant, x- vs. v-prediction, skip connection and etc) are ablated, and the theoretical section states its assumptions clearly.
Critical weaknesses
  1. Starting the generative transport from a Gaussian centered at a condition-dependent estimate, instead of N(0,I)N(0, I), is well established. Eq. 5 is a special case of the data-dependent coupling of Albergo et al. (ICML 2024, Eqs. 18, 20), with a learned rather than fixed center. Grad-TTS (ICML 2021) already transports from N(μ,I)N(\mu, I), with μ\mu predicted by an encoder trained jointly under an MSE-type loss, which mirrors LFM+LpL_{FM} + L_p. In robot learning, BRIDGeR (RSS 2024) makes the same “don’t start from noise” argument with an informative source policy. None of these works seemed to be cited. Switching from diffusion to FM is not a substantive difference. The contribution is therefore incremental.

  2. Table 1 compares systems that differ in backbone, robot pretraining, camera views (DRIFT uses main + wrist) and chunk size. It is also not clear from the text whether DRIFT is trained per suite or jointly on all Libero suites, whereas $\pi$0.5 is trained jointly. The authors themselves state that the VLM’s pretraining strongly affects results. A matched-backbone comparison against Flow/Diffusion decoders, which is the only direct evidence for the proposed decoder, is provided only for Libero-Long (Tab. 5a). I would prefer additional clarification of the training regime and setup of every method in Table 1, and extending the matched comparison to Simpler, the other Libero suites and etc.

  3. “Outperform VLM4VLA… demonstrating the effectiveness of DRIFT”: VLM4VLA uses a single view, no proprioception and chunk size 4. Under the authors’ own setup, a plain MLP already reaches 88.2 vs. 55.8 on Libero-Long, so the gap is due to the setup. “Across all settings… improves over… flow-matching decoders”: flow baselines are reported only for Libero-Long.

Questions for authors
  1. Can you provide the matched-backbone comparison of DRIFT against MLP, Flow and Diffusion decoders beyond Libero-Long?
  2. Is DRIFT trained separately for each Libero suite or jointly on all four?
  3. Can you report quantitative results for the MLP and Flow heads on TVG, which are currently shown only qualitatively (Fig. 3, 5)?
  4. Have you tried applying DRIFT on top of the backbones of pretrained VLAs, e.g., $\pi$0/$\pi$0.5 (whose weights are public), so that DRIFT can be compared with their native flow decoders on the same backbone and pretraining? This would isolate the effect of the decoder from that of the backbone in Table 1.
  5. It is not entirely clear where the baseline numbers in Table 1 come from (only OpenVLA is marked as reproduced). Could you specify the source of each number and, if possible, the setup of each method (camera views, proprioception, action chunk size)?
  6. FastWAM (Tab. 4): was the baseline reproduced? The Fast-WAM paper reports 97.6 average on Libero (vs. 96.9 in Tab. 4), which would reduce the gain to 0.6 pp.
4 comments
AuthorsOpenReview ↗

Response to evDc (Part 1)

We thank the reviewer for the constructive feedback. Throughout our responses, Tables/Figures 1, 2, etc. refer to those in the original paper, while Tables A, B, etc. refer to those in the rebuttal. We address the reviewer's questions below.

W1: Novelty

We thank the reviewers for highlighting these relevant precedents.

DRIFT's primary contribution is a general framework for adapting pretrained vision-language models to produce precise continuous outputs. It supports diverse base predictors, ranging from discrete tokenization to simple MLPs (Table 1), and applies to multimodal large language models (Tables 2–3), vision-language-action models (Table 1), and world-action models (Table 4). We further demonstrate its applicability across vision-language-action tasks (Table 1), temporal video grounding (Table 2), and spatial grounding (Table 3). Across these settings, DRIFT consistently outperforms the corresponding baselines, demonstrating its effectiveness and generalizability.

We acknowledge that DRIFT shares structural similarities with prior methods leveraging informative-source transport. We will incorporate these works into the related work section and clarify the distinctions.

W2 and Q2: Clarification of the training setting in LIBERO

For the results in Table 1, DRIFT is trained separately for each Libero suite when evaluated. For the results in Table 4, following the training setting of the FastWAM, we train the DRIFT jointly on the Libero suite when evaluated.

W3: Clarification of the performance statement

Thank you for highlighting this distinction. We agree that the LIBERO-Long comparison with VLM4VLA does not isolate DRIFT’s contribution because the experimental setups differ. We will revise and restrict the VLM4VLA statement to the reported Simpler WidowX result (61.5% vs. 49.0% with Qwen3-VL-2B; Table 1). Our controlled evidence comes from Table 5(a): under the same training setup, DRIFT achieves 96.33 ± 0.23%, compared with 88.20 ± 0.00% for MLP and 93.67 ± 0.35% for standalone flow matching.

The direct regression and flow-based baseline comparisons are also reported beyond LIBERO-Long. FastWAM already uses a flow-matching action decoder (§4.3). Following its training and evaluation settings, integrating DRIFT improves Spatial/Goal/Long success from 97.6/95.6/95.0% to 99.2/98.2/96.2%, raising the suite average from 96.9% to 98.2% (Table 4), with a small decrease on Object (99.4% to 99.0%). Table 1 additionally reports higher LIBERO-average and Simpler performance than flow-based π0\pi_{0} and π0.5\pi_{0.5}.

On temporal video grounding, we now also report matched decoders: under the same backbone, data and training schedule, DRIFT outperforms the flow decoder by 13.2 mIoU on Charades-STA (43.4 vs. 30.2; see our response to Q3 and the TVG decoder ablation table below).

Beyond robotic control, DRIFT improves ET-Chat’s temporal video grounding mIoU from 42.3% to 43.8% on Charades-STA and from 18.9% to 37.3% on ActivityNet-Captions (Table 2). For spatial grounding, it improves Qwen3-VL-2B from 89.6/81.8/85.5% to 91.7/85.8/87.9% on RefCOCO/RefCOCO+/RefCOCOg (Table 3).

These results collectively support the claim we made in the introduction.

W2 and Q1: Addition VLA structural ablation on Simpler

We conduct an additional ablation study of action decoders on Simpler WidowX, following the training and evaluation protocol in Section 4.1. The results are presented in Table A below. Both Diffusion and Flow substantially outperform the MLP on the challenging Stack Green Block task. DRIFT further improves overall performance by 4.6, 4.9, and 6.3 percentage points over Flow, Diffusion, and the MLP, respectively, further validating its effectiveness.

Table A: Structural ablation on Simpler. The numbers are averaged over 3 different seeds. SGB: Stack Green Block; PC: Put Carrot; PS: Put Spoon; PE: Put Eggplant.

DecoderSGBPCPSPEAvg.
MLP4.258.383.375.055.2
Diffusion23.6 ± 0.650.0 ± 1.476.4 ± 0.576.4 ± 0.756.6 ± 0.4
Flow30.5 ± 0.944.4 ± 0.268.1 ± 1.784.7 ± 0.556.9 ± 0.6
DRIFT33.3 ± 0.750.0 ± 0.475.0 ± 0.787.5 ± 0.461.5 ± 0.2
AuthorsOpenReview ↗

Response to evDc (Part 2)

Q3: Addition TVG structural ablation

We trained the MLP and Flow heads on TVG under exactly the same setting as DRIFT: the same ET-Chat-3.8B backbone and LoRA configuration, and the same TimeLens-100K data, schedule and losses, so that only the decoder differs. All decoders, including DRIFT, were retrained for this comparison. The Flow head uses the same refiner network as DRIFT but starts from Gaussian noise instead of the base prediction. We also added a Diffusion head: the same refiner network trained as a DDPM noise predictor with the configuration of our VLA diffusion baseline (100-step cosine schedule, 10 DDIM steps). The Table B below reports zero-shot results on the 3,720 Charades-STA test queries. The generative heads average 20 noise samples per prediction (Flow and DRIFT: 10 midpoint steps; Diffusion: 10 DDIM steps), and we report the mean ± std over 3 sampling seeds. DRIFT improves mIoU over the MLP, Diffusion and Flow heads by 6.9, 7.1 and 13.2 points. On this task the Flow head even falls below the MLP head, whereas the same refiner started from the base prediction becomes the best decoder. More sampling steps do not close the gap for Diffusion: with 20, 50 and 100 DDIM steps (the full chain), it reaches 36.3, 36.4 and 36.4 mIoU.

These results highlight the effectiveness of the DRIFT.

Table B: Structural ablation on TVG (Charades-STA, %). All decoders share the ET-Chat-3.8B backbone, training data and schedule. Generative decoders: mean ± std over 3 sampling seeds.

Decoder[email protected][email protected][email protected]mIoU
MLP60.336.911.936.5
Diffusion59.0 ± 0.335.9 ± 0.311.4 ± 0.336.3 ± 0.0
Flow49.0 ± 0.220.8 ± 0.65.9 ± 0.230.2 ± 0.1
DRIFT66.9 ± 0.246.0 ± 0.319.1 ± 0.243.4 ± 0.1

Q4: Build DRIFT on π0.5\pi_{0.5}

The goal of DRIFT is to adapt Vision-Language Models (VLMs) with discrete output interfaces to produce precise continuous predictions. Since π0.5\pi_{0.5} is already pretrained with a continuous output interface through its flow-based action decoder, extending DRIFT to π0.5\pi_{0.5} falls outside the scope of this work.

Nevertheless, Table 1 demonstrates that DRIFT can be integrated with OpenVLA, a pretrained VLA model with a discrete action interface, and further improve its performance through action refinement.

Furthermore, Table 5(a), compares DRIFT against direct flow matching under identical Qwen3-VL-2B backbones and training settings, showing an improvement in LIBERO-Long success from 93.67% to 96.33%. Table 4 further demonstrates that DRIFT can be integrated with FastWAM's native flow decoder using its original initialization and training recipe, improving average LIBERO success from 96.9% to 98.2%. Together, these results validate the effectiveness and generalizability of DRIFT's residual refinement over standard flow-based decoding.

Q5: Clarification of the source of the number

In Table 1, we compare against two groups of baselines: (1) models with large-scale robotic pretraining, including OpenVLA, π0\pi_0, and π0.5\pi_{0.5}, and (2) methods based on adaptation, such as VLA-Adaptor, VLM4VLA, and InternVLA. The results for the adaptation-based methods are taken from the corresponding paper. The results for OpenVLA are re-evaluated by us using the corresponding checkpoints. We collect the results for π0\pi_0 from previous work, InternVLA (Table 2, 4). We gather the results for π0.5\pi_{0.5} from the official openpi repository for the Libero and previous work, Stare-vla (Table 1).

Chen, Xinyi, et al. "Internvla-m1: A spatially guided vision-language-action framework for generalist robot policy." arXiv preprint arXiv:2510.13778 (2025).

Xu, Feng, et al. "Stare-vla: Progressive stage-aware reinforcement for fine-tuning vision-language-action models." arXiv preprint arXiv:2512.05107 (2025).

Q6: Clarification of the source of the WAM number

The numbers presented in Table 4 are the reproduced results.

Reviewer_evDcOpenReview ↗

Response to Rebuttal

Thank you for the detailed rebuttal, which partially addresses my concerns. However, some issues remain.

First, the comparison with VLM4VLA is still not fair. In Table A, the plain MLP head already reaches 55.2% on Simpler, compared with 49.0% for VLM4VLA with the same backbone. Much of the gap therefore comes from the experimental setup, and this comparison does not isolate the effect of DRIFT.

Second, it is not clear to me why applying DRIFT to π0\pi0/ $\pi$0.5 falls outside the scope of this work, given that it is applied to FastWAM, which also natively produces continuous outputs through a flow-based decoder.

Most importantly, my concern about novelty remains. The response acknowledges the similarity to prior work but does not specify the distinctions. The work appears incremental: it mainly applies a known approach to new tasks. So, I will maintain my score.

AuthorsOpenReview ↗

Response to evDc (Round 2)

Thanks for your response. We are glad to hear that the rebuttal has addressed some of your concerns. Here are the responses to your follow-up questions.

Round 2 Q1: Higher performance of plain MLP and comparison with VLM4VLA.

The higher performance of our MLP baseline (55.2% vs. 49.0%) is primarily due to the difference in training data: our model is trained on both Bridge and RT-1/Fractal, whereas VLM4VLA uses Bridge only. Therefore, the comparison should be interpreted as an end-to-end system-level comparison rather than a controlled comparison of decoding methods.

To isolate the contribution of DRIFT, we conduct controlled ablations in Table 5(a) and Table A, where the backbone, training data, training setup, and evaluation protocol are kept fixed, with only the decoding method varied. These experiments directly measure the effect of DRIFT relative to MLP and Flow decoders. We believe these controlled results provide direct evidence of DRIFT's effectiveness, complementary to the system-level comparisons in Table 1.

Round 2 Q2: applying DRIFT to 0.5 falls outside the scope.

DRIFT aims to enable VLMs with discrete output interfaces to generate precise continuous predictions. In contrast, π0.5\pi_{0.5} is pretrained at scale with a flow-based action decoder which is specifically optimized for continuous action generation. Therefore, we do not expect DRIFT to provide substantial additional gains on π0.5\pi_{0.5}, and such an investigation falls outside the scope of this work.

We instead evaluate DRIFT on FastWAM to demonstrate its potential for world-action models, where the action expert is initialized from a pretrained video generation model without explicit optimization for continuous action decoding. This setting aligns more closely with DRIFT's objective of adapting pretrained models for precise continuous prediction.

Round 2 Q3: Novelty.

We agree that starting generation from an informative estimate is established. DRIFT’s structural contribution is how it connects a pretrained VLM’s coarse prediction to a continuous decoder. The coarse prediction can come from the VLM’s existing token decoder or from a regression head trained on its hidden states. DRIFT uses that prediction twice: it centers the initial Gaussian in Eq. 5, and a learned gate in Eq. 4 blends it directly into the predicted endpoint at every refinement step.

Albergo et al. already give the shifted interpolation in Eqs. 18–20. Grad-TTS is the closest precedent for a learned center: its encoder predicts the Gaussian mean, which also conditions its score network. BRIDGeR refines samples from a source policy. Our claim is therefore specific to the gated adapter that reuses a VLM prediction throughout refinement and supports the same decoding design for boxes, timestamps, and robot actions.

Reviewer_Tf3nOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces DRIFT (Decoding via Residual Flow Adapter), a general framework designed to adapt pretrained Vision-Language Models (VLMs) with discrete token decoders to tasks requiring continuous outputs, such as temporal video grounding and robot action generation.

DRIFT adopts a two-stage approach:

  1. The base predictor, which is either an existing discrete tokenizer or a lightweight MLP regressor, produces an initial coarse estimate from the VLM representations.
  2. Then, a Flow Refiner module learns a velocity field that transports a localized Gaussian prior centered at the coarse prediction toward the true target distribution.

On the experiment side, DRIFT is evaluated across multiple modalities and architectures, such as VLA, TVG (temporal video grounding), and WAM (world action models). And across all tasks, DRIFT consistently outperforms direct regression, discrete tokenization, and direct diffusion/flow-matching baselines.

Critical strengths
  1. The paper frames continuous output generation as residual flow matching around an informed coarse anchor rather than unconditional noise-to-data flow, which is neat.

  2. The paper provides solid statistical justifications for the framework.

  3. The experiment is comprehensive by evaluating DRIFT across multiple modalities and architectures. The paper also provides clear ablation studies on every design axis.

Critical weaknesses
  1. The robotics/VLA evaluations are conducted exclusively in simulation environments (Libero and Simpler simulated WidowX) but not in real-world cases.

  2. Although DRIFT adds minimal overhead relative to standard flow matching, it still requires numerical ODE integration (10 forward-Euler steps), which is slower than a single forward pass of a direct MLP regressor.

  3. In the TVG implementation, the coarse predictor employs a heuristic "1.2x width expansion factor" before conversion to start-end coordinates. It is unclear how sensitive the TVG gains are to this hyperparameter.

Questions for authors
  1. How does DRIFT perform with fewer Euler steps (e.g., K = 1, 2, 3)?
  2. Have you tested DRIFT on physical robot hardware?
  3. How does DRIFT behave when the base predictor predicts the average mode?
  4. What is the impact of the 1.2x width expansion heuristic used in the TVG base predictor?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Tf3n (Part 1)

We thank the reviewer for the constructive feedback. Throughout our responses, Tables/Figures 1, 2, etc. refer to those in the original paper, while Tables A, B, etc. refer to those in the rebuttal. We address the reviewer's questions below.

W1 and Q2: Experiment on the Real Robot

The primary goal of this paper is to introduce a general framework for decoding continuous outputs from VLMs across different model architectures, output formats, and tasks. Towards this goal, we conducted extensive evaluations on VLA and TVG tasks using multiple VLM backbones, and further demonstrated the applicability to 2D spatial grounding and world action models.

We believe that these experiments provide sufficient evidence to support our main claims. Real-world robotic deployment involves additional system-level considerations beyond the scope of the present work. We therefore view such evaluation as a future direction rather than a necessary component of this paper.

Q1: Fewer Euler steps

VLA. We evaluate the DRIFT on the Libero-10 with different K=0,1,2,3,5,10K=0,1,2,3,5,10. The results are presented in Table A below. As Table 5(b) shows, the base predictor (K=0K=0) obtains 93% accuracy on the Libero-10. With larger KK, the performance gradually increases from 94.07 (K=1K=1) to 96.33 (K=10K=10), demonstrating the effectiveness of the flow refiner. It also shows the potential latency-accuracy trade-off in DRIFT deployment.

TVG. On temporal video grounding (Charades-STA), averaging 20 noise samples as in our TVG evaluation, DRIFT reaches 40.1, 41.3, 41.7, 41.9 and 42.3 mIoU with K=1,2,3,5,10K = 1, 2, 3, 5, 10 Euler steps (43.4 with the ten midpoint steps used for our TVG results), compared with 36.5 for an MLP decoder trained under the same setting (see our response to Reviewer evDc, Q3). A single step already improves over the MLP decoder by 3.6 mIoU, and more steps improve further.

Table A. DRIFT performance with fewer Euler steps KK and the latency. Results are averaged over 3 random seeds.

KKLibero-Long (%)ms
093.0028.4
194.0728.7
294.2629.1
394.8629.4
595.4030.0
10 (Default)96.3331.4

W2: Slower than MLP

VLA. As in Table 6(b) in the main paper, DRIFT improves success by 8.13 percentage points over MLP while adding a small amount of inference overhead, increasing latency from 28.4 to 31.4 ms (10.5%). Relative to direct flow, it improves success by 2.66 points for 0.3 ms additional latency. Further, the additional fewer Euler steps experiment shown above demonstrates the latency-accuracy trade-off in DRIFT deployment. In the extreme case, using the base predictor only from the DRIFT during inference still provides 4.8% improvement without adding any additional inference cost.

TVG. On temporal video grounding, the vision encoder and the LLM dominate inference: on one H200, a Charades-STA query takes about 250 ms with the MLP decoder (36.5 mIoU). DRIFT adds 0.9 ms with one Euler step (40.1 mIoU) and 8.6 ms with ten Euler steps averaged over 20 noise samples (42.3 mIoU). The ten midpoint steps with 20 samples used for our TVG results add 16.8 ms (6.7%, 43.4 mIoU). The noise samples run as one batch, so averaging over them costs less than 1 ms. These TVG measurements (533 queries from the first 200 test videos) were taken on a different machine from Table 6(b), so their absolute values are not directly comparable with it. These results collectively show that DRIFT can improve over the MLP decoder with a small amount of inference cost.

W3 and Q4: 1.2x width expansion

It is a fixed constant, applies to the center and the width alike, and the head is trained end to end through it. To measure its impact, we retrained DRIFT without it (the head regresses center and width directly, and the start and end points are clamped to the video) under otherwise identical settings. Table B below shows that DRIFT does not depend on the factor: without it, DRIFT reaches 43.4 mIoU instead of 41.6, and the base predictor alone 35.7 instead of 36.6. With either parameterization, DRIFT clearly outperforms the MLP (36.5), Diffusion (36.3) and Flow (30.2) decoders trained under the same setting (see our response to Reviewer evDc, Q3, and the TVG decoders table). We will rewrite this sentence of Appendix B.3.

Table B. Effect of the 1.2 factor in the TVG base predictor (Charades-STA, %). DRIFT rows: mean ±\pm std over 3 sampling seeds.

Base predictor output[email protected][email protected][email protected]mIoU
DRIFT, tanh⁡(⋅)×1.2\tanh(\cdot)\times 1.2 (Appendix B.3)64.1±0.364.1\pm0.344.1±0.344.1\pm0.318.8±0.318.8\pm0.341.6±0.141.6\pm0.1
DRIFT, without the factor66.9±0.266.9\pm0.246.0±0.346.0\pm0.319.1±0.219.1\pm0.243.4±0.143.4\pm0.1
Base predictor alone, tanh⁡(⋅)×1.2\tanh(\cdot)\times 1.260.635.911.936.6
Base predictor alone, without the factor50.726.311.935.7
AuthorsOpenReview ↗

Response to Reviewer Tf3n (Part 2)

Q3: DRIFT with Average mode

We use the mean action over the training set as the action average mode, and use it as the prior and train the DRIFT on top of it following the training schedule used in Section 4.1. The result is presented in Table C below. With the average mode as the prior, DRIFT then reaches 94.1%, close to direct flow matching (94.2%), whereas the trained-prior model reaches 96.4%. This observation is similar to those in Table 6(a) in the main paper.

Table C. Results with the prior predicting the average mode.

DecoderLibero-Long (%)
Flow94.2
DRIFT (Avg mode base predictor)94.1
DRIFT96.4

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.