- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes DRIFT, a residual flow matching framework for adapting pretrained vision-language models to continuous output prediction. DRIFT first obtains a coarse prediction from a base predictor and then uses a flow-based module to refine it, with a learnable skip connection and a joint training strategy. The authors evaluate the method on robotic manipulation, temporal and spatial grounding, and world action models, showing consistent improvements over several baselines.
Critical strengths
- The paper presents a simple and intuitive idea of combining a coarse prediction with flow-based refinement for continuous output generation. The joint training strategy also seems useful, as it improves the base predictor itself rather than only relying on the refinement module.
- The method is evaluated across multiple tasks, including robotic manipulation and visual grounding, with consistent improvements over regression and generative baselines. I think the broad applicability and practical improvements are meaningful, although the gains over direct flow matching are relatively modest.
Critical weaknesses
-
My main concern is the limited novelty of the proposed method. The idea of using an informative prior for generative refinement has been explored in previous works, such as BRIDGeR (RSS 2024) [1], which also starts from a source policy rather than Gaussian noise. The relative idea is even more broader in visual generation such as Residual Denoising Diffusion Models (CVPR 2024) [2]. I could not find this work discussed or compared in the paper, and I think the authors should better clarify what is different from existing approaches.
-
The improvements over direct flow matching seem relatively modest, and it is not entirely clear where the gains come from. In Table 5(b), jointly training the MLP with the flow refiner improves its performance from 88.2% to 93.0%. However, it is unclear whether this improvement comes from the proposed joint training strategy or simply from the additional model capacity and training compute. I think a more controlled comparison with a standalone MLP / flow-matching trained under a matched parameter and FLOPs budget would help clarify this.
[1] Behavioral Refinement via Interpolant-based Policy Diffusion, RSS 2024. [2] Residual Denoising Diffusion Models, CVPR 2024.
Questions for authors
Same as weakness.
Response to Reviewer amGD
We thank the reviewer for the constructive feedback. Throughout our responses, Tables/Figures 1, 2, etc. refer to those in the original paper, while Tables A, B, etc. refer to those in the rebuttal. We address the reviewer's questions below.
W1: Novelty:
We thank the reviewers for highlighting these relevant precedents. DRIFT's primary contribution is a general framework for adapting pretrained vision-language models to produce precise continuous outputs. It supports diverse base predictors, ranging from discrete tokenization to simple MLPs (Table 1), and applies to multimodal large language models (Tables 2–3), vision-language-action models (Table 1), and world-action models (Table 4). We further demonstrate its applicability across vision-language-action tasks (Table 1), temporal video grounding (Table 2), and spatial grounding (Table 3). Across these settings, DRIFT consistently outperforms the corresponding baselines, demonstrating its effectiveness and generalizability.
We acknowledge that DRIFT shares structural similarities with prior methods leveraging informative-source transport. DRIFT’s structural contribution is how it connects a pretrained VLM’s coarse prediction to a continuous decoder. The coarse prediction can come from the VLM’s existing token decoder or from a regression head trained on its hidden states. DRIFT uses that prediction twice: it centers the initial Gaussian in Eq. 5, and a learned gate in Eq. 4 blends it directly into the predicted endpoint at every refinement step. We will incorporate a discussion into the related work section and clarify the distinctions.
W2: Contribution of the performance improvement:
VLA. We create a new baseline Flow (Enlarged) by extending its feedforward layer from 3072 to 3328 to closely match DRIFT's decoder parameter count (33.96M vs. 33.97M). The result is shown in Table A below. The Flow (Enlarged) achieves a 93.9% success rate on the Libero-Long with 6.37 GFlops, which is mostly the same as the normal Flow (94.2%, 6.06 GFlops), and underperforms DRIFT (96.4%, 6.08 GFlops). This additional ablation demonstrates the improvement of DRIFT by refining the residual flow instead of additional parameters or an additional FLOPs budget.
Table A: Results with the prior predicting the average mode.
TVG. The same holds on temporal video grounding (see our response to Reviewer evDc, Q3, and the TVG decoders table): the Flow and Diffusion decoders use the same refiner network as DRIFT (107.1M parameters), and DRIFT adds only its base predictor (9.5M parameters, 0.5 ms per query), yet it improves over them by 13.2 and 7.1 mIoU on Charades-STA. Capacity alone does not explain this: the Flow decoder is 6.3 mIoU below the much smaller MLP decoder (23.6M parameters).