- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Bilinear Flow Policy addresses goal-conditioned visuomotor imitation when numerical goals lie outside the demonstrated range. It retrieves a demonstration anchor, computes the difference between the query and anchor representations, and conditions a flow-matching policy through a low-rank bilinear interaction between these two factors. A learned scorer selects anchors within phase-compatible candidates, while deployment-time filtering favors familiar goal differences. Experiments include controlled multimodal regression, five simulated manipulation tasks across seven settings, and two real-robot tasks. The paper also provides a conditional extrapolation bound and investigates flow-loss and residual-coverage diagnostics.
Critical strengths
-
The architectural constraint is well motivated by the extrapolation problem. Rather than simply adding retrieved examples to a conventional policy, BFP explicitly separates anchor and residual processing and restricts their interaction. This provides a mechanism for reusing learned relationships across previously unseen combinations. Applying the structure to a flow field extends deterministic bilinear prediction to multimodal action generation, and the controlled regression experiment provides direct evidence for this capability.
-
The empirical gains are substantial, and the paper investigates their contributing components. BFP achieves approximately 79% average visuomotor OOD success in simulation, compared with 30% for standard Flow and 58% for BRP, with additional improvements on two real-robot tasks. Beyond aggregate performance, the paper examines retrieval, pretrained visual representations, predictor backbones, and bilinear rank. The controlled regression experiment separately investigates multimodal prediction, making the evaluation informative about both effectiveness and design choices.
Critical weaknesses
-
The contribution of the bilinear interaction is not fully separated from the retrieval procedure. BFP differs from standard Flow in both its conditioning inputs and architecture, while the BRP comparison changes both anchor selection and deterministic versus generative prediction. DARP provides a retrieval-based flow baseline, but uses a different retrieval procedure and extrapolation rule, so it does not isolate the bilinear interaction under identical conditioning. The retrieval ablations demonstrate that anchor selection matters, but do not by themselves establish how much of the improvement requires the bilinear flow formulation. A matched non-bilinear flow predictor receiving the same anchors and residuals would make the architectural claim more convincing.
-
The reliability of anchor retrieval under deployment shift remains insufficiently characterized. BFP relies on phase-compatible references and on query–anchor differences remaining meaningful in the chosen representation. Although the retrieval ablations show an advantage over synchronized and random selection, they do not establish how sensitive the method is to phase-estimation errors, limited reference coverage, or visual differences between the query and stored demonstrations. These dependencies matter because the claimed extrapolation benefit requires a suitable anchor–residual decomposition. Deployment also requires retaining a compatible reference dataset, which limits portability relative to a standalone policy checkpoint.
Questions for authors
-
Additional comparison: How does BFP compare with a non-bilinear flow predictor receiving the same anchor and residual, with the same retrieval procedure? This would help isolate the architectural contribution from the benefit of retrieval-based conditioning.
-
Clarification: What per-task rules and inputs determine phase at deployment? How does anchor selection handle delayed progress or failed actions, and is there evidence measuring sensitivity to phase-estimation errors?
-
Robustness evidence: How sensitive is performance to the size and composition of the demonstration reference set? Evidence from reduced reference sets or queries with poor anchor coverage would help establish when retrieval remains useful and when extrapolation breaks down.
6 comments
Response to reviewer wbsb (2/3)
W2b. BFP with a learned phase estimator
TL;DR. BFP maintains performance with learned phase estimator
We agree that a hand-designed phase estimator may be difficult to use in complex tasks. However, BFP can also use learned alternatives. We evaluated SARM [1] and CaR [2] on PushCube-D1, Relocate-D1, PickCube, and StackCube (3 seeds, 100 OOD episodes per seed). Notably, SARM was developed for long-horizon, contact-rich manipulation, including T-shirt folding [1].
Result.
| Method | OOD success | OOD reward |
|---|---|---|
BFP + hand-designed | 61.2 ± 2.0 | 50.6 ± 1.1 |
BFP + SARM | 60.7 ± 1.4 | 51.5 ± 0.7 |
BFP + CaR | 56.0 ± 2.0 | 49.5 ± 1.0 |
BT | 44.9 ± 1.5 | 41.1 ± 0.8 |
Flow | 26.8 ± 1.6 | 30.5 ± 0.6 |
Takeaway. BFP with SARM nearly matches the hand-designed estimator (60.7% vs. 61.2% OOD success), and both learned estimators outperform BT and Flow. These results support the use of learned phase estimators, while evaluating BFP on long-horizon, contact-rich tasks remains future work. We added more details in Appendix F.4 of the manuscript.
References.
[1] Chen et al., SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation, ICLR 2026
[2] Venuto et al., Code as Reward, ICML 2024.
W2c. Relation to reference coverage
TL;DR. (1) Dataset is the reference bank. (2) Query with less support extrapolates less. (3) BFP is better than FP in all data-regime.
-
We agree that reference coverage could be important for retrieval and extrapolation. We clarify in the manuscript that the reference bank is the training demonstration dataset itself; BFP does not require a separate reference dataset.
-
However, for a given training dataset, some queries may be poorly supported by that dataset; our delta-coverage diagnostic (Fig. 5a) associates poorer coverage with higher OOD flow-matching error.
-
A smaller training dataset may reduce this support, so we varied the demonstration fraction from 12.5% to 100% on six simulation tasks. We train BFP and Flow on the same subset of demonstrations across three seeds; BFP also uses this subset as its reference bank. We evaluate on 50 OOD episodes per seed on PushCube/Relocate and 100 on PickCube/StackCube.
Result. Task-averaged OOD success vs Dataset Size:
| Method | 100% | 75% | 50% | 25% | 12.5% |
|---|---|---|---|---|---|
BFP | 72.8 | 66.6 | 58.7 | 47.8 | 33.4 |
Flow | 24.0 | 25.8 | 19.7 | 15.0 | 8.1 |
BFP achieves higher task-averaged OOD success than Flow at every dataset size. With 12.5% of the demonstrations, BFP reaches 33.4% versus 8.1% for Flow.
Takeaway These results suggest that BFP can be effective in the low-data regime, although sensitivity varies by task. Because the training data and references shrink together, the experiment affects both the predictor and retrieval and does not isolate retrieval sensitivity. Please see Appendix F.9 for per-task details.
W2d. Robustness under visual shifts
We agree that visual differences between queries and stored demonstrations may affect retrieval. However, BFP can still outperform Flow under the tested shifts. We evaluated BFP and Flow on unseen goals under additional visual shifts in object color, light color, camera pose, and distractors on PushCube-D1 and Relocate-D1 (3 seeds, 100 OOD episodes per seed), ablating pretrained visual representations (PVRs), specifically frozen R3M and DINOv3 [1–3]. These shifts are introduced only at deployment and affect the test-time query observations.
Result. Task-averaged OOD success (%; mean ± SEM):
| Encoder | Policy | Clean | Object color | Light color | Camera pose | Distractors |
|---|---|---|---|---|---|---|
| R3M | Flow | 25.8 ± 0.7 | 16.7 ± 0.6 | 5.3 ± 1.5 | 13.0 ± 0.6 | 19.2 ± 1.4 |
| R3M | BFP | 65.2 ± 1.7 | 36.5 ± 1.5 | 8.5 ± 1.2 | 29.8 ± 1.2 | 35.5 ± 2.2 |
| DINOv3 | Flow | 32.2 ± 1.2 | 32.2 ± 0.6 | 30.7 ± 1.4 | 19.7 ± 0.6 | 25.0 ± 1.3 |
| DINOv3 | BFP | 68.5 ± 1.0 | 71.3 ± 1.7 | 40.0 ± 3.6 | 31.3 ± 0.8 | 43.8 ± 2.9 |
Result. BFP exceeds Flow on average under each tested shift, but performance can decline substantially. For example, under light-color changes, success falls from 65.2% to 8.5% with R3M and from 68.5% to 40.0% with DINOv3. We provide detailed experimental results in Appendix F.8.
Takeaway. These results show that BFP can generalize to unseen goals under additional deployment-time visual shifts, while robustness to these shifts also depends on the encoder. A more robust representation can help, while severe visual changes may remain a limitation.
References.
[1] Pumacay et al., The Colosseum, RSS 2024. [2] Nair et al., R3M, CoRL 2022. [3] Siméoni et al., DINOv3, 2025.
Response to reviewer wbsb (3/3)
W2e. Deployment concern
We agree that retaining reference data adds a deployment cost. However, the memory/time cost is modest.
- Memory. BFP reuses training demonstrations and caches compact latent features rather than raw images. For PnPMug and InsertPipette, 127 and 133 demonstrations occupy 173 MB and 313 MB, respectively. Feature-bank memory scales linearly, O(F), with the number of stored frames.
- Time. At inference, BFP scores a small phase-compatible candidate window. For fixed task settings, per-plan retrieval cost is O(1) in F. On an RTX 4090 GPU, retrieval takes 21.9/22.7 ms and policy prediction takes 68.9/69.4 ms on PnPMug / InsertPipette, respectively.
The measured storage and retrieval overhead is modest in these settings, while feature-bank storage grows with the number of stored frames.
Questions
Q1. Bilinear head control
- We provide additional ablation. Please see W1 for details.
Q2. Clarification of Retrieval Policy
- Thank you for pointing out our misclarification on per-task phase rule. We updated the manuscript to include the used per-task hand-designed phase rule in Appendix D.
- As reviewer pointed out, such hand-designed estimator is vulnerable to delayed progress or failed actions. Hence, we show BFP can be used with the learned phase estimators that can address delayed progress, failed actions (W2b).
- We also provide sensitivity to phase-estimation errors (W2a).
Q3. Robustness evidence of BFP
- We provide additional experiments. Please see W2c for details.
Response to reviewer wbsb (1/3)
We sincerely thank the reviewer for the careful reading and constructive suggestions. We are grateful that the reviewer recognized the well-motivated bilinear architecture, its extension to multimodal action generation, the substantial empirical gains, and the analysis of the method’s design choices. Below, we address the reviewer’s concerns.
W1. Missing Bilinear Head Control
We thank the reviewer for suggesting this experiment. To address this, we evaluated a matched non-bilinear flow predictor.
We froze BFP’s learned retrieval scorer and trained a parameter-matched, non-bilinear flow predictor using the same anchor and residual inputs. The non-bilinear predictor uses a flow action head conditioned on the concatenation of the resulting anchor and residual, in place of the bilinear product. We evaluate across seven simulation settings with state and visuomotor observations (3 training seeds, 100 OOD rollouts)
Result.
FlowBFP (non-bilinear)BFPTakeaway. Compared with Flow, the matched non-bilinear control provides a modest OOD success improvement (40.9% vs. 36.4%), whereas BFP achieves 84.7%—43.8 percentage points higher than the control. This supports a contribution from the bilinear interaction beyond retrieval-based conditioning.
W2. Sensitivity to anchor retrieval
We appreciate reviewer for suggesting more analysis on sensitivity of BFP to anchor retrieval. We evaluate sensitivity to phase retrieval (W2a, W2b), reference coverage (W2c), visual shift (W2d). We additionally address the deployment concern (W2e).
W2a. Robustness under phase-estimation errors
TL;DR. BFP is robust to phase-estimation error
We agree that phase-estimation errors may lead to unsuitable candidates. However, the phase estimator proposes a coarse candidate window, and the learned retrieval policy selects an anchor within it, allowing BFP to tolerate errors in the tested settings. To examine this, we constructed a noisy estimator that sometimes selects the wrong phase and perturbs estimated progress, with errors increasing with noise scale σ. We evaluated BFP with a noisy estimator on PushCube-D0, PushCube-D1, Relocate-D0, Relocate-D1, StackCube, and PickCube (3 seeds, 100 OOD rollouts per seed) We measure temporal intersection over union (tIoU) between the noisy and clean candidate windows, using an interval-overlap metric common in temporal action localization [1, 2].
Result.
BFP (σ = 0) (clean)BFP (σ = 0.1)BFP (σ = 0.25)BFP (σ = 0.5)BFP + Sync RetrievalBFP + Random RetrievalBTFlowAt σ = 0.5, temporal IoU falls to 0.43, below 0.5, a commonly used matching threshold in temporal action localization. BFP achieves 66.2% OOD success versus 73.4% without noise and remains above the clean-clock BFP + Sync Retrieval (60.6% / 49.4), BFP + Random Retrieval (62.6% / 52.1), BT (58.2% / 48.3), and Flow (26.1% / 26.7) references on average across tasks.
Takeaway. The phase estimator proposes a coarse window, and the learned retrieval policy selects an anchor within it. The results suggest tolerance to the tested errors, although success declines as noise increases. However, severe misalignment may exclude all useful candidates and degrade performance, so selecting a good-enough phase estimator remains important. We provide further details in Appendix F.5 of the manuscript.
References.
[1] Idrees et al., The THUMOS Challenge, CVIU 2017.
[2] Caba Heilbron et al., ActivityNet, CVPR 2015.