acceptodds
Under review as a conference paper at ICLR 2027

Sample-Anchored Advantage Weighting for Flow Policy Learning

Abstract

Offline-to-online reinforcement learning combines offline pretraining with online fine-tuning, resulting in replay buffers that mix offline data from diverse behavior policies with online experience from the evolving policy. Flow policies are well suited to capturing multimodal action distributions through iterative generation, but their implicit action distributions make conventional RL objectives difficult to apply, leaving behavior cloning as a natural yet limited training strategy. To improve performance, advantage-weighted regression (AWR) retains the imitation objective while weighting it according to estimated advantages. However, this weighting compares actions across modes on a shared value scale, which can suppress valid behavior modes whose estimated values are temporarily lower and thereby reduce behavioral diversity. To address this issue, we propose Sample-anchored Advantage-Weighted Regression (SAWR), a variant of AWR for flow policies that replaces conventional global value comparison with a sample-specific local comparison against a reference action constructed from each replay action. Specifically, SAWR constructs the reference action by partially noising the replay action and completing it with the flow policy, then weights the flow matching loss by the value difference between the replay action and its reference. This sample-specific comparison uses local value differences to guide policy learning while mitigating the suppression of valid behavior modes. Integrated into Flow Q-Learning, SAWR improves average online success rates over conventional AWR on OGBench and D4RL Adroit. Navigation and manipulation diagnostics further demonstrate better preservation of diverse successful behaviors, while a route-blockage test shows improved robustness relative to conventional AWR when previously preferred behaviors become unavailable. Code is provided in the supplementary material.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, flow matching, advantage-weighted regression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_T6RcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies a failure mode of advantage-weighted regression when the policy is a flow model. Conventional AWR uses a state-value baseline shared by all actions at a given state, so the relative weighting of different behavior modes depends on global exponentiated action values; noisy or miscalibrated value estimates can therefore suppress valid modes. SAWR replaces that shared baseline with a per-sample reference action, obtained by partially noising the replay action and completing it with the current flow policy. The value difference between the anchor and this reference then weights the conditional flow-matching loss. The authors prove that when the reference remains in the anchor’s mode, the common mode offset cancels. They also introduce a reference-time distribution, robust batch normalization with ESS targeting, and a critic-reliability gate. Instantiated on FQL, SAWR improves offline-to-online success on OGBench and D4RL Adroit; diagnostics suggest higher mode entropy and better robustness after route blockage.

Critical strengths

The intervention is clean and modular: it adds no trainable parameters and only reweights the flow-matching loss, so it can be applied to other flow-policy algorithms. The empirical study also moves beyond aggregate success, using mode entropy, same-mode rates, manipulation-order diagnostics, and a route-blockage test; the Q-chunking instantiation further supports generality.

Critical weaknesses
  1. First, the mode-preservation claim outruns the evidence. Entropy gains are modest and high-variance—e.g., Scene 0.213±0.140 versus 0.000±0.000 for FQL-V; Puzzle 0.027±0.060 to 0.282±0.332—and SAWR recovers only part of FQL’s diversity. The route-blockage result rests on one task. The claim of mitigating mode suppression should be stated more cautiously.
  2. Second, key design choices lack justification. The reference-time density, ESS target, Euler steps, and gate parameters are hand-tuned without systematic sensitivity analysis, so the locality–signal trade-off is not operationalized.
  3. Moreover, FQL-V learns its value target from a Monte Carlo estimate under the one-step policy rather than a standard TD or expectile objective, which may disadvantage the baseline; no standard AWR control is reported.
Questions for authors
  1. Can you provide paired tests or confidence intervals for mode-entropy differences, and report per-task entropy across all diagnostics?
  2. Why use the nonstandard value target for FQL-V, and would a conventional AWR baseline close the gap?
  3. What is the computational overhead of SAWR relative to FQL and FQL-V?
  4. How sensitive is performance to the shape of Eq. 14, the ESS target, and the number of Euler steps?
  5. How often is the reliability gate active, and how much of the observed mode preservation is attributable to the gate versus the sample-anchored advantage?
3 comments

We thank the reviewer for the constructive assessment. We address the five questions with additional analyses and experiments.

Q1: Mode Entropy

We selected these three tasks because they admit distinct solutions with comparable returns, making them suitable for diagnosing mode preservation. We report per-task entropy and paired SAWR–FQL-V differences with 95% t-CIs:

TaskFQLFQL-VSAWRPaired Δ
AntMaze large-stitch task50.335 [−0.093, 0.763]0 [0, 0]0.154 [−0.028, 0.336]+0.154 [−0.028, 0.336]
Scene noisy task10.609 [0.493, 0.725]0 [0, 0]0.213 [0.040, 0.387]+0.213 [0.040, 0.387]
Puzzle 4×4 noisy task10.625 [0.409, 0.841]0.027 [−0.048, 0.102]0.282 [−0.130, 0.694]+0.255 [−0.124, 0.635]

These results suggest that SAWR can partially mitigate mode suppression.

Q2: FQL-V Value Target

FQL-V uses a policy-averaged value baseline, consistent with AWAC-style advantage estimation (Nair et al., 2020). To examine whether this choice affects the comparison, we retrained FQL-V with a bootstrapped TD value target (TD-V), keeping other components unchanged.

Task familyTD-VFQL-VSAWR
AntMaze large stitch58.0±4.1 → 99.5±0.255.7±5.0 → 99.6±0.463.4±5.3 → 99.8±0.3
AntSoccer arena stitch6.4±1.5 → 28.3±4.67.8±2.3 → 38.8±4.210.6±1.3 → 55.8±5.7
Scene noisy23.2±2.0 → 40.4±0.643.2±3.3 → 60.0±0.039.4±1.6 → 60.0±0.0
Puzzle 4×4 noisy1.7±1.0 → 11.5±3.414.1±2.8 → 38.6±4.911.2±2.4 → 39.8±4.3
AntMaze giant navigate8.1±5.2 → 63.9±9.48.9±6.8 → 69.1±7.77.4±3.7 → 80.5±2.8
Humanoid medium navigate55.2±11.3 → 54.4±10.348.2±8.3 → 44.8±7.757.9±1.9 → 56.2±7.5
Humanoid large navigate2.4±2.3 → 13.3±1.15.3±3.6 → 8.2±2.52.8±0.8 → 6.7±4.6
Average22.2±2.7 → 44.5±2.726.2±0.8 → 51.3±1.727.5±1.2 → 57.0±1.8

FQL-V provides an architecture-controlled AWR-style baseline, using a shared state-value reference while retaining the same flow-policy backbone. The TD-V experiment further tests whether its value-target construction accounts for the observed gap. Neither variant closes the average online performance gap with SAWR, suggesting that the gains cannot be explained solely by the original value-target choice.

Q3: Computational Overhead

On RTX 3090 GPUs (one process/GPU), training times for FQL / FQL-V / SAWR were 2.00 / 2.65 / 2.30 h on AntSoccer arena navigate task1 and 1.75 / 2.42 / 1.89 h on Adroit pen-expert.

SAWR increases training time by only 15% and 8% over FQL, respectively, while remaining faster than FQL-V. It introduces no additional trainable networks or inference-time overhead.

Q4: Hyperparameter Sensitivity

Reference-time distribution. Eq. 14 is an interior-peaked, piecewise-linear density balancing local comparisons with near-zero reference times for cross-modal value selection.

Our Early/Mid/Late/Uniform ablations in Figure 6 illustrate this trade-off: later times better preserve diversity but may weaken value discrimination, whereas earlier times favor value-based selection. These results support the locality–selection trade-off, rather than the unique optimality of Eq. 14. Our goal is to encourage local comparisons while retaining sufficient global comparisons, balancing multimodal behavior preservation with task success. Alternative distribution shapes may affect quantitative performance, but the key mechanism lies in controlling this locality–selection trade-off.

ESS and Euler steps. On five AntSoccer arena-stitch tasks, online success rates (%) were:

ESS targetSuccessEuler stepsSuccess
0.552.8±13.7151.6±4.7
0.7 (default)55.8±5.73 (default)55.8±5.7
0.945.3±3.3548.9±4.3

All tested settings achieve higher mean success than FQL-V (38.8±4.2). Thus, these parameters affect quantitative performance, but the advantage persists across the tested settings. The reference-time distribution plays a more direct role in controlling the locality–selection trade-off underlying SAWR's mechanism.

Q5: Reliability Gate

The gate uses w^i=(1−g)+gwi\hat{w}_i=(1-g)+gw_i, where g≈1g\approx1 retains full advantage weighting.

For both SAWR and FQL-V, g>0.99g>0.99 in 99.96% of OGBench offline updates and 100% of OGBench online and Adroit updates.

Thus, the gate almost never attenuates weighting and is unlikely to explain the performance gap. Since the gate is effectively inactive in nearly all updates, it is unlikely to account for the observed performance gap. This supports sample-anchored weighting and reference-time selection as the principal differences from FQL-V.

Thank you for the clarifications and additional experiments. The added TD-value baseline, sensitivity analyses, computational-cost measurements, and gate statistics address most of my concerns. The paired confidence intervals for entropy differences still include zero in two of the three diagnostic tasks, so the mode-preservation claims should remain appropriately qualified. Overall, the response strengthens my assessment of the paper, and I maintain my rating of 3 (weak accept) and confidence of 3.

Thank you for your thoughtful follow-up and positive assessment of our response. We sincerely appreciate your constructive feedback, which has helped us improve the rigor and clarity of our paper. We will continue to refine the manuscript in accordance with your suggestions.

Reviewer_ubFqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper wants to address the limitation of behavior cloning as the training strategy by proposing Sample- anchored Advantage-Weighted Regression (SAWR) by replacing conventional global value comparison with a sample-specific local comparison against a reference action constructed from each replay action.

Critical strengths

Clear writing and demonstration.

Critical weaknesses
  • The paper proposes to maintain diversity of the final policy by calculating a “sample-anchored advantage”, which compares the sampled action with an anchor action that is in the same mode. However, this ignores the difference in mode quality. Say we have two different modes with different constant values (say 100 vs 1), the proposed method will not prefer the mode with value 100.
  • Selecting FQL as the backbone is not very clean. FQL contains a distilled policy and backpropagation with the raw Q-value part, which serves as a “global value” across modes, that cannot really isolate the “sample-anchored advantage”. Using GFP or QAM may be cleaner.
  • The Mechanism Diagnostics section (Figure 4) shows different modes in trajectories instead of actions on a fixed state, which is different from the notion of mode that claimed to be preserved.
  • The results in Figure 4 somehow shows that FQL is better? With similar success in the original map and highest success in the lower route blocked map.
  • Why is the method targeting offline-to-online RL? Not sure which part is specifically designed for this. Feel like this method can be used for pure offline RL, but in Table 1, offline results are not very good compared with baseline FQL-V.
Questions for authors

See weakness

3 comments

We thank the reviewer for the constructive feedback. We address each concern below and provide new GFP results to assess SAWR's generality and robustness.

W1: Differences in Mode Quality

SAWR balances behavior preservation and value-based selection by allowing comparisons both within and across modes. Reference actions are constructed through flow completion from xt=(1−t)ϵ+tax_t=(1-t)\epsilon+ta. Larger tt favors local comparisons, while smaller tt allows cross-modal comparisons, capturing inter-modal value differences. Thus, the reference-time distribution balances behavior preservation and value-based selection.

Our reference-time ablation provides empirical evidence for this trade-off. Our proposed reference time distribution includes samples with reference times close to zero, allowing SAWR to favor higher-quality modes while retaining local optimization. This enables SAWR to preserve multimodal behaviors while improving task success rates. In contrast, the late reference time distribution in our ablation preserves multimodality but struggles to distinguish between high- and low-quality modes, leading to degraded performance.

W2: Backbone Selection

The effectiveness of SAWR is not limited to FQL. In our FQL experiments, all other learning components remain unchanged, isolating the effect of sample-anchored weighting. Additionally, our Q-chunking experiments with best-of-N action selection, without direct Q-gradient-based policy optimization, improve average online success from 67.5% (QC-V) to 72.0% (SAWR).

We further integrate SAWR into GFP. Specifically, we replace μθ(s,z)\mu_\theta(s,z) in Eq.10 of GFP with our proposed sample-anchored reference action. Across four task families, average online success increases from 53.37% to 57.00% under matched settings.

Task FamilyGFPGFP-SAWR
Cube triple play19.07 ± 2.95 → 43.73 ± 4.9023.87 ± 1.05 → 50.40 ± 10.83
Cube triple noisy23.47 ± 0.75 → 69.73 ± 10.1321.07 ± 5.49 → 72.00 ± 8.54
Puzzle 4×4 noisy19.47 ± 1.51 → 40.00 ± 0.0011.60 ± 2.67 → 45.60 ± 7.92
Scene noisy58.93 ± 1.00 → 60.00 ± 0.0055.47 ± 2.17 → 60.00 ± 0.00
Average30.23 ± 1.23 → 53.37 ± 3.7128.00 ± 2.18 → 57.00 ± 2.12

We also conduct an additional AntMaze route-blockage diagnostic. Both methods achieve comparable success on the original map (99.8% vs. 99.6%). However, after blocking the lower route, GFP-SAWR substantially improves success from 21.4% to 74.2%, with route entropy increasing from 0 to 0.091.

These additional results demonstrate that the benefits of sample-anchored weighting extend beyond FQL, both in online performance and robustness to route blockage.

W3: Action Modes in Mechanism Diagnostic

Trajectory diversity is not strictly equivalent to action multimodality at a fixed state. However, all trajectories in Figure 4 start from the same initial state, controlling for initial conditions and making behavioral differences easier to interpret.

Compared with low-dimensional projections of high-dimensional actions, trajectory visualization more directly illustrates distinct behaviors and their relevance to task completion. We use it as a behavioral-level diagnostic, not a direct measure of state-conditional action multimodality.

W4: FQL Performance in Mechanism Diagnostic

FQL achieves the best performance in the route-blockage diagnostic (Figure 4). Our claim is not that SAWR consistently outperforms unweighted FQL in every setting, but that it mitigates the behavioral concentration arising from value estimation errors and their amplification through conventional advantage weighting.

Specifically, SAWR partially restores the diversity and robustness lost by FQL-V. Moreover, in the Puzzle diagnostic, FQL achieves 76.8% success, whereas both FQL-V and SAWR reach 100%. Importantly, SAWR preserves more diverse successful behaviors than FQL-V despite their identical success rates.

These results highlight the intended benefit of SAWR: mitigating excessive behavioral concentration while maintaining strong task performance, rather than maximizing diversity or success in isolation.

W5: Focus on Offline-to-Online RL

Although SAWR can be applied during purely offline training, we focus on offline-to-online RL because behavioral diversity is particularly valuable for subsequent online exploration and optimization.

Preserving alternative behaviors may sacrifice some immediate offline performance while providing useful options during online adaptation. This trade-off is illustrated by our new GFP experiments: GFP-SAWR achieves slightly lower average offline success than GFP (28.00% vs. 30.23%), but higher success after the same online training budget (57.00% vs. 53.37%).

Thus, offline success alone does not fully capture the potential value of preserving alternative behaviors for subsequent online learning.

Thanks for your response. The response clarified some points, but the weakness was not fully addressed.

Thank you for your response and valuable feedback. We appreciate your suggestions and will continue to refine our manuscript accordingly.

Reviewer_XAgGOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors proposes SAWR . SAWR constructs local reference actions through partial noising and flow completion, then uses critic value differences to reweight the flow matching objective.

Critical strengths

SAWR is simple to integrate into existing flow-based RL algorithms. The experiments cover multiple benchmark families and task settings (offline ,off2on ), and some tasks show meaningful performance improvements.

The trajectory diversity and route-blockage analyses provide useful empirical evidence that local weighting can preserve alternative behaviors.

Critical weaknesses

The key motivation is not sufficiently justified. The observed mode collapse may stem from inaccurate value estimation or inappropriate weighting temperatures. With accurate value estimates and a suitable temperature, AWR should not severely suppress modes of comparable quality.

The policy improvement rationale is unclear. Local value comparisons may preserve diversity but also suppress meaningful differences between genuinely better and worse behavior modes.

The theoretical and empirical evidence is limited. the empirical gains are concentrated in a few tasks. Comparisons with simpler temperature-adjusted AWR baselines are missing.

Questions for authors

Please refer to weakness.

3 comments

We thank the reviewer XAgG for the constructive feedback and for recognizing SAWR's simplicity, experimental coverage, and behavioral diversity preservation. We clarify our motivation and mechanism, and provide additional temperature-controlled experiments below.

W1: Motivation and Mode Collapse

SAWR does not assume that AWR causes mode collapse under ideal conditions. Our motivation concerns mode suppression from imperfect critic estimates amplified by exponential advantage weighting.

As shown in Figure 1, modes with comparable optimal values may receive different estimates due to noise or limited coverage. Exponential weighting can amplify these differences, suppressing potentially valuable modes.

SAWR instead employs sample-anchored relative value comparisons. When nearby anchor and reference actions share a component of critic estimation error, that component can cancel in their value difference, potentially reducing sensitivity to estimation errors.Our goal is to improve robustness under imperfect value estimation, without assuming that AWR fails under accurate value estimation

W2: Policy Improvement and Mode Quality

SAWR allows both within-mode and cross-modal comparisons while considering differences in mode quality. Reference actions are constructed through flow completion from xt=(1−t)ϵ+tax_t=(1-t)\epsilon+ta. Larger reference times favor local comparisons and preserve diversity; smaller times reduce dependence on the anchor, enabling cross-modal comparisons and value selection.

The reference-time distribution thus balances behavior preservation and value-based selection, instead of simply eliminating comparisons between modes of different quality.

Our reference-time ablation supports this trade-off. The proposed distribution includes near-zero reference times, enabling higher-quality mode selection while preserving multimodality. In contrast, late-time sampling yields lower success, consistent with weaker mode-quality discrimination. As tt approaches zero, cross-modal comparisons become more likely: a higher-valued reference reduces the anchor's weight, favoring higher-value behaviors across modes.

Preserved behaviors may remain suboptimal after offline training, but can provide additional opportunities for exploration and policy improvement during online training. This helps explain why preserving behavioral diversity can improve final performance without necessarily maximizing immediate offline success.

W3: Experimental Evidence and Temperature Adjustment

SAWR achieves improvements across multiple benchmarks and policy backbones. On OGBench, average performance improves from 54.4 (FQL-V) to 56.7 (SAWR), and on Adroit from 85.9 to 91.0. Beyond FQL, SAWR improves online success rates from 67.5% to 72.0% on Q-chunking and from 53.37% to 57.00% on GFP. (For details of the GFP experiments, see our response to Reviewer ubFq.)

To directly address the temperature concern, we additionally evaluate FQL-V with fixed β values of 0.3, 1, and 3 for ω∝exp⁡(βA(s,a))\omega \propto \exp(\beta A(s, a)).

Multimodal behavior preservation:

Metricβ=0.3β=1β=3FQL-VSAWR
Puzzle: mode entropy0.000±0.0000.116±0.2000.019±0.0320.027±0.0600.282±0.332
Scene: mode entropy0.033±0.0570.000±0.0000.019±0.0320.000±0.0000.213±0.140
AntMaze: route entropy0.141±0.1050.164±0.0530.019±0.0270.000±0.0000.154±0.099
AntMaze: blocked success (%)63.67±38.3282.67±10.5371.33±39.8440.20±32.5068.80±18.60

All methods achieve 100% success on Puzzle and Scene, while SAWR achieves higher mean mode entropy. On AntMaze, β=1 achieves better blocked-route success than SAWR, showing that temperature tuning can be effective on individual tasks.

Additional offline-to-online evaluation on OGBench AntSoccer arena-stitch (5 tasks):

Metricβ=0.3β=1β=3FQL-VSAWR
Offline (1M)6.8±3.14.8±1.40.0±0.07.8±2.310.6±1.3
Online (2M)35.7±3.13.5±1.43.6±3.238.8±4.255.8±5.7

These results suggest that no single tested fixed β consistently reproduces SAWR's benefits across the evaluated settings, although temperature tuning can outperform SAWR on individual tasks.

For a fixed critic, adjusting β changes weighting sharpness without altering action rankings under a shared state-dependent baseline. SAWR instead introduces sample-dependent reference actions, changing the relative value comparisons underlying the weights. This provides a distinct mechanism for balancing behavioral diversity and value-based selection, beyond global temperature adjustment.

Thank you for the additional experiments. While I still find the paper’s insights limited, I will raise my rating to 2.

Thank you for your feedback and suggestions. We appreciate your time and will take your comments into consideration as we continue to improve our manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers and Area Chair,

We thank you for your constructive feedback and for recognizing SAWR's simplicity, modularity, broad evaluation, and informative behavioral diagnostics. Several concerns regarding our motivation and cross-mode comparisons relate to points already discussed in the original submission (Sections 2.2 and 3.2–3.3). We apologize that our current presentation may not have made the relevance of these discussions sufficiently clear. We have therefore emphasized and elaborated on these existing discussions to make their connection to the questions raised more explicit. We also provide additional experimental results that further support our conclusions. The main revisions are highlighted in blue in the revised manuscript.

  1. Motivation. The original submission discussed how exponential AWR weighting can amplify critic errors (Section 2.2). We have made this motivation more explicit in the Abstract and Introduction and clarified the shared-bias cancellation mechanism in Section 3.2.

  2. Cross-mode value comparisons. The original method already balances local and cross-mode comparisons through reference time selection and distribution design (Sections 3.2–3.3). Smaller reference times allow comparisons across modes of different quality, while larger times favor local comparisons. We have further emphasized this role of reference time in the Abstract, Introduction, and Method section.

  3. Additional experiments. We added fixed-temperature AWR comparisons (Appendix B.6), GFP experiments and a route-blockage diagnostic (Appendix B.5), ESS-target and Euler-step ablations (Appendix B.7), and training-time measurements (Appendix A.5). These experiments further support our conclusion that sample-anchored comparisons can improve online performance and mitigate behavioral concentration, with benefits extending beyond FQL. No single tested fixed temperature consistently reproduces these benefits across the evaluated settings.

We thank you again for your time and valuable feedback, which have helped us improve the paper. We hope these clarifications and additional results address your main concerns, and we look forward to further discussion with you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.