- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies a failure mode of advantage-weighted regression when the policy is a flow model. Conventional AWR uses a state-value baseline shared by all actions at a given state, so the relative weighting of different behavior modes depends on global exponentiated action values; noisy or miscalibrated value estimates can therefore suppress valid modes. SAWR replaces that shared baseline with a per-sample reference action, obtained by partially noising the replay action and completing it with the current flow policy. The value difference between the anchor and this reference then weights the conditional flow-matching loss. The authors prove that when the reference remains in the anchor’s mode, the common mode offset cancels. They also introduce a reference-time distribution, robust batch normalization with ESS targeting, and a critic-reliability gate. Instantiated on FQL, SAWR improves offline-to-online success on OGBench and D4RL Adroit; diagnostics suggest higher mode entropy and better robustness after route blockage.
Critical strengths
The intervention is clean and modular: it adds no trainable parameters and only reweights the flow-matching loss, so it can be applied to other flow-policy algorithms. The empirical study also moves beyond aggregate success, using mode entropy, same-mode rates, manipulation-order diagnostics, and a route-blockage test; the Q-chunking instantiation further supports generality.
Critical weaknesses
- First, the mode-preservation claim outruns the evidence. Entropy gains are modest and high-variance—e.g., Scene 0.213±0.140 versus 0.000±0.000 for FQL-V; Puzzle 0.027±0.060 to 0.282±0.332—and SAWR recovers only part of FQL’s diversity. The route-blockage result rests on one task. The claim of mitigating mode suppression should be stated more cautiously.
- Second, key design choices lack justification. The reference-time density, ESS target, Euler steps, and gate parameters are hand-tuned without systematic sensitivity analysis, so the locality–signal trade-off is not operationalized.
- Moreover, FQL-V learns its value target from a Monte Carlo estimate under the one-step policy rather than a standard TD or expectile objective, which may disadvantage the baseline; no standard AWR control is reported.
Questions for authors
- Can you provide paired tests or confidence intervals for mode-entropy differences, and report per-task entropy across all diagnostics?
- Why use the nonstandard value target for FQL-V, and would a conventional AWR baseline close the gap?
- What is the computational overhead of SAWR relative to FQL and FQL-V?
- How sensitive is performance to the shape of Eq. 14, the ESS target, and the number of Euler steps?
- How often is the reliability gate active, and how much of the observed mode preservation is attributable to the gate versus the sample-anchored advantage?
3 comments
Thank you for the clarifications and additional experiments. The added TD-value baseline, sensitivity analyses, computational-cost measurements, and gate statistics address most of my concerns. The paired confidence intervals for entropy differences still include zero in two of the three diagnostic tasks, so the mode-preservation claims should remain appropriately qualified. Overall, the response strengthens my assessment of the paper, and I maintain my rating of 3 (weak accept) and confidence of 3.
Thank you for your thoughtful follow-up and positive assessment of our response. We sincerely appreciate your constructive feedback, which has helped us improve the rigor and clarity of our paper. We will continue to refine the manuscript in accordance with your suggestions.
We thank the reviewer for the constructive assessment. We address the five questions with additional analyses and experiments.
Q1: Mode Entropy
We selected these three tasks because they admit distinct solutions with comparable returns, making them suitable for diagnosing mode preservation. We report per-task entropy and paired SAWR–FQL-V differences with 95% t-CIs:
These results suggest that SAWR can partially mitigate mode suppression.
Q2: FQL-V Value Target
FQL-V uses a policy-averaged value baseline, consistent with AWAC-style advantage estimation (Nair et al., 2020). To examine whether this choice affects the comparison, we retrained FQL-V with a bootstrapped TD value target (TD-V), keeping other components unchanged.
FQL-V provides an architecture-controlled AWR-style baseline, using a shared state-value reference while retaining the same flow-policy backbone. The TD-V experiment further tests whether its value-target construction accounts for the observed gap. Neither variant closes the average online performance gap with SAWR, suggesting that the gains cannot be explained solely by the original value-target choice.
Q3: Computational Overhead
On RTX 3090 GPUs (one process/GPU), training times for FQL / FQL-V / SAWR were 2.00 / 2.65 / 2.30 h on AntSoccer arena navigate task1 and 1.75 / 2.42 / 1.89 h on Adroit pen-expert.
SAWR increases training time by only 15% and 8% over FQL, respectively, while remaining faster than FQL-V. It introduces no additional trainable networks or inference-time overhead.
Q4: Hyperparameter Sensitivity
Reference-time distribution. Eq. 14 is an interior-peaked, piecewise-linear density balancing local comparisons with near-zero reference times for cross-modal value selection.
Our Early/Mid/Late/Uniform ablations in Figure 6 illustrate this trade-off: later times better preserve diversity but may weaken value discrimination, whereas earlier times favor value-based selection. These results support the locality–selection trade-off, rather than the unique optimality of Eq. 14. Our goal is to encourage local comparisons while retaining sufficient global comparisons, balancing multimodal behavior preservation with task success. Alternative distribution shapes may affect quantitative performance, but the key mechanism lies in controlling this locality–selection trade-off.
ESS and Euler steps. On five AntSoccer arena-stitch tasks, online success rates (%) were:
All tested settings achieve higher mean success than FQL-V (38.8±4.2). Thus, these parameters affect quantitative performance, but the advantage persists across the tested settings. The reference-time distribution plays a more direct role in controlling the locality–selection trade-off underlying SAWR's mechanism.
Q5: Reliability Gate
The gate uses w^i=(1−g)+gwi, where g≈1 retains full advantage weighting.
For both SAWR and FQL-V, g>0.99 in 99.96% of OGBench offline updates and 100% of OGBench online and Adroit updates.
Thus, the gate almost never attenuates weighting and is unlikely to explain the performance gap. Since the gate is effectively inactive in nearly all updates, it is unlikely to account for the observed performance gap. This supports sample-anchored weighting and reference-time selection as the principal differences from FQL-V.