- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies adversarial observation perturbations against recently developed one-step flow policies in offline RL and introduces BDAP, a generator-based attack trained against a frozen FQL surrogate to simultaneously lower surrogate value and move the action away from a behavior-only reference.
Critical strengths
BDAP is conceptually simple but practically interesting: after task-specific offline training against an FQL surrogate, it creates perturbations with one generator forward pass and no deployed-victim queries, yet transfers strongly to MeanFlowQL, SSCP, and GFP; this is meaningfully different from iterative victim-dependent attacks.
Critical weaknesses
RCF is not evaluated against a sufficiently adaptive adversary that explicitly targets the full DMBP+RCF pipeline, so the current experiments do not establish robustness to a knowledgeable attacker. In addition, RCF requires simulator rollouts with access to the underlying clean state to construct residual targets, which is a substantial assumption in an offline-RL setting; the paper should not imply that the defense can necessarily be trained from a static offline dataset alone.
Questions for authors
-
Can the authors attack the complete defended pipeline using EOT/BPDA or another appropriate approximation through the stochastic DMBP reconstruction and RCF correction? Without such an evaluation, it is difficult to exclude the possibility that part of the observed robustness comes from attacking a different computation graph than the one actually used at deployment.
-
RCF training requires uniformly perturbed simulator rollouts and residual targets formed using the clean underlying state. This is substantially stronger than the usual static-data assumption motivating offline RL, where additional simulator/environment interaction may not be available. Please state this requirement prominently and evaluate whether an RCF variant can be trained from the original offline dataset alone.
-
The APGD baseline maximizes squared action deviation, while BDAP explicitly combines value degradation with departure from a behavior reference. Could the authors include a stronger gradient-based baseline that directly minimizes the victim/surrogate critic value or optimizes an objective more closely matched to downstream return degradation? Existing RL robustness research has shown that the attack objective can substantially affect the apparent vulnerability of a policy.
-
BDAP is described as computationally efficient because deployment requires only one generator evaluation, but each generator is trained separately for a task and perturbation budget. Please report the complete offline training cost and the number of surrogate interactions/gradient steps required to obtain the generator. A cross-task or cross-budget transfer experiment would also clarify how well this preparation cost can be amortized.
-
Please report the performance of DMBP and DMBP+RCF on completely clean, unperturbed observations, rather than comparing the attacked defended policies only to “Raw clean.”
-
Several OGBench victim/task combinations have very low or even zero raw-clean success, particularly for some MeanFlowQL and SSCP settings, creating substantial floor effects in attack evaluation. Could the authors additionally report relative degradation or an aggregate restricted to tasks where all compared victims achieve a meaningful minimum clean performance?
-
The theoretical analysis bounds residual error under a joint distribution Q, but the predictor is trained under P_tr generated by uniformly perturbed FQL rollouts and then evaluated under adversarial distributions and different victim policies. Could the authors explain how the theory accounts for this distribution shift, or provide a divergence/coverage analysis between the RCF training distribution and the defended trajectory distributions?
2 comments
Your Comments are Highly Valuable to Us [Deadline in One Day]
Dear reviewer z88v,
We have carefully considered all the valuable points and concerns you raised in your initial review through the initial rebuttal. As the deadline for discussion is approaching in only one day, we respectfully ask whether our response has addressed your concerns, or you still have questions to raise. Your comments are highly valuable to us. Thank you once again for your professional time and insightful effort in reviewing our submission.
Best regards,
Authors
We sincerely thank the reviewer for the thoughtful assessment of our work. We also appreciate the constructive feedback and suggestions.
W1/Q1. Full-pipeline adaptive attacks. RCF improves adaptive recovery. We attack DMBP and DMBP+RCF separately, minimizing frozen victim-critic value at the true state using 40 steps, EOT=16, three restarts, identity BPDA through DMBP, exact RCF gradients, and actual stochastic defense forward passes. We fix policy latent/history during search and select the lowest EOT-value candidate. At ϵ=0.1, ρ=0.3, pooled OGBench success is 11.4%/12.8% for DMBP/RCF, a 1.4-point gain. We report white-box evaluation separately from zero-target-query transfer (Appendix D.2, Table 22(a)). New evaluations use 5 seeds x 50 episodes.
W1/Q2. Offline-only fitting. RCF trained from the original offline dataset improves all six reported aggregates with zero new simulator transitions. Original RCF uses clean-state simulator rollouts. The offline variant uniformly perturbs stored states at scale 0.4 and replays frozen DMBP with logged actions/history, resetting at episode boundaries. Stored clean states label residuals. The predictor and correction interface are retained.
Scores are success percentages except Adroit normalized return. Adversarial ϵ=0.1, Gaussian scale 0.2. Twenty logged trajectories yield 3,256/15,980/3,180 pairs for cube-single/antmaze-medium-play/pen-human, split 90%/10% by sample. Cube-single's simulator comparator uses 3,180 pairs. Replay uses logged actions (Appendix D.3, Table 22(b)).
Q3. Stronger gradient objective. We directly test value-based gradient optimization. APGD-Q uses BDAP's frozen FQL actor and arithmetic-mean two-head critic. min∥δ/σ∥∞≤0.1QFQL(s,πFQL(s+δ,z)). We fix z and evaluate the critic at the true state. APGD-Q (10) changes only the original objective/direction. 40 x 4 uses 160 updates. Transfer uses no victim access.
Metrics are as above. Lower is stronger. APGD-Q (40 x 4) yields the lowest means. BDAP produces perturbations in one forward pass (Appendix D.4, Table 25(a)).
Q4. Preparation and reuse. Preparation uses 68 FQL surrogates (1M/0.5M updates on OGBench/D4RL) and 28 references (0.5M updates). Each generator uses 200,000 updates, batch 256, eight latents, giving 51.2M state occurrences and 409.6M evaluations per frozen actor/critic ensemble/reference. Median process time over 68 fits at ϵ=0.05 is 369.0 s, including loading/compilation/fitting/checkpointing, excluding pretraining.
Freezing the ϵ=0.1 generator and scaling its output at test budgets 0.05/0.1/0.2 retains 87.9-98.7% of benchmark-level attack-induced degradation. For three budgets, generators/updates fall from 3/600,000 to 1/200,000 (66.7% fewer generator updates). Surrogate/reference preparation is shared. Deployment stays one forward, and the same generator serves all four policy families (Appendix D.5, Table 26).
Q5. Clean observations. As requested, both defenses stay active on clean inputs (ϵ=0, ρ=0.3), with frozen weights. Raw/DMBP/RCF scores are reported below.
RCF improves on DMBP in all three benchmarks. The table reports clean-input performance with defenses active (Appendix D.2, Table 22(c)).
Q6. Floor effects. To control floor effects, we select tasks where every victim has at least 10% clean success (20 pairs). Clean/APGD/min-Q/BDAP success is 72.1/23.0/16.2/10.4%. BDAP causes 85.6% relative degradation. Its advantage over original APGD/min-Q persists excluding FQL and at a 20% threshold (12 pairs. Appendix D.4, Table 25(b)).
Q7. Distribution shift. Our bounds relate reconstruction error under the defended deployment law to performance. Each attack/victim induces its own law, including history/randomness. With the same target map as training, let Qp be the deployment law of (φ,ytr). If Qp≪Ptr and dQp/dPtr≤C, then LQ(f)≤CLPtr(f). This is a sufficient population condition. Transfer performance is evaluated empirically (Appendix D.1).