acceptodds
Under review as a conference paper at ICLR 2027

Attacking and Defending One-Step Flow Policies in Offline Reinforcement Learning

Abstract

One-step flow policies enable efficient action generation in offline reinforcement learning, but remain vulnerable to observation perturbations. We introduce Behavior-Decoupled Adversarial Perturbation (BDAP), which trains a generator against a frozen surrogate to lower estimated action value at the true state while increasing action deviation from a behavior-only reference. At deployment, BDAP generates perturbations in one forward pass without victim queries or gradients. The existing Diffusion Model-Based Predictor (DMBP) partially mitigates these attacks but leaves reconstruction errors. We therefore propose the Residual-Corrected Filter (RCF), which predicts these residuals from the perturbed observation and the DMBP-reconstructed observation and corrects the reconstruction before policy execution. RCF is trained only on simulator rollouts with uniform random observation perturbations, keeping the policy and DMBP frozen, and requires no attack-type information at deployment. Our analysis bounds reconstruction error through prediction loss, target clipping, and correction constraints, and connects the remaining error to a return-gap bound relative to the clean policy under Lipschitz assumptions. Experiments on OGBench and D4RL across four policy families demonstrate effective cross-policy transfer of BDAP and aggregate performance gains from RCF over DMBP under adversarial attacks and Gaussian noise.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, one-step flow policies, Behavior-Decoupled Adversarial Perturbation, Residual-Corrected Filter

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_z88vOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies adversarial observation perturbations against recently developed one-step flow policies in offline RL and introduces BDAP, a generator-based attack trained against a frozen FQL surrogate to simultaneously lower surrogate value and move the action away from a behavior-only reference.

Critical strengths

BDAP is conceptually simple but practically interesting: after task-specific offline training against an FQL surrogate, it creates perturbations with one generator forward pass and no deployed-victim queries, yet transfers strongly to MeanFlowQL, SSCP, and GFP; this is meaningfully different from iterative victim-dependent attacks.

Critical weaknesses

RCF is not evaluated against a sufficiently adaptive adversary that explicitly targets the full DMBP+RCF pipeline, so the current experiments do not establish robustness to a knowledgeable attacker. In addition, RCF requires simulator rollouts with access to the underlying clean state to construct residual targets, which is a substantial assumption in an offline-RL setting; the paper should not imply that the defense can necessarily be trained from a static offline dataset alone.

Questions for authors
  1. Can the authors attack the complete defended pipeline using EOT/BPDA or another appropriate approximation through the stochastic DMBP reconstruction and RCF correction? Without such an evaluation, it is difficult to exclude the possibility that part of the observed robustness comes from attacking a different computation graph than the one actually used at deployment.

  2. RCF training requires uniformly perturbed simulator rollouts and residual targets formed using the clean underlying state. This is substantially stronger than the usual static-data assumption motivating offline RL, where additional simulator/environment interaction may not be available. Please state this requirement prominently and evaluate whether an RCF variant can be trained from the original offline dataset alone.

  3. The APGD baseline maximizes squared action deviation, while BDAP explicitly combines value degradation with departure from a behavior reference. Could the authors include a stronger gradient-based baseline that directly minimizes the victim/surrogate critic value or optimizes an objective more closely matched to downstream return degradation? Existing RL robustness research has shown that the attack objective can substantially affect the apparent vulnerability of a policy.

  4. BDAP is described as computationally efficient because deployment requires only one generator evaluation, but each generator is trained separately for a task and perturbation budget. Please report the complete offline training cost and the number of surrogate interactions/gradient steps required to obtain the generator. A cross-task or cross-budget transfer experiment would also clarify how well this preparation cost can be amortized.

  5. Please report the performance of DMBP and DMBP+RCF on completely clean, unperturbed observations, rather than comparing the attacked defended policies only to “Raw clean.”

  6. Several OGBench victim/task combinations have very low or even zero raw-clean success, particularly for some MeanFlowQL and SSCP settings, creating substantial floor effects in attack evaluation. Could the authors additionally report relative degradation or an aggregate restricted to tasks where all compared victims achieve a meaningful minimum clean performance?

  7. The theoretical analysis bounds residual error under a joint distribution Q, but the predictor is trained under P_tr generated by uniformly perturbed FQL rollouts and then evaluated under adversarial distributions and different victim policies. Could the authors explain how the theory accounts for this distribution shift, or provide a divergence/coverage analysis between the RCF training distribution and the defended trajectory distributions?

2 comments

We sincerely thank the reviewer for the thoughtful assessment of our work. We also appreciate the constructive feedback and suggestions.

W1/Q1. Full-pipeline adaptive attacks. RCF improves adaptive recovery. We attack DMBP and DMBP+RCF separately, minimizing frozen victim-critic value at the true state using 40 steps, EOT=16, three restarts, identity BPDA through DMBP, exact RCF gradients, and actual stochastic defense forward passes. We fix policy latent/history during search and select the lowest EOT-value candidate. At ϵ=0.1\epsilon=0.1, ρ=0.3\rho=0.3, pooled OGBench success is 11.4%/12.8% for DMBP/RCF, a 1.4-point gain. We report white-box evaluation separately from zero-target-query transfer (Appendix D.2, Table 22(a)). New evaluations use 5 seeds x 50 episodes.

W1/Q2. Offline-only fitting. RCF trained from the original offline dataset improves all six reported aggregates with zero new simulator transitions. Original RCF uses clean-state simulator rollouts. The offline variant uniformly perturbs stored states at scale 0.4 and replays frozen DMBP with logged actions/history, resetting at episode boundaries. Stored clean states label residuals. The predictor and correction interface are retained.

SettingDMBPOffline-only RCFOriginal RCF
OGBench/BDAP15.216.017.1
OGBench/APGD19.520.522.4
OGBench/min-Q19.320.321.7
OGBench/Gaussian23.124.025.4
AntMaze/BDAP43.044.146.3
Adroit/BDAP32.433.235.2

Scores are success percentages except Adroit normalized return. Adversarial ϵ=0.1\epsilon=0.1, Gaussian scale 0.2. Twenty logged trajectories yield 3,256/15,980/3,180 pairs for cube-single/antmaze-medium-play/pen-human, split 90%/10% by sample. Cube-single's simulator comparator uses 3,180 pairs. Replay uses logged actions (Appendix D.3, Table 22(b)).

Q3. Stronger gradient objective. We directly test value-based gradient optimization. APGD-Q uses BDAP's frozen FQL actor and arithmetic-mean two-head critic. min⁡∥δ/σ∥∞≤0.1QFQL(s,πFQL(s+δ,z)).\min_{\|\delta/\sigma\|_\infty\leq0.1}Q_{\mathrm{FQL}}(s,\pi_{\mathrm{FQL}}(s+\delta,z)). We fix zz and evaluate the critic at the true state. APGD-Q (10) changes only the original objective/direction. 40 x 4 uses 160 updates. Transfer uses no victim access.

AttackOGBenchAntMazeAdroit
APGD7.261.535.6
min-Q9.649.638.9
BDAP3.138.734.4
APGD-Q (10)4.241.834.1
APGD-Q (40 x 4)2.836.032.6

Metrics are as above. Lower is stronger. APGD-Q (40 x 4) yields the lowest means. BDAP produces perturbations in one forward pass (Appendix D.4, Table 25(a)).

Q4. Preparation and reuse. Preparation uses 68 FQL surrogates (1M/0.5M updates on OGBench/D4RL) and 28 references (0.5M updates). Each generator uses 200,000 updates, batch 256, eight latents, giving 51.2M state occurrences and 409.6M evaluations per frozen actor/critic ensemble/reference. Median process time over 68 fits at ϵ=0.05\epsilon=0.05 is 369.0 s, including loading/compilation/fitting/checkpointing, excluding pretraining.

Freezing the ϵ=0.1\epsilon=0.1 generator and scaling its output at test budgets 0.05/0.1/0.2 retains 87.9-98.7% of benchmark-level attack-induced degradation. For three budgets, generators/updates fall from 3/600,000 to 1/200,000 (66.7% fewer generator updates). Surrogate/reference preparation is shared. Deployment stays one forward, and the same generator serves all four policy families (Appendix D.5, Table 26).

Q5. Clean observations. As requested, both defenses stay active on clean inputs (ϵ=0\epsilon=0, ρ=0.3\rho=0.3), with frozen weights. Raw/DMBP/RCF scores are reported below.

BenchmarkRawDMBPRCF
OGBench success (%)31.530.931.1
AntMaze success (%)69.368.368.7
Adroit return42.241.541.8

RCF improves on DMBP in all three benchmarks. The table reports clean-input performance with defenses active (Appendix D.2, Table 22(c)).

Q6. Floor effects. To control floor effects, we select tasks where every victim has at least 10% clean success (20 pairs). Clean/APGD/min-Q/BDAP success is 72.1/23.0/16.2/10.4%. BDAP causes 85.6% relative degradation. Its advantage over original APGD/min-Q persists excluding FQL and at a 20% threshold (12 pairs. Appendix D.4, Table 25(b)).

Q7. Distribution shift. Our bounds relate reconstruction error under the defended deployment law to performance. Each attack/victim induces its own law, including history/randomness. With the same target map as training, let Qp\mathcal Q_p be the deployment law of (φ,ytr)(\varphi,y^{\mathrm{tr}}). If Qp≪Ptr\mathcal Q_p\ll P_{\mathrm{tr}} and dQp/dPtr≤Cd\mathcal Q_p/dP_{\mathrm{tr}}\leq C, then LQ(f)≤CLPtr(f)\mathcal L_{\mathcal Q}(f)\leq C\mathcal L_{P_{\mathrm{tr}}}(f). This is a sufficient population condition. Transfer performance is evaluated empirically (Appendix D.1).

AuthorsOpenReview ↗

Your Comments are Highly Valuable to Us [Deadline in One Day]

Dear reviewer z88v,

We have carefully considered all the valuable points and concerns you raised in your initial review through the initial rebuttal. As the deadline for discussion is approaching in only one day, we respectfully ask whether our response has addressed your concerns, or you still have questions to raise. Your comments are highly valuable to us. Thank you once again for your professional time and insightful effort in reviewing our submission.

Best regards,

Authors

Reviewer_mxdPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies observation attacks against one-step flow policies in offline reinforcement learning. BDAP trains a bounded perturbation generator using a frozen FQL surrogate, its critic, and an independently learned behavior-only reference. The trained generator attacks other policy families without target queries at deployment. The defense adds RCF to a frozen DMBP state reconstructor. RCF learns to predict reconstruction residuals from uniformly perturbed simulator rollouts with clean training targets, then applies masked and bounded corrections before policy execution. The analysis gives a reconstruction-error bound and a conditional connection to the clean policy's return. Experiments cover OGBench and D4RL with four policy families and report transferable attacks and aggregate improvements over DMBP.

Critical strengths
  • BDAP provides a concrete and effective transfer attack with no deployment-time target queries. In Table 1, the FQL-trained generator reduces MeanFlowQL success on cube-single from 49.6% to 4.8% and GFP success on scene from 49.2% to 8.8%. These results matter because the attack is not merely optimized separately against every reported victim. Table 11 also gives useful evidence that combining the value and behavior terms can improve attack effectiveness.

  • The defense is specified precisely enough to separate its learned correction from the frozen policy and reconstructor. Section 3.2 accounts for masking, target clipping, and correction limits rather than analyzing an unconstrained predictor that differs from the implementation. Its conditional analysis and the empirical gains make RCF a plausible calibration mechanism, even though they do not establish adaptive robustness.

Critical weaknesses
  1. The defense is evaluated against attacks that do not attack the defended system. Section B.1 explicitly says that APGD and min-Q select perturbations using the unfiltered policy, after which DMBP and RCF process the observation. BDAP is likewise trained against the raw FQL surrogate. A perturbation that maximally changes the raw action need not remain damaging after filtering, thus recovery from these perturbations does not establish resistance to an attacker that anticipates the filter. This distinction is consequential for the defense contribution. DiffPure evaluates attacks through its purification procedure precisely because preprocessing changes the attacked function. I am not asking for certified robustness or target queries in the transfer setting. A defense-aware surrogate attack could still use zero target queries. The paper should either support that stronger setting or explicitly limit its defense claim to recovery against filter-oblivious attacks. The return-gap bound does not resolve this issue because it requires reconstruction error under the defended deployment distribution.

  2. What part of RCF's improvement requires the proposed residual design rather than the additional supervised calibration data? Section B.2 collects fresh FQL simulator rollouts and uses clean states to label reconstruction errors, while the DMBP comparator remains frozen. I appreciate that this access is disclosed. It nevertheless changes the available information relative to the fixed-dataset motivation and to DMBP. Table 2 establishes a benefit from adding this trained module, but does not separate the residual mechanism from the benefit of extra clean-target data. Report the interaction budget and compare with a calibration or reconstructor adaptation baseline given the same data, without retraining the control policy. If that comparison is unavailable, the contribution should be framed more narrowly as simulator-assisted calibration of frozen offline policies, with the remaining attribution limitation made explicit.

Questions for authors
  1. Do any existing results generate perturbations against a defended surrogate, including the filter, rather than the raw policy? If not, is filter-oblivious attack generation an intended restriction of the defense claim, and what information about DMBP and RCF is the attacker assumed to know? This is a clarification of the evaluated threat model, not a request for a new experiment during rebuttal.

  2. How many simulator transitions and clean targets are used for each RCF predictor, how are training and validation trajectories separated, and was any comparison already conducted in which an alternative calibration method receives the same data? Existing protocol details or results could materially change my assessment of the second concern.

3 comments

We sincerely thank the reviewer for the thoughtful assessment of our work. We also appreciate the constructive feedback and suggestions.

W1/Q1. Threat model and adaptive evaluation. The original results use filter-oblivious attack generation. The added evaluation directly attacks the full pipeline. Originally, APGD uses unfiltered victim-action gradients, min-Q also uses the victim critic, and BDAP transfers from a raw FQL surrogate without victim access. Original attack generation uses neither DMBP/RCF parameters nor defended outputs.

We additionally attack each complete deployed pipeline separately, exposing its frozen victim actor/critic, DMBP, and RCF. The attack minimizes critic value at the true state using 40 steps, EOT=16, three restarts, identity BPDA through stochastic DMBP, and exact RCF gradients. Forward evaluation uses the actual defense. With ϵ=0.1\epsilon=0.1, ρ=0.3\rho=0.3, and 5 seeds x 50 episodes, the table reports pooled OGBench success over 200 combinations.

AttackDMBPDMBP+RCF
Filter-oblivious BDAP15.217.1
Full-pipeline adaptive11.412.8

RCF retains a 1.4-percentage-point gain when the attacker targets the deployed computation. This adaptive test grants victim-model access. BDAP's transfer deployment uses zero target queries. The return bound complements the experiment by relating defended-deployment reconstruction error to return (Appendix D.2, Table 22).

W2. Residual design versus extra supervision. Our added data-matched experiments establish a benefit beyond additional supervision. We compare RCF with closed-form ridge calibration, a size-matched Direct-State MLP, and DMBP fine-tuning. All fitted alternatives share RCF's simulator rollouts, clean-state supervision, and sample split. Policies remain frozen. RCF, ridge, and Direct-State MLP receive identical (s~t,s^t)(\tilde s_t,\hat s_t) inputs. The latter two predict clean states. RCF predicts residuals. Frozen DMBP is the unadapted reference.

The table reports pooled OGBench success (%) across 200 combinations.

DefenseBDAPAPGDmin-QGaussian
DMBP15.219.519.323.1
+ Ridge15.720.220.023.8
+ Direct-State MLP16.220.820.624.3
+ DMBP fine-tuning16.521.321.124.8
+ RCF17.122.421.725.4

Adversarial budgets are ϵ=0.1\epsilon=0.1. Gaussian scale is 0.2. RCF ranks best in all four perturbation aggregates and each policy-family aggregate under BDAP. The complete RCF design provides a further benefit with the same supervision (main-text Table 3 and Appendix D.3, Table 23). Optimization budgets for each method are reported below.

Q2. Actual data budget and fitting protocol. We report the actual counts for the three task-specific predictors below, each using 20 simulator rollouts.

TaskTransitionsClean targetsTrainValidation
OGBench/cube-single3,2003,1802,862318
D4RL/antmaze-medium-play16,00015,98014,3821,598
D4RL/pen-human3,2003,1802,862318

Collection uses frozen FQL and DMBP, RCF disabled, and uniform training perturbations of scale 0.4. Clean states label residuals and retain the original filter initialization/first-step warmup. Later reconstruction uses perturbed inputs and history. The 90%/10% split is by sample using shuffle seed 0. Samples from a trajectory can appear in both sets. All fitted alternatives use the same split.

Direct-State MLP and RCF use two 256-unit ReLU hidden layers, Adam at 10−310^{-3}, batch 256, and 400/300 epochs on OGBench/D4RL. Fine-tuning updates the entire DMBP noise predictor with the original non-Markovian diffusion loss, Adam at 10−510^{-5}, batch 256, and 25 epochs. Ridge is solved in closed form. For cube-single/AntMaze/pen-human, MLP and RCF use 4,800/17,100/3,600 updates, versus 300/1,425/300 for fine-tuning. OGBench RCF uses minimum-validation-MSE checkpoint selection and a validation-derived coordinate mask. D4RL uses the final checkpoint and all coordinates (Appendix D.3, Table 24).

These data-matched controls use the same sample allocations and frozen control policies as RCF.

AuthorsOpenReview ↗

Your Comments are Highly Valuable to Us [Deadline in One Day]

Dear reviewer mxdP,

We have carefully considered all the valuable points and concerns you raised in your initial review through the initial rebuttal. As the deadline for discussion is approaching in only one day, we respectfully ask whether our response has addressed your concerns, or you still have questions to raise. Your comments are highly valuable to us. Thank you once again for your professional time and insightful effort in reviewing our submission.

Best regards,

Authors

I checked the two new comparisons. The adaptive evaluation shows a 1.4-point gain over frozen DMBP, but it omits every alternative trained on the same simulator data. In the separate data-matched comparison, RCF leads the strongest baseline by 0.6 to 1.1 points. My main concern remains unresolved because neither table shows whether RCF still outperforms an equally supervised alternative when the attack targets the full defense. I keep the original rate.

Reviewer_wgaTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper provides BDAP, a bounded perturbation generation algorithm that is trained against a frozen FQL surrogate and its critic to serve as the attacker. It also provides a defense mechanism, RCPF that predicts residuals left by frozen DMBP reconstruction to defend. Overall, this is an important problem and the paper description is good.

Critical strengths
  1. Relevant problem and one that has not been examined in the literature.
  2. The evaluation is strong.
  3. Presentation is good.
Critical weaknesses

There are multiple major concerns that I have:

  1. BDAP requires the offline dataset. There is no mention of training cost, which is not query free in the black box sense.
  2. It is not clear what is specific to flow policies in the method.
  3. The baselines are weak and there are multiple uninformative cells in the attack comparison.
  4. There do not seem to be any adaptive attacks in defense evaluation. Also, the gains seem quite small.
  5. The main results tables are quite dense and Figure 2 is hard to read.
  6. Can you report clean-observation performance of DMBP and DMBP+RCPF at rho=0.3 with no perturbation?
Questions for authors

Please refer to weaknesses above.

2 comments

We sincerely thank the reviewer for the thoughtful assessment of our work. We also appreciate the constructive feedback and suggestions.

W1. Offline access and preparation cost. BDAP achieves zero-target-query deployment with one generator forward pass, without victim queries or attack gradients. We report preparation separately. It uses offline data and frozen FQL surrogate/reference models. Each generator uses 200,000 updates, batch 256, and eight latents per state, totaling 409.6M state-latent evaluations per frozen model. Median process wall time across 68 fits at ϵ=0.05\epsilon=0.05 is 369.0 s for generator fitting, including loading/compilation/fitting/checkpointing. Surrogate pretraining uses 1M/0.5M updates on OGBench/D4RL. Reference pretraining uses 0.5M. Cross-budget reuse cuts generator updates by 66.7%, with surrogate/reference preparation shared (Appendix D.5, Table 26).

W2. Connection to flow policies. The flow-policy connection is built into our surrogate and behavior-reference construction. We use an FQL surrogate and a separately learned, critic-free behavior flow distilled into a one-step reference. The reference supplies a behavioral anchor independent of critic optimization. Shared latents align the action comparison. These one-step maps permit generator training without backpropagating through a flow ODE solver. The existing objective ablation finds Full BDAP better than both single-channel variants in 17/20 comparisons, tying in three. This directly supports combining value degradation and behavioral displacement. Our evaluation covers four one-step flow-policy families. RCF and its reconstruction bounds are architecture-independent.

W3. Stronger baselines and floor effects. We directly test a stronger value-based baseline, APGD-Q, minimizing BDAP's frozen FQL critic at the true state. The 10-step version changes only the original APGD objective/direction. 40 x 4 uses four 40-step restarts. All use ϵ=0.1\epsilon=0.1. Scores below are success percentages, except Adroit normalized return. Lower is stronger.

AttackOGBenchAntMazeAdroit
APGD7.261.535.6
min-Q9.649.638.9
BDAP3.138.734.4
APGD-Q (10)4.241.834.1
APGD-Q (40 x 4)2.836.032.6

APGD-Q (40 x 4) achieves the lowest aggregate scores using 160 gradient updates. BDAP generates each perturbation with one forward pass. On common tasks where every victim has at least 10% clean success, clean/APGD/min-Q/BDAP success is 72.1/23.0/16.2/10.4%. BDAP's advantage over the original baselines persists when excluding FQL and at a 20% threshold (Appendix D.4, Table 25).

W4. Full-pipeline adaptive attacks. RCF improves recovery under an attack optimized against the complete defense. We attack DMBP and DMBP+RCF separately by minimizing frozen victim-critic value at the true state using 40 steps, EOT=16, three restarts, identity BPDA through DMBP, exact RCF gradients, and actual defense forward passes. With ϵ=0.1\epsilon=0.1, ρ=0.3\rho=0.3, and 5 seeds x 50 episodes, the table reports pooled OGBench success over 200 combinations.

AttackDMBPDMBP+RCFGain (pp)
Filter-oblivious BDAP15.217.1+1.9
Full-pipeline adaptive11.412.8+1.4

Under adaptive optimization, the 1.4-point pooled gain is a 12.3% relative increase over DMBP. We report this white-box evaluation separately from zero-target-query transfer (Appendix D.2, Table 22).

W5. Main-table and Figure 2 readability. For the dense attack comparison, Table 25(a) provides a compact benchmark-level summary. Table 22(a,c) summarizes adaptive and clean-input defense results. For Figure 2, Appendix C.4, Table 21 provides the exact curve values and fixed 94-combination subset definition. Main-text Table 3 summarizes calibration controls. We organize the additional evidence into five multi-panel tables with explicit metrics and budgets. Original Tables 1-2 and Figure 2 are retained.

W6. Clean observations. As requested, we evaluate unperturbed inputs (ϵ=0\epsilon=0) with both defenses active at ρ=0.3\rho=0.3, all weights frozen, and 5 seeds x 50 episodes. RCF retains training scale 0.4.

BenchmarkRaw policyDMBPDMBP+RCF
OGBench success (%)31.530.931.1
AntMaze success (%)69.368.368.7
Adroit normalized return42.241.541.8

RCF improves all three DMBP aggregates. Full-pipeline clean scores are 0.4/0.6 percentage points below raw policies on OGBench/AntMaze and 0.4 return units below on Adroit (Appendix D.2, Table 22).

AuthorsOpenReview ↗

Your Comments are Highly Valuable to Us [Deadline in One Day]

Dear reviewer wgaT,

We have carefully considered all the valuable points and concerns you raised in your initial review through the initial rebuttal. As the deadline for discussion is approaching in only one day, we respectfully ask whether our response has addressed your concerns, or you still have questions to raise. Your comments are highly valuable to us. Thank you once again for your professional time and insightful effort in reviewing our submission.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.