- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes AID-RRL, a sampling-time acceleration of Adversarial Diffusion for Robust Reinforcement Learning. It replaces the full ancestral denoising chain with a shorter stochastic DDIM-style chain and recalibrates the adversarial coefficients according to the number of guided steps. It also investigates whether adversarial guidance meaningfully shifts generated trajectories toward low returns, attributing its observed ineffectiveness to a mismatch between the scaling of guidance displacement and sampling noise.
Evaluation includes sampler-level measurements across nine trained Hopper checkpoints, guided/unguided controls, and limited end-to-end MuJoCo experiments. The authors report approximately 6–12× faster trajectory generation and 2.7–3.1× lower wall-clock cost for fixed training budgets. However, the accelerated Hopper run fails the stated final-return criterion, and neither sampler produces the intended low-return shift in the reported percentile experiment.
The computational savings and diagnostic controls are useful. However, the central scaling argument is internally inconsistent as written, the accelerated risk-envelope guarantee is insufficiently justified, and the mixed single-seed training results do not establish reliable performance-preserving acceleration. The observed lack of a meaningful low-return shift could support a valuable diagnostic contribution, but its proposed structural explanation requires substantive reanalysis and broader validation.
Critical strengths
- The paper addresses a substantial computational bottleneck with a lightweight modification.
Trajectory generation accounts for approximately 80% of the measured training time, and the shorter sampler produces considerable computational savings without requiring an additional distillation or adaptation stage. The distinction between per-batch sampling speedup and fixed-budget training speedup is practically useful.
- The controlled examination of guidance effectiveness is potentially informative.
The unguided sampler-level control tests whether conventional fidelity metrics actually respond to adversarial guidance. The authors also disclose the failed final-return criterion and training collapse. Separating computational efficiency, rollout fidelity, guidance effectiveness, and training stability is valuable, and the observed lack of a meaningful low-return shift could support a useful diagnostic contribution if interpreted within its demonstrated scope.
Critical weaknesses
- The central theoretical diagnosis is inconsistent, and the risk-preservation argument is incomplete.
Appendices A.3 and C specify matrix-normal covariance factors , implying Under this convention, the noise RMS norm is , while the calibrated guidance displacement in Equation (5) is . Their ratio is therefore , independent of for fixed . The claimed -versus- separation consequently mixes incompatible covariance conventions. If the implementation instead injects noise with standard deviation , the guidance calibration must be reconciled with that actual sampling distribution. This undermines the principal explanation offered for “structural failure,” although it does not invalidate the reported empirical observation.
Independently, a small per-step displacement-to-noise ratio does not establish negligible change in the terminal distribution: systematic drift can accumulate, and subsequent denoising transitions can amplify or contract perturbations. No corresponding propagation or distributional argument is provided.
The risk guarantee also requires more than restoring . The cited AD-RRL Proposition 4.3 includes an additional boundary condition, and its simplified form requires sufficiently many steps; these conditions are not verified for the shortened chains. Budget allocation, likelihood-ratio feasibility, and attainment of the CVaR-optimal distribution are distinct claims. In particular, underutilizing an otherwise valid budget does not itself violate the risk envelope. [AD-RRL Proposition 4.3]
- The evidence does not yet establish reliable performance-preserving acceleration or a clear task-level benefit from the proposed recalibration.
The reported wall-clock savings establish faster execution of fixed training budgets, but evidence that these savings translate into reliable reductions in the cost of obtaining an equivalently performing final policy remains limited. The main accelerated Hopper run finishes at approximately 39 return versus 3288 for the ancestral baseline, and an accelerated Hopper collapse also occurs in the replication. Although the Walker2d comparison favors the implicit sampler, the mixed outcomes and limited seed coverage do not establish reliable performance preservation across tasks.
The assertion that the data rule out the sampler as a cause of the Hopper collapse is too strong. Screening logged metrics over the preceding 70k steps and observing lower imagination MSE cannot exclude sampler–training interactions. The controller correlations suggest a possible mechanism, but the paper acknowledges that no controller intervention was performed and that the initiating cause remains unresolved.
The algorithmic modification combines an established stochastic implicit sampler with a step-count adjustment of existing adversarial coefficients. This is a practical adaptation, but the experiments do not demonstrate a task-level benefit attributable to the recalibration, and compatible stochastic acceleration alternatives are not evaluated. These gaps leave the added value of the proposed modification insufficiently established beyond the computational savings from shortening the denoising chain.
Questions for authors
-
What is the exact distribution of the noise injected by the implementation? Please reconcile the matrix-normal factors in Appendices A.3/C with the noise norm used in Section 6.2, and recompute the guidance-to-noise ratio under a single consistent convention. Which parts of the structural-failure argument remain valid?
-
How are the boundary condition and sufficiently-large-step requirement of AD-RRL Proposition 4.3 satisfied for the accelerated kernels? Please distinguish the guarantee for the prescribed guided-step calibration from that for the as-run implementation, which uses the subsequence length instead. Does the result establish feasibility within the risk envelope, full allocation of the permitted bound, or attainment of the worst-tail distribution?
-
The paper appropriately notes that the sampler-level experiment does not establish end-to-end failure. What theoretical analysis or controlled evidence connects the local displacement calculation to the final generated return distribution and supports the proposed structural explanation? What justifies extending this explanation beyond the tested checkpoint, environment, and risk level, rather than restricting the conclusion to the observed absence of a meaningful low-return shift in that setting?
-
Beyond the reported single-seed comparisons, are additional matched-seed replications available that demonstrate lower computational cost at comparable final-policy performance? What controlled evidence, such as interventions on the policy-guidance controller, supports excluding sampler–training interactions as a cause of the Hopper collapse?
Thank you for the response. I keep my score, for the reasons below.
W2 / Q2
The response states that the two forms "are not the same distribution, and the single-factor drift ΣG is the consistent pairing with that noise." This confirms an error in Eq. (1), not an "explanatory convention." That Eq. (1) "faithfully transcribes the public AD-RRL implementation" does not make it correct.
Q2 asked for the Table 3 results with corrected guidance, which do not need retraining. Instead, the response argues that the push "stays a few percent of the noise." This alone does not show that the corrected guidance has no effect.
The rest of the response does not address my question. For example, it is unclear to me how the statement that "Table 3 evaluates progressively scaled guidance" is related to my question.
W1 / Q1
The response agrees that the method is "respaced ancestral sampling." The point that full-chain coefficients give "a budget of 1.46 instead of 10" is correct, but it does not change that the recalibration itself is a straightforward extension.
On Q1, "AD-RRL is a published CVaR-diffusion method" does not answer how widely AD-RRL is used. Also, "we do not claim it is the only route to robustness" does not answer whether the same robustness could be achieved with other methods that avoid this sampling cost.