acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Noise Adaptation for Diffusion Models

Abstract

Creating data from noise is generative modeling; its converse, creating noise from data, has been neglected yet could be unexpectedly non-trivial for diffusion models. Conventional diffusion models use a stepwise noising and denoising training mechanism, which can be viewed as a serial generation of training samples for different noise levels. Major diffusion models predefine the noising schedule, which treats the samples from the easy-to-learn and the hard-to-learn distributions equally. In this work, we investigate the adaptive noise schedule. We propose the Reinforcement Noise Adaptation (ReiNA), a framework that casts forward noise scheduling as a sequential decision-making problem. During the forward and backward learning process of the diffusion models, an adaptive policy observes the current learning state and adjusts the noise schedule, thereby reallocating learning capacity to the under-learned distributions. On the public datasets (CIFAR-10, CelebA-HQ, and LSUN Bedroom), ReiNA consistently improves FID, IS, Precision, and Recall compared with some SOTA methods. In addition, experiments on structured material simulation (StructSim) show the adaptive framework can be applied to sophisticated physical simulation tasks. The ablation studies and further analyses verify the contribution of each ReiNA component and reveal that ReiNA learns phase-aware and batch-adaptive noising strategies.

Primary area:
generative models
Keywords:
diffusion, adaptive noise

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jm3FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper examines diffusion training from a noise-sampling perspective, arguing that appropriate noise schedules vary across training stages. It proposes a PPO-based adaptive scheduler and demonstrates its effectiveness on standard benchmarks. Overall, the work offers a compelling perspective, solid theoretical grounding, and strong empirical results.

Critical strengths

Strengths

  • The paper offers an interesting perspective by treating forward noise scheduling as an adaptive training decision rather than a fixed design choice.
  • The proposed schedule is compact and interpretable: two parameters control its shape, while fixed endpoints and monotonicity preserve a valid noising process.
  • The PPO-based controller is paired with a structured reward design, and the ablation studies provide evidence for the contributions of its main components.
  • The evaluation covers multiple datasets and random seeds under matched backbones and sampling protocols. The reported early-stage results and training overhead also help assess the performance–cost trade-off.
  • The manuscript is well organized and clearly written.
Critical weaknesses

Weaknesses and questions

  • The PPO reward components appear largely heuristic and lack sufficient theoretical justification, which may limit their interpretability and transferability.
  • The evaluation is confined to U-Net backbones. Although this enables controlled comparisons, experiments with more recent transformer-based architectures would strengthen the generality of the results.
  • Most baselines use Gaussian noising–denoising diffusion. It would be valuable to test whether adaptive time scheduling provides similar benefits in flow-matching models.
  • The interpretation of Figure 4 needs clarification. The schedule is selected before the current batch is sampled, while the policy observes only lagged batch features. How, then, does the method systematically assign higher noise to harder current batches? The authors should clarify how batch difficulty is measured and what produces the reported correlation.
Questions for authors

See the weakness

1 comment

We thank the reviewer for the positive assessment of the perspective and the empirical basis, and for the question about the timing of the batch feature in Figure 4. To make the data flow more explicit, we have updated the caption to clarify that fkf_k is computed from the clean minibatch before the action and before noise is added. We have also added controls to separate the current-minibatch signal from adjacent batches. Figure 4 now clearly reflects a current-minibatch correlation.

Furthermore, the revised manuscript clarifies the system mechanics: a shared (m,n)(m,n) covers the entire minibatch, L0L_0 functions strictly as a held-out difficulty label, and the FID improvement over cosine is diven by the phase-dependent clipped schedule. The following sections detail the reward terms, backbone scope, flow-matching extension, and Figure 4 timing mechanics.

1. Reward terms and ablation

In response to the question on heuristic reward components, we clarify the theoretical grounding of the alignment term and report the sensitivity of the core rewards.

RalignR_{\mathrm{align}} is the inner product of the binned clipped weight and the residual EMA. With fixed endpoints, the unclipped objective is invariant to (m,n)(m,n); after clipping, the bin masses still depend on (m,n)(m,n), and this inner product is the part of the schedule that the reward can still adjust. RcycR_{\mathrm{cyc}} measures preview-denoiser recoverability. RtempR_{\mathrm{temp}} checks whether the residual decreases after a short horizon HH. RlearnR_{\mathrm{learn}} penalizes distance from a target residual of 0.4. RcostR_{\mathrm{cost}} and RsmoothR_{\mathrm{smooth}} regularize noise magnitude and action jumps.

At 200 epochs on CIFAR-10, setting λcyc\lambda_{\mathrm{cyc}} to 0 gives FID 8.898.89; setting λalign\lambda_{\mathrm{align}} to 0 gives 8.288.28; the default weights give 7.197.19. Among these, RcycR_{\mathrm{cyc}} and RalignR_{\mathrm{align}} show the larger FID sensitivity.

2. U-Net backbone

In response to the concern regarding the U-Net backbone, we clarify that our evaluation is intentionally scoped to this matched setting to ensure controlled comparisons.

All image entries in the main result table use the same U-Net backbone and 1000-step ancestral sampling (about 35.735.7M parameters at 32×3232\times 32; about 113113M at 256×256256\times 256).

3. Flow matching

In response to the question about generalization to flow-matching models, we report a separate 200-epoch rectified-flow experiment. We clarify the matched baseline to isolate the mechanism transfer.

NFDM is the neural-flow baseline trained on the same U-Net. The same sigmoid schedule maps uniform time to interpolation time, the reward keeps RcycR_{\mathrm{cyc}}, RalignR_{\mathrm{align}}, and RsmoothR_{\mathrm{smooth}}, and the loss is the velocity loss. Uniform time reaches FID 10.52±0.2210.52\pm 0.22; the time warp reaches 9.61±0.149.61\pm 0.14 (revised-appendix flow panel).

This experiment is designed to test the transferability of the time-warping mechanism under a strictly matched 200-epoch velocity-loss budget. The relevant baseline is therefore the uniform-time run, which our method improves upon by a clear margin. The full-training NFDM Gaussian FID of 7.657.65 is reported elsewhere in the main table as a separate baseline.

4. Figure 4 timing

In response to the question about the batch feature timing, we clarify the exact data flow before noise addition. We show that the reported correlation acts on the current minibatch without any leakage.

The clean minibatch is drawn first. The batch feature fkf_k is a frozen ResNet-18 feature of those clean images, pooled over the minibatch and computed before the policy samples (m,n)(m,n) and before noise is added. This feature does not depend on the denoiser loss of the current step.

Batch difficulty L0L_0 is the MSE of a separate frozen reference denoiser at a fixed noise level. It is used only as a held-out difficulty label in Figure 4, not as an input to the policy. The correlations with the current minibatch are R2=0.07R^2=0.07, 0.520.52, and 0.870.87 (early, mid, late). The previous minibatch gives 0.060.06, 0.310.31, and 0.470.47; adjacent minibatches give 0.210.21, 0.120.12, and 0.160.16. In the late stage, the current minibatch is clearly the stronger correlate, and the adjacent-batch correlation of 0.160.16 does not account for the late-stage R2R^2 of 0.870.87.

One slope and one inflection cover the entire minibatch. In 63.1%63.1\% of minibatches the within-batch spread of L0L_0 exceeds the easy--hard gap. The correlation reported above operates at the minibatch level.

Reviewer_BTytOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This submission proposes a novel reinforcement-learning-based noise schedule control method for improving diffusion models. Data distributions may have mixed easy-to-learn and hard-to-learn areas, and the proposed ReiNA aims to reallocate learning capacity on hard areas. ReiNA conducts reinforcement learning of the noise-level curvatures with reference to the dynamics, loss, and image features. In the experiments using CIFAR-10, CelebA, and LSUN, ReiNA improved the generation quality over DDPM and the existing adaptive-noise method MuLAN.

Critical strengths
  • Motivation of the work and formulation is clear and easy to follow.
  • Emergent difficulty-wise slope difference in the schedule is observe and interesting, supporting the valid design of ReiNA about difficulty-based adaptivity.
Critical weaknesses
  • Absent comparisons with RL-difufsion methods: I think that comparisons about "how to incorporate RL in diffusion models" is needed given the ReiNA's aim on general improvement in generation quality.
  • Experiments are a bit small-scale-ish in 2026. I am concerned on the RL stability on large datasets and text-to-image generation.
  • State-space design is heavily handcrafted including multiple features and hyperparameters. Although the ablative study is well done, design concept/principle insights, which is more important to sclae up the work to practitional datasets, might be weak.
Questions for authors

See weaknesses.

2 comments

We appreciate the reviewer's questions, which prompted us to clarify where ReiNA sits relative to existing RL-diffusion methods and to make the experimental scope more explicit. We have revised the manuscript to state that all image results use matched U-Net backbones with 1000-step ancestral sampling, and that text-to-image generation and transformer diffusion backbones fall outside the present scope. Below, we address each point, starting with the pipeline comparison.

1. Comparison with RL-diffusion methods

In response to the request for comparisons with RL-diffusion methods, we clarify the exact pipeline stage ReiNA operates on. The table below contrasts the control targets and inference costs.

ReiNA trains a PPO policy on the forward noise schedule σ(t)\sigma(t) and the induced clipped loss weight w(t)w(t) during diffusion training. At inference time, sampling uses the converged schedule and adds no extra cost.

MethodWhat is controlledWhenInference
ReiNAforward schedule σ(t)\sigma(t) and loss weightduring trainingconverged schedule
DDPO, DPOKreverse sampler or sample rewardafter trainingextra fine-tuning
SpeeDtimestep distribution p(t)p(t)during trainingunchanged sampler
MuLANper-pixel noise levelduring training, then frozenfrozen noise level
Grad-Reactnoise level via gradient-norm ruleduring trainingunchanged sampler

The frozen late clipped schedule (m=3.5m=3.5, n=0.83n=0.83) reaches FID 7.58±0.157.58\pm 0.15 at 200 epochs on CIFAR-10. The online controller reaches 7.19±0.137.19\pm 0.13, and cosine reaches 11.46±0.2111.46\pm 0.21. The frozen late schedule closes most of the gap from cosine. The online controller provides a further 0.390.39 FID improvement. We view PPO primarily as the mechanism that discovers and tracks the phase-dependent clipped schedule during training, rather than as the sole source of the full gap. DDPO and DPOK act on a different pipeline stage (reverse sampling or post-training fine-tuning), and we do not report their FIDs.

2. Datasets and backbone

In response to the concerns regarding experimental scale and RL stability on large datasets, we report the LSUN Bedroom scale and a local convergence probe. We also clarify the scope of our contribution in relation to large-scale foundation models.

All image results use matched U-Net backbones and 1000-step ancestral sampling. CIFAR-10 uses about 35.735.7M parameters at 32×3232\times 32. CelebA-HQ and LSUN Bedroom use about 113113M parameters at 256×256256\times 256. LSUN Bedroom trains on about 3.033.03M images for 50 epochs (about 2.372.37M steps), with ReiNA FID 7.91±0.187.91\pm 0.18 over three seeds. On a 5,000-step window of that run (revised-appendix Figure 16), the value loss is centered at 0.420.42 and the slope variance is 0.0400.040. This probe is designed specifically to measure local RL convergence behavior within the training window.

To isolate the forward-schedule control mechanism under strict variable matching, our study is intentionally scoped to U-Net settings. Our primary contribution is the algorithmic formulation of this dynamic schedule control, which we have thoroughly validated within this controlled experimental environment. Text-to-image generation and transformer diffusion backbones represent a fundamentally different system scale, where evaluating this mechanism is a matter of large-scale application rather than core methodological validation. We plan to extend this method to larger foundation models in future product applications, and we recognize this as a natural next step.

3. State design and reward terms

In response to the concern about handcrafted state design, we clarify the principle behind each term and provide the full ablation. We also show the transferability of the learned policy across datasets.

The state contains the progress k/Kmax⁡k/K_{\max}, recent loss descriptor ℓk\ell_k, previous gradient norm gkg_k, current minibatch feature fkf_k, and action history hkh_k. In the state-ablation table (revised appendix), the full state gives FID 6.51±0.086.51\pm 0.08 on CIFAR-10. Removing ℓk\ell_k raises FID to 7.087.08; removing fkf_k raises it to 6.916.91; keeping only progress raises it to 7.427.42. The loss descriptor and the batch feature contribute the most.

Each reward term has a clear definition and a corresponding ablation. RcycR_{\mathrm{cyc}} measures recoverability of the chosen corruption; removing it raises FID from 6.516.51 to 7.507.50 (+0.99). RalignR_{\mathrm{align}} is the inner product of the binned clipped weight and the residual EMA; removing it raises FID to 7.167.16 (+0.65). RtempR_{\mathrm{temp}} checks whether the residual on emphasized noise levels decreases after a short horizon; removing it raises FID to 7.317.31 (+0.80). The remaining three terms (RlearnR_{\mathrm{learn}}, RcostR_{\mathrm{cost}}, RsmoothR_{\mathrm{smooth}}) serve as regularizers and cause smaller degradations when removed (+0.43, +0.28, +0.07). The ablation ranks the terms by FID impact.

A 20-epoch CelebA-HQ probe gives FID 18.4±0.3718.4\pm 0.37 for a frozen CIFAR-10 policy, 17.6±0.4417.6\pm 0.44 for a policy trained from scratch on CelebA-HQ, and 22.1±0.5122.1\pm 0.51 for cosine. The frozen policy stays close to the from-scratch policy and below cosine.

Reviewer_DfnmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReiNA, which formulates the forward noise schedule of diffusion training as an MDP. A small PPO policy observes training statistics and outputs, at each step, the slope and inflection point (m,n)(m, n) of a sigmoid-shaped noise curve σ(t)\sigma(t) shared by the minibatch. It is rewarded by a combination of internal training signals. Experiments on CIFAR-10, CelebA-HQ, LSUN Bedroom and a structured CFD-configuration task report consistent improvements in FID/IS/Precision/Recall over fixed and learned schedules.

Critical strengths
  • Using RL to adapt the noise schedule during training is a novel and interesting perspective. The noise schedule is a critical factor in diffusion training.
  • The schedule parameterization is clean. Conditioning the network on log-SNR keeps schedule changes during training consistent.
  • Ablations on reward terms, state components, action dimensions and the PPO interval are extensive and mostly reported over multiple seeds.
  • The method adds no inference cost.
Critical weaknesses

Several aspects of the method are unclear to me, and I would like the authors to clarify them.

  • What the schedule actually changes. Both the noise curve σ(t)\sigma(t) and the loss weight w(t)=−V′(t)w(t) = -V'(t) are determined by (m,n)(m, n). With t∼U(0,1)t \sim U(0,1), a steeper curve near nn samples those noise levels less often but assigns them proportionally larger weights. With fixed endpoints and log-SNR conditioning, the two effects cancel, and the expected objective becomes independent of the schedule shape. The remaining effect is on the variance of the gradient estimator. Counterintuitively, the emphasized region near nn receives fewer samples with larger weights, which locally increases rather than reduces its variance. This is the opposite of the claimed mechanism of concentrating capacity on hard noise levels. Could the authors clarify where the improvement actually comes from, e.g., variance redistribution, or the weight clipping and normalization?
  • Granularity of adaptation. The paper claims the policy adapts the schedule to the training samples on the fly. However, one action is shared by the whole minibatch. Statistically, the difficulty of a randomly shuffled batch should be more stable at a larger batch size. How can the policy distinguish batches, or handle a batch that mixes samples of very different difficulty? Especially, when we scale the batch size?
  • Cost and stability of RL training. The RL controller adds a complex pipeline: a policy, a value network, a preview denoiser and a pretrained feature extractor. Please report the overall training cost compared with the baselines, including LSUN. Please also show whether the policy learning converges (e.g., return and value-loss curves) and how consistent the learned policy is across seeds. Since the RL environment, i.e., the diffusion model being trained, keeps changing during training, is the RL training stable?
Questions for authors

Please address points 1–3 above. I will consider raising my score depending on the authors' response.

2 comments

We are grateful that the reviewer identified the unclipped schedule identity so precisely. Building on this observation, we have revised the Method and Appendix to state the unclipped fixed-endpoint identity explicitly, to spell out the clipped and normalized weight w~(t)\tilde{w}(t) used in training, and to organize the clip on/off and batch-level controls around this mechanism. Overall, the gain originates from the clipped and normalized objective, rather than from the unclipped schedule shape or from variance reduction at the inflection.

1. What the noise schedule actually changes

In response to the question of what the schedule actually changes, we clarify that the operative training objective is the clipped and normalized one. The theoretical identity applies only to the unclipped objective, and we isolate the mechanism below.

With t∼U(0,1)t\sim U(0,1), fixed endpoints, and log-SNR conditioning, a steeper schedule near the inflection visits that noise level less often and assigns it a proportionally larger unclipped weight. The product is invariant, so the expected unclipped objective does not depend on mm or nn. The local Monte Carlo variance at that noise level rises. We build our analysis upon this exact characterization.

The implemented objective differs in one step. At each training iteration, the continuous-time weight w(t)=2σ′(t)/σ(t)3w(t)=2\sigma'(t)/\sigma(t)^3 is first computed from the current schedule σk(t)\sigma_k(t). This weight grows without bound as σ\sigma approaches zero, so training clips it to [0, wmax⁡][0,\,w_{\max}] with wmax⁡=100w_{\max}=100. The clipped weight is then divided by its minibatch average E[clip⁡(w)]\mathbb{E}[\operatorname{clip}(w)] to give the normalized effective weight w~(t)=clip⁡(w(t),0,wmax⁡)/E[clip⁡(w)]\tilde{w}(t)=\operatorname{clip}(w(t),0,w_{\max})/\mathbb{E}[\operatorname{clip}(w)]. This w~(t)\tilde{w}(t) replaces the raw w(t)w(t) in the gradient step that updates the denoiser. The cap removes different parts of the curve for different (m,n)(m,n), and the denominator E[clip⁡(w)]\mathbb{E}[\operatorname{clip}(w)] varies with the schedule shape (39.7 for the early anchor, 58.1 for the late anchor), so the training objective after this operation depends on (m,n)(m,n).

To clarify where the schedule actually matters, we break down the analysis into three parts.

Unclipped Early and Late stay close. At 200 epochs on CIFAR-10, removing the clip gives FID 8.76±0.198.76\pm 0.19 on the early schedule and 8.83±0.118.83\pm 0.11 on the late one. The gap is smaller than either standard deviation, consistent with the identity.

Clipping makes the objective schedule-dependent. The implemented loss clips at wmax⁡=100w_{\max}=100. This discards 99.996%99.996\% of the unclipped mass early and 99.994%99.994\% late. E[clip⁡(w)]\mathbb{E}[\operatorname{clip}(w)] is 39.739.7 early and 58.158.1 late, so the normalized objective depends on (m,n)(m,n). After normalization, the inflection bin carries mass 0.01770.0177 early and 0.01210.0121 late; the late schedule does not place more clipped mass on that bin. At 200 epochs under one x0x_0-weighted loss, late without the clip gives FID 8.83±0.118.83\pm 0.11, the same late anchors with the clip retained give 7.58±0.157.58\pm 0.15, and the online clipped run gives 7.19±0.137.19\pm 0.13. Raw w(t)w(t) on those late anchors gives 9.08±0.279.08\pm 0.27. The larger FID change is the clip gap of 1.251.25; the additional online gap is 0.390.39. Cosine at the same budget gives 11.46±0.2111.46\pm 0.21 and serves as a generation baseline. These results are in the revised appendix (Table 15).

The gain is not a variance reduction at the inflection. On the inflection bin, the unclipped variance proxy rises from 2.012.01 to 3.173.17, as the reviewer predicts. The equal-sample second moment of clip⁡(w)\operatorname{clip}(w) has a late-to-early ratio of 2.132.13, and dividing by E[clip⁡(w)]\mathbb{E}[\operatorname{clip}(w)] brings this ratio to about 11. The revised-appendix figure (Figure 10) shows this second moment of the clipped weight, not a back-propagated gradient variance.

Taken together, the unclipped objective is invariant and the local variance rises. The schedule enters the optimized objective through the clipped normalization constant E[clip⁡(w)]\mathbb{E}[\operatorname{clip}(w)], and the clip-on/off comparison (Table 15) confirms that this normalization step, rather than variance reduction, is where the FID gain originates.

2. Granularity of adaptation

In response to the question of how a shared action handles mixed difficulty, we clarify the empirical data flow and the batch-level design. We show that the policy reads the current minibatch and preserves the gain even at larger batch sizes.

The policy reads the current minibatch. Figure 4 of the revised manuscript correlates the action-induced noise level with the current minibatch's difficulty L0L_0. The fit gives R2=0.07R^2=0.07 early, 0.520.52 mid, and 0.870.87 late. The previous minibatch gives 0.060.06, 0.310.31, and 0.470.47; adjacent minibatches give 0.210.21, 0.120.12, and 0.160.16. In the late stage the current minibatch is the stronger correlate. These controls are in the revised appendix (Figure 11).

Larger batches preserve the gain over cosine. At 200 epochs on CIFAR-10, batch sizes 32, 128, and 512 give Var(L0)=0.152\mathrm{Var}(L_0)=0.152, 0.0380.038, and 0.0100.010, and Var(m)=0.48\mathrm{Var}(m)=0.48, 0.190.19, and 0.0410.041. ReiNA reaches FID 7.86±0.217.86\pm 0.21, 7.19±0.137.19\pm 0.13, and 7.64±0.167.64\pm 0.16. Cosine at the same budget reaches 11.62±0.2211.62\pm 0.22, 11.46±0.2111.46\pm 0.21, and 11.28±0.1711.28\pm 0.17. At batch 512 the slope varies less, but the FID stays well below cosine. The batch-size sweep is in the revised appendix (Table 16 and Figure 13).

Within-batch heterogeneity. In 63.1%63.1\% of minibatches the interquartile range of L0L_0 exceeds the gap between the easy and hard tertiles, with median ratio 1.421.42. All images in one minibatch receive the same slope and inflection. The policy optimizes a batch-level expected objective. MuLAN, the per-pixel baseline in the main result table, addresses finer granularity. The main FID gap relative to cosine comes from the phase-dependent clipped schedule (Table 15), not from within-batch difficulty separation.

Overall, the policy is minibatch-level by construction and tracks the current minibatch. It remains well below cosine even when batches are large; within-batch heterogeneity is addressed at batch level (revised appendix, Figure 12).

3. Training cost and stability

In response to the questions on training cost and RL stability, we provide the exact overhead, the step-wise stability curves across seeds, and the negative controls.

Overhead. On CIFAR-10, ReiNA takes 0.180.18 s/step (1.50×1.50\times DDPM) at 7.57.5 GB. On CelebA-HQ it takes 0.680.68 s/step (1.31×1.31\times DDPM) at 28.628.6 GB. On LSUN Bedroom (256 U-Net, batch 64), DDPM takes 0.550.55 s/step and 22.022.0 GB; ReiNA takes 0.720.72 s/step and 29.429.4 GB (1.31×1.31\times). Over about 2.372.37M steps this amounts to about 362 GPU-hours for DDPM and about 474 GPU-hours for ReiNA. The added parameters total roughly 1.31.3M (policy and value network ≈0.8\approx 0.8M, preview denoiser ≈0.5\approx 0.5M). Each environment step is one denoiser update, so the RL loop adds no separate training pass.

Stability across seeds. Step-wise PPO metrics are logged on CIFAR-10 with three seeds at a 200-epoch budget. Across three seeds, the return is 0.12±0.050.12\pm 0.05 at 10% of training and 0.61±0.040.61\pm 0.04 at the end. The value loss drops from 1.48±0.211.48\pm 0.21 to 0.28±0.070.28\pm 0.07, and the entropy falls from 1.08±0.101.08\pm 0.10 to 0.47±0.040.47\pm 0.04. The late slope converges to 3.5±0.33.5\pm 0.3 and the late inflection to 0.84±0.070.84\pm 0.07. The mean pairwise distance of (m,n)(m,n) falls from 0.520.52 early to 0.240.24 late. These curves are computed from step-wise logs recorded during the 200-epoch re-run and averaged over three seeds. They are in the revised appendix (Figure 14).

Negative controls. On 50-epoch CIFAR-10 controls, the boomerang reward reaches return 0.150.15--0.400.40 with late-slope standard deviation 0.120.12. A time-shuffled reward has mean return 0.010.01 and late-slope standard deviation 0.900.90. A constant reward has late-slope mean 0.440.44. These controls (revised appendix, Figure 15) test whether the policy converges and whether (m,n)(m,n) is consistent at this budget.

The cost is 1.31.3--1.5×1.5\times DDPM with about 1.31.3M added parameters. Across three seeds the policy converges consistently, and the negative controls show that removing or shuffling the boomerang reward prevents this convergence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.