acceptodds
Under review as a conference paper at ICLR 2027

RACE: Reward-Guided Adaptive Candidate Exploration for Sample-Efficient Diffusion Fine-Tuning

Abstract

Reward-guided diffusion fine-tuning requires informative candidates under a limited scoring budget. Independent sampling provides coverage, while a shared scalar perturbation cannot adapt exploration across latent coordinates. We introduce Reward-Guided Adaptive Candidate Exploration (RACE), which learns a state-conditioned, entry-wise perturbation map to construct additional candidates around selected generated trajectories while retaining independent base samples. When paired training images are available, RACE uses deterministic inversion to provide reference-derived exploration centers within the same candidate budget. Both variants jointly optimize the generator and perturbation predictor through a single reward-weighted objective. Our analysis derives sufficient conditions for improved expected candidate reward, separating parent-selection benefits and perturbation effects and quantifying the roles of reference quality and inversion error. Experiments cover natural-image generation, including a multi-topic COCO subset, remote sensing, and medical imaging. Both variants achieve higher mean scores than DiffusionNFT across all five COCO evaluators. On brain MRI, RACE achieves a joint score of , compared with for DiffusionNFT. These findings support learning candidate construction as a means of improving diffusion fine-tuning under matched scoring budgets.

Primary area:
generative models
Keywords:
diffusion models, reward-guided fine-tuning, sample-efficient adaptation, candidate exploration, image generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2S4zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RACE (Reward-Guided Adaptive Candidate Exploration), aiming to address candidate construction in reward-weighted diffusion fine-tuning, building on DiffusionNFT’s reward-weighted forward-process objective. In the algorithm, a frozen collection copy (v_old) generates B independent base samples per prompt; the top-P are selected as parents. A lightweight state-conditioned predictor Γ_φ(x_t, t) replaces DiffusionNFT’s single global scalar perturbation coefficient γ with an entry-wise map, used to construct additional “child” candidates around selected parents by mixing their implied noise direction with fresh Gaussian noise at varying, learned, per-coordinate strength.

RACE claims a learnable, state-conditioned, and entry-wise perturbation predictor replacing DiffusionNFT’s scalar perturbation for constructing candidates, jointly trained this light weight predictor with the generator. Under the same shared objective, RACE+’s reference-guided extension using inverted paired images as added exploration centers. These are validated via matched-budget comparisons against DiffusionNFT across natural-image, medical-imaging, and remote-sensing benchmarks, showing higher mean scores.

Critical strengths

1. The idea is interesting. Learning a small predictor to adaptively shape candidate exploration (rather than using a fixed scalar perturbation) is an interesting direction — letting the model decide the perturbation based on the current state is slightly more flexible than a hand-tuned global constant.

2. Clean integration with existing objective. Both the learned perturbation and the reference-guided candidates are folded into DiffusionNFT’s existing reward-weighted loss without introducing a separate objective, keeping the training pipeline relatively simple.

3. Broad empirical coverage. Experiments span natural images, medical imaging, and remote sensing, showing the method isn’t narrowly tuned to one domain.

Critical weaknesses

1. Marginal technical contribution compared with DiffusionNFT:

RACE’s core change relative to DiffusionNFT is replacing a single fixed scalar perturbation coefficient γ with a small online-learned, state-conditioned function Γ_φ(x_t, t) that predicts a per-coordinate version of the same scalar — the construction mechanics, the shared reward-weighted regression loss (Eq. 15), and the overall collection procedure are otherwise unchanged from DiffusionNFT. RACE+’s further addition is reusing inverted reference images as alternative exploration centers for the same predictor, which is a data-source substitution, not a new mechanism. Neither change touches the reward weighting scheme or the generator. This reads as an incremental refinement on top of an existing method, and the paper does not establish that the entry-wise structure itself is doing meaningful work beyond what a well-tuned scalar could achieve.

2. Added training complexity is not justified by the empirical gains, and RACE+ shows worse stability:

Jointly optimizing (θ, φ) introduces complexity the paper doesn’t adequately analyze: the construction relies on an engineered stop-gradient boundary, and the authors’ own derivation shows φ's gradient loses reward-dependence entirely whenever v_θ and v_old coincide. Despite this added machinery, the measured benefit is marginal — in Tables 2 and 4, even RACE+, which carries the most additional construction machinery, shows only modest mean improvement over DiffusionNFT, and on top of that its variance is larger. The added complexity of jointly learning φ alongside θ is not shown to buy a reliable improvement over the simpler baseline. (Even though it's slightly more flexible)

3. (Minor) The presentation of the paper is not clear:

Though the problem itself and the methods are interesting, the presentation of the paper is not very clear, and much of the setup is left under-specified, apparently assuming the reader already has DiffusionNFT’s formulation in hand — e.g. the problem formulation is not clearly stated.

Questions for authors

1. On the degenerate joint-training regime (relates to Weakness 2): Appendix A.4 (Eq. 35) shows that φ's gradient loses reward-dependence entirely whenever v_θ and v_old coincide. Given that the collection copy is refreshed via EMA (Table 9), how often does this degenerate regime actually occur during training?

2. On computational cost: The paper acknowledges that “matched scored-candidate budgets” do not imply “matched compute” (inversion and continuation add extra overhead). Could the authors provide actual computation cost comparisons between RACE/RACE+ and DiffusionNFT?

3. On statistical reliability of the reported gains: Across most tables, RACE’s improvement over DiffusionNFT appears quite small. Given that only 3 training seeds are used throughout, is it possible that these gains are not statistically distinguishable from noise?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2S4z (Part 1/2)

Thank you for the careful assessment of our contribution and for raising questions about entry-wise construction, joint optimization, statistical reliability, and presentation. We have clarified the problem formulation and the relation to DiffusionNFT, and added control experiments and resource measurements to support the responses below; manuscript revisions are marked in blue. W1-W3 are addressed here; Q1-Q3 follow in Part 2.


W1. RACE changes candidate construction while retaining DiffusionNFT's reward-weighted loss form; the new controls test whether entry-wise structure adds value. DiffusionNFT collects and scores clean images before forward-process training. RACE adds children around selected intermediate states through a state-conditioned, entry-wise map; RACE+ additionally uses inverted training references within the same scoring budget. The shared-coefficient construction is a restricted ablation of our child map, not the entire DiffusionNFT baseline.

Table S1. Matched-initialization reference-aware AID controls; mean ± sample SD.

SeedFixedLearned sharedEntry-wise
10.2860.2930.303
20.2880.2980.305
30.2810.2900.298
Mean ± SD0.285 ± 0.0040.294 ± 0.0040.302 ± 0.004

All three constructions initialize at γ0=1\gamma_0=1 and use the same 120-image test bank. The mean entry-wise-minus-fixed difference is approximately +0.017; entry-wise-minus-learned-shared is +0.0083, positive in every seed (for entry-wise versus learned shared: reported two-sided paired tt-test, p=0.011p=0.011). Sec. 4.6 presents this controlled comparison, which supports an additional benefit of entry-wise structure in the tested setting.


W2. Measured cost and matched-initialization controls make the benefit–complexity tradeoff explicit. The predictor has 19,024 parameters (approximately 0.10% of trainable LoRA parameters) and is unused at inference. Appendix C.1 reports GPUh and memory. Table S1 shows similar seed SDs in the three reference-aware arms. The fixed-context diagnostic additionally reports mean child-reward gains of +0.0051 over frozen initialization and +0.0063 over the tuned-fixed control (Sec. 4.6; Appendix C.4). These observations concern the measured configurations; they do not establish lower variance across every original benchmark. Part 2, Q1 clarifies the gradient-path condition; Q2 gives measured costs, and Q3 adds direct comparisons against DiffusionNFT.


W3. Sec. 3 now starts with a self-contained problem statement. The blue paragraph defines the pretrained generator, conditions, reward evaluator, scored-candidate budget, optional training references, and held-out reward objective. It states that test generation starts from fresh noise and uses the condition alone. Sec. 4.6 summarizes the new coefficient and candidate-reward controls; Appendices C.1-C.4 give the resources, gradient scope, inversion procedure, and checkpoint measurements.

AuthorsOpenReview ↗

Response to Reviewer 2S4z (Part 2/2)

Thank you again for the detailed questions; we continue below with Q1-Q3 on gradient behavior, computation, and statistical reliability.


Q1. The four additional probes show a nonzero reward-dependent predictor-gradient component at the sampled updates. These probes use the detached-old-output graph at updates 1, 10, 100, and 600. The equal-field cancellation argument in Sec. 3.4 and Appendix A.4 assumes that both fields coincide as functions with identical input-gradient paths. Detaching the old output changes that path, so equal forward values alone do not establish reward independence. Moreover, reward independence would not imply a zero total gradient, since a residual-loss gradient may remain. Appendix C.2 specifies the implementation scope. These four probes do not measure the frequency of cancellation over the full training run.


Q2. The recorded MRI jobs show lower mean whole-job GPUh with RACE and RACE+, and the additional validation comparison shows higher scores at common GPUh budgets. The resource measurements cover three jobs per method, each with 600 updates and 4,800 scored training candidates:

Table S2. Recorded MRI resources.

MethodA100 GPUhPeak GB
DiffusionNFT5.899 ± 0.38144.2
RACE5.102 ± 0.16544.5
RACE+4.871 ± 0.52644.6

GPUh are measured whole-job allocated A100 time, including optimization (mean ± sample SD); memory is the jobstats-sampled peak. Appendix C.1 reports these measurements, and Appendix C.2 specifies their implementation scope.

Table S3. Additional MRI validation joint score versus reported cumulative A100 GPUh.

Method1 h2 h3 h4 h5 h
DiffusionNFT0.7450.7620.7750.7840.790
RACE0.7580.7850.8050.8190.827
RACE+0.7620.7920.8130.8270.835

Both variants attain higher validation scores at the displayed compute points. Table S3 is an additional validation diagnostic; Table S2 reports the recorded resource cohort. The validation scores are separate from the final held-out test scores.


Q3. The additional AID comparison shows gains over DiffusionNFT in all three seeds, with a reported paired-test result of p=0.011p=0.011 for RACE versus DiffusionNFT. All methods below use the same 120-image test bank:

Table S4. Additional AID comparison; mean ± sample SD across three seeds.

MethodAID scoreMean paired gain over NFTSeeds above NFT
DiffusionNFT0.268 ± 0.005——
RACE0.279 ± 0.003+0.01103/3
Reference + entry-wise0.302 ± 0.004+0.03373/3

The RACE-minus-DiffusionNFT differences are +0.011, +0.009, and +0.013; the reported test is a two-sided paired tt-test over the three seeds. Reference + entry-wise uses the matched-initialization configuration in Table S1 and also improves over DiffusionNFT in each seed. These additional AID results address seed variability in this setting. They do not establish statistical significance for every original benchmark, whose means and SDs remain descriptive. Appendix D.4 defines the aggregation units.

Dear Reviewer 2S4z, could you check if the responses from the authors have addressed your concerns?

Reviewer_cSEMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RACE, a framework for improving reward-guided diffusion fine-tuning through adaptive candidate exploration. Rather than generating all training candidates independently, RACE selects high-reward parent trajectories and constructs additional candidates using a learned, state-conditioned, coordinate-wise perturbation map. RACE+ further extends this framework by introducing reference-derived exploration centers obtained through deterministic inversion of paired training images.

Both methods jointly optimize the generator and perturbation predictor using a shared DiffusionNFT-style reward-weighted objective, without requiring an additional reconstruction loss. The paper also provides theoretical analyses of perturbation geometry and sufficient conditions for improving expected candidate rewards. Experiments on natural-image, medical-image, and remote-sensing generation demonstrate improvements over DiffusionNFT under matched reward-evaluation budgets.

Critical strengths

1. A promising approach to improving reward-guided candidate exploration. The paper addresses an important limitation of reward-guided diffusion fine-tuning: independently generated candidates may not efficiently exploit promising regions discovered during sampling. Learning a state-conditioned, coordinate-wise perturbation map provides a flexible way to construct additional candidates around high-reward trajectories. This approach is practically appealing because it improves candidate construction without introducing a separate reward optimization objective or substantial additional trainable parameters.

2. Encouraging results across multiple domains with useful theoretical analysis. The method is evaluated on natural-image, medical-image, and remote-sensing generation, reporting improvements over DiffusionNFT under matched scoring budgets. The MRI component ablation provides evidence for the successive benefits of local perturbation, adaptive exploration, and reference guidance. The theoretical analysis also carefully characterizes the conditions for reward improvement and acknowledges the limitations of the surrogate objective, which strengthens the methodological presentation.

Critical weaknesses

1. Insufficient evidence that the learned perturbation predictor actually improves reward-guided exploration.

The central contribution of RACE is learning an adaptive perturbation policy, but the experiments primarily evaluate the final fine-tuned generator rather than the exploration behavior of the learned predictor itself.

This distinction is important because the predictor is optimized through a surrogate regression objective rather than directly maximizing child rewards. As acknowledged in Section 3.4 and Appendix A.5, the predictor gradient can be reward-independent when the current and old generator fields coincide, and minimizing the surrogate does not guarantee reward improvement. Theorem 3 establishes improvement only under an additional gradient-alignment condition.

Although the MRI scalar-perturbation ablation provides supporting evidence, it remains unclear whether the improvements arise specifically from learning a reward-beneficial exploration policy or from parent selection and local sampling. Fixed-context reward measurements across predictor checkpoints and comparisons with a frozen predictor are needed to validate the central learning mechanism.

2. Limited isolation of reference-guided exploration and insufficient computational evaluation.

RACE+ incorporates paired training images through deterministic inversion, providing additional structural information unavailable to the generated-only exploration in RACE. However, the experiments do not sufficiently distinguish the benefit of the proposed adaptive reference construction from the benefit of accessing training references.

A comparison against simpler reference-aware alternatives, such as deterministic inversion with a fixed perturbation coefficient, would help establish the necessity of the learned adaptive construction. Furthermore, the different perturbation initializations used for RACE and RACE+ in CT complicate the attribution of their performance differences.

The computational comparison is also incomplete. While the paper correctly acknowledges that matched scoring budgets do not imply matched computation, it does not provide end-to-end training costs that account for reference inversion, child continuation, and optimization overhead. Controlled reference-aware baselines and total-compute measurements are necessary to substantiate the specific contribution and practical efficiency of RACE+.

Questions for authors

1. Does the learned perturbation predictor actually improve candidate rewards during training?

Appendix C.4 describes fixed-context diagnostics, but the corresponding empirical results are not presented. Could the authors report the mean child reward, parent-relative improvement, and fraction of children outperforming their parents across predictor checkpoints? Comparisons against a frozen predictor and a tuned scalar perturbation, using identical parent contexts and continuation models, would help establish whether learning the perturbation map provides benefits beyond local sampling itself.

2. How significant is the mismatch between the construction target and the reward-scored endpoint?

In Eq. (6), the child's flow-matching target is constructed using the parent endpoint, whereas its reward is evaluated on the actual generated child endpoint. Although Appendix A.3 provides a theoretical bound, the empirical magnitude of this mismatch remains unclear. Could the authors report the discrepancy dk=∣∣x0−Ft(xt,k,c)∣∣2/td_k = ||x_0 - F_t(x_{t,k},c)||_2 / t for both generated and reference children, and examine how it affects the reliability of the predictor's learning signal?

3. What is the actual computational efficiency of RACE and RACE+?

While the number of reward evaluations is matched, the methods have different computational requirements. Could the authors report the reference inversion solver-step count, total generator evaluations, and end-to-end training GPU hours, including child continuation and optimization overhead? A performance-versus-total-compute comparison would help determine whether the proposed methods provide efficiency improvements beyond reducing the number of reward evaluations required to achieve a given performance.

4 comments
AuthorsOpenReview ↗

Response to Reviewer cSEM (Part 1/2)

Thank you for the detailed review and for distinguishing the quality of the final generator from the effectiveness of learned candidate construction. We respond below with fixed-context checkpoint comparisons, reference-aware controls, and computational measurements, with the corresponding manuscript revisions marked in blue. W1, W2, and Q1 are addressed here; Q2-Q3 follow in Part 2.


W1. The fixed-context evaluation shows a learned candidate-reward gain over both frozen initialization and a tuned fixed coefficient. Across 300 AID contexts, the pooled final-minus-initial gain is +0.0051 (95% CI [0.0031, 0.0072]); final-minus-tuned is +0.0063 ([0.0035, 0.0092]). Q1 gives the checkpoint measurements. Sec. 4.6 and Appendix C.4 report this empirical learning benefit under fixed contexts. Theorem 3 states sufficient conditions for reward improvement; the measurements do not establish that its gradient-alignment condition holds for every update.


W2. With reference access and coefficient initialization matched, entry-wise construction improves over the requested fixed-coefficient control. The additional AID comparison uses a common 120-image test bank and three seeds; all three reference-aware constructions initialize at γ0=1\gamma_0=1:

Table C1. Reference-aware AID controls; mean ± sample SD.

ConstructionAID score
Fixed coefficient0.285 ± 0.004
Learned shared coefficient0.294 ± 0.004
Entry-wise coefficients0.302 ± 0.004

Entry-wise construction yields mean paired score improvements of approximately 0.017 over the fixed control and 0.0083 over the learned-shared control, with positive differences in all three seeds. The learned-shared control predicts one state-dependent coefficient and broadcasts it across coordinates. Sec. 4.6 reports this comparison. The original CT comparison uses different coefficient initializations and therefore does not by itself isolate the effect of reference guidance. The matched-initialization AID controls above test adaptive construction while keeping reference access fixed. Appendix D.2 documents the CT settings; Q3 in Part 2 reports computational costs.


Q1. Yes, the final predictor improves mean child reward over both frozen and tuned-fixed controls in the fixed-context evaluation. The continuation model and context bank are held fixed within this diagnostic:

Table C2. Fixed-context candidate measurements; 300 AID contexts.

SourceMapRewardGainWin (%)
GeneratedTuned fixed0.35450.005252.0
GeneratedInitial, frozen0.35640.007160.0
GeneratedUpdate 120.35710.007860.8
GeneratedUpdate 3000.35980.010563.9
GeneratedUpdate 6000.36120.011966.1
ReferenceTuned fixed0.45590.041658.3
ReferenceInitial, frozen0.45640.042258.3
ReferenceUpdate 120.45740.043259.4
ReferenceUpdate 3000.46010.045962.5
ReferenceUpdate 6000.46180.047665.0

Table C3. Pooled paired reward differences; 95% CI.

ComparisonDifference95% CI
Final - Parent+0.0298[0.0210, 0.0390]
Final - Initial+0.0051[0.0031, 0.0072]
Final - Tuned fixed+0.0063[0.0035, 0.0092]

Initial freezes the initialized map at approximately 1.4; tuned fixed uses a single selected coefficient. This diagnostic is separate from the γ0=1\gamma_0=1 training controls in W2. The final pooled child reward is 0.4115. The positive final-minus-initial and final-minus-tuned intervals distinguish an additional learning benefit from a parent-relative local-sampling benefit. Appendix C.4 gives the checkpoint table; Sec. 4.6 presents its initial-to-final summary.

AuthorsOpenReview ↗

Response to Reviewer cSEM (Part 2/2)

Thank you again for these questions; we continue below with Q2 on endpoint mismatch and Q3 on computational cost.


Q2. At the final predictor checkpoints, generated and reference children have similar mean coordinate-normalized discrepancies of approximately 0.56. The diagnostic uses 30 contexts, three predictor seeds, and two children per context and seed (180 children per source). We measure d=∥xˉ0−Ft(xt,k,c)∥2/td=\lVert\bar{x}_0-F_t(x_{t,k},c)\rVert_2/t and report the means of dd and its per-child RMS normalization d/Dd/\sqrt{D}, with D=65,536D=65{,}536. Here tt denotes the actual flow level: 0.75 for generated children and 0.50 for reference children.

Table C4. Endpoint discrepancy at final checkpoints and context-level correlations.

SourceMean dMean RMSr(d, reward)r(d, gain)
Generated144.110.5629−0.446+0.065
Reference144.590.5648−0.299−0.035

Correlations use 30 context means after averaging predictor seeds and siblings. Discrepancy is negatively associated with absolute reward, while its linear association with parent-relative gain is weak. These associations do not isolate the causal effect of mismatch on predictor gradients. Separately, the 300-context experiment in Part 1 (Q1) measures positive final-minus-initial and final-minus-tuned reward gains of 0.0051 and 0.0063, respectively.


Q3. In the recorded MRI jobs, RACE and RACE+ use lower mean whole-job GPUh than DiffusionNFT at the same scored-candidate budget. The resource measurements cover three jobs per method, each with 600 updates and 4,800 scored training candidates:

Table C5. MRI resources and reconstructed collection/diagnostic sampling NFE.

MethodA100 GPUhPeak GBSampling NFE
DiffusionNFT5.899 ± 0.38144.2134,400
RACE5.102 ± 0.16544.5100,800
RACE+4.871 ± 0.52644.6105,000

GPUh are measured whole-job allocated A100 time, including optimization (mean ± sample SD); memory is the jobstats-sampled peak. Sampling NFE sums solver steps over samples for collection and diagnostics, including child continuation and reference inversion, before CFG-branch expansion. For example, DiffusionNFT uses 4,800 × 28 = 134,400 sampling evaluations. These counts are reconstructed from code and run configurations, not runtime measurements of batched calls. Optimization and validation forwards are excluded, so this is not a complete generator-evaluation total.

At the recorded MRI API setting 0.25 on the 28-step schedule, each reference child uses 7 inversion and 7 continuation steps. RACE+ has 8,400 training-candidate and 2,100 diagnostic inversion evaluations; their sum of 10,500 is already included in Table C5.

Table C6. Additional MRI validation joint score versus reported cumulative A100 GPUh.

Method1 h2 h3 h4 h5 h
DiffusionNFT0.7450.7620.7750.7840.790
RACE0.7580.7850.8050.8190.827
RACE+0.7620.7920.8130.8270.835

At each reported GPUh budget, both variants achieve higher MRI validation scores than DiffusionNFT. Table C6 is an additional validation diagnostic; Table C5 reports the recorded resource cohort. The validation scores are separate from the final held-out test scores. Appendix C.1 reports the resource measurements, and Appendix C.2 specifies their implementation scope.

Reviewer_cSEMOpenReview ↗

Official Comment by Reviewer cSEM

Thank you for the substantial additional experiments. The fixed-context checkpoint comparisons (Table 12) provide useful evidence that the learned predictor improves child rewards over frozen and tuned-scalar controls, substantially addressing W1 and Q1. The matched-initialization reference-aware comparisons (Table 6) also strengthen the evidence for entry-wise construction, while the endpoint-discrepancy analysis and GPU-hour measurements provide valuable responses to Q2 and Q3.

My remaining concern is the connection between the theoretical learning mechanism and the implementation used in the reported experiments. I appreciate the clarification in Appendix C.2 and the response to reviewer 2S4z regarding gradient paths and equal-field cancellation. However, Appendix C.2 states that the inspected MRI implementation detaches old-field outputs and caches reference states and targets, so reference losses do not directly update the predictor. This differs from the analyzed gradient formulation in Section 3.4 and Appendix A.4, particularly in how reference-derived children contribute to predictor updates. I would appreciate clarification on two points:

  1. Experimental implementation: Which gradient formulation was used for each reported benchmark, and what motivated the detached and cached implementation in MRI?
  2. Learning mechanism: Given these implementation differences, how does the theoretical predictor-gradient analysis relate to the observed reward improvements? In particular, how should the contribution of reference-guided exploration in RACE+ be interpreted when reference losses do not directly update the predictor?

The additional experiments strengthen the empirical case, but clarifying this theory–implementation connection would help establish which aspects of the proposed learning mechanism are supported by the reported results.

AuthorsOpenReview ↗

Response to Reviewer cSEM: Gradient Implementation and Reference Guidance

Thank you for the follow-up and for your positive assessment of the additional experiments. We clarify the implementation used in each set of results and the role of reference-guided exploration.


Q1. Experimental implementation and motivation.

Our main benchmarks use the differentiable formulation in Sec. 3.4 and Appendix A.4. The old generator's parameters are frozen, but its input Jacobian is retained. Generated and reference child states and targets are rebuilt with the live predictor during optimization. Collection, continuation, and reward evaluation remain outside the backward graph.

ExperimentOld input gradientGenerated childrenReference childrenDirect predictor gradient
Architecture & Interior (Table 1)RetainedLive-rebuiltLive-rebuiltGenerated + reference
COCO (Table 1)RetainedLive-rebuiltLive-rebuiltGenerated + reference
MRI (Tables 2, 5)RetainedLive-rebuiltLive-rebuiltGenerated + reference
CT (Table 3)RetainedLive-rebuiltLive-rebuiltGenerated + reference
AID (Table 4)RetainedLive-rebuiltLive-rebuiltGenerated + reference
AID learned controls (Table 6a)RetainedLive-rebuiltLive-rebuiltGenerated + reference
AID fixed control (Table 6a)RetainedFixed coefficientFixed coefficientN/A
AID checkpoint evaluation (Tables 6b, 12)Retained in source trainingLive in source trainingLive in source trainingGenerated + reference in training; none in evaluation
MRI resource cohort (Apps. C.1–C.2)DetachedLive-rebuilt from scored-image encodingsCachedGenerated only

Reference columns apply to RACE+; RACE has only generated children. DiffusionNFT and the fixed-coefficient control have no trainable perturbation predictor.

The MRI resource cohort uses a different training graph. Detaching the old prediction avoids the backward pass through the old field's input. Reference states and targets are saved at collection so that training uses the tuples associated with the scored continuations. Generated training tuples are reconstructed with the live predictor, scored-image encodings, and fresh training noise. Reference children still contribute to generator updates. The GPU-hour and memory measurements in Appendix C.1 correspond to this MRI resource cohort. They do not quantify the training cost of the fully differentiable main-table MRI runs or isolate the cost of retaining the old-field input Jacobian.

The separate detached-old gradient probes examine this gradient configuration at selected checkpoints. The equal-field cancellation in Eq. (35) requires identical input-gradient paths; it is not automatic when the old output is detached. These probes characterize the local gradient behavior, not its frequency over an entire training run.


Q2. Learning mechanism and reference-guided exploration.

Reference images provide additional exploration centers. During collection, the shared predictor constructs children around inverted reference states. These children are continued, scored, and included in the shared reward-weighted training pool.

In the cached-reference implementation, reference children contribute to generator updates, while generated children provide direct predictor gradients. The collection predictor is an EMA copy of the learned predictor, so updates from generated children also influence reference exploration in subsequent rounds. Reference scores further affect the normalized training weights of generated children; these weights remain fixed during optimization.

Theorem 2 characterizes the candidate-pool benefit at fixed model snapshots. Holding the distribution of the remaining candidates fixed, replacing generated children with reference children changes the expected pool mean by

JRACE+−JRACE=KrK(Jr−Jg).J_{\mathrm{RACE}^{+}}-J_{\mathrm{RACE}}=\frac{K_r}{K}(J_r-J_g).

When reference children have higher mean reward than the generated children they replace, the candidate-pool mean increases. This benefit follows from candidate construction and does not require direct reference-to-predictor gradients.

Appendix A.4 derives the predictor gradient for the fully differentiable formulation. With cached reference tuples, the reference-to-predictor term is absent, while generated children continue to update the predictor. Theorem 3 gives sufficient conditions for expected child-reward improvement, including positive update–reward alignment and a suitable step size.

Table 12 reports an increase in reference-child reward from 0.4564 at initialization to 0.4618 at the final predictor checkpoint, with contexts and continuation fixed. This provides empirical evidence that the final predictor constructs higher-reward candidates in the evaluated reference contexts, complementing the gradient-path analysis.

Reviewer_ztf1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Reward-Guided Adaptive Candidate Exploration (RACE), a method designed to improve reward-guided diffusion fine-tuning under limited scoring budgets. RACE learns to dynamically generate higher-quality candidate samples. It achieves this by applying state-conditioned, adaptive perturbations around high-reward trajectories, or by using deterministic inversion to create exploration centers when reference images are available. By jointly optimizing the diffusion generator and the perturbation predictor, RACE significantly outperforms existing baselines (like DiffusionNFT) across diverse domains, including natural images (COCO), remote sensing, and medical imaging.

Critical strengths

Adaptive, Smart Exploration: Instead of applying uniform noise, it learns a state-conditioned, coordinate-specific perturbation map. This allows the model to explore latent spaces intelligently around high-quality trajectories.

Dual-Mode Flexibility: The framework is highly adaptable, functioning both with paired training data (by utilizing deterministic inversion to find exploration centers) and without paired data (by perturbing generated trajectories).

Strong Empirical Performance: It consistently outperforms strong baselines (like DiffusionNFT) across highly varied and complex domains, including natural images (COCO), remote sensing, and medical imaging (MRI).

Critical weaknesses

Increased Training Complexity: RACE requires jointly optimizing an additional component (the perturbation predictor) alongside the main diffusion generator, which increases architectural complexity and memory overhead during fine-tuning.

Sensitivity to Inversion Errors: In the paired-data variant, the method relies heavily on deterministic inversion to create exploration centers. The paper notes that performance is tied to "reference quality and inversion error," meaning poor inversions will directly degrade the exploration quality.

Questions for authors

na

1 comment
AuthorsOpenReview ↗

Response to Reviewer ztf1

Thank you for your positive assessment of the framework's flexibility and cross-domain results, and for highlighting training complexity and inversion sensitivity. We have added resource measurements and an inversion-level diagnostic to address these concerns, with the corresponding manuscript changes marked in blue.


W1. The predictor is small, and the MRI measurements quantify whole-job training costs. It has 19,024 parameters, approximately 0.10% of the 18,776,064 trainable LoRA parameters. Appendix C.1 reports:

Table Z1. Recorded MRI resources; three jobs per method.

MethodA100 GPUhPeak GB
DiffusionNFT5.899 ± 0.38144.2
RACE5.102 ± 0.16544.5
RACE+4.871 ± 0.52644.6

Each job uses 600 updates and 4,800 scored training candidates. GPUh are whole-job allocated time, including optimization (mean ± sample SD); memory is the jobstats-sampled peak. The memory increment is 0.3–0.4 GB. These costs concern the inspected implementation specified in Appendix C.2. Neither the predictor nor reference inversion is used for test generation (Appendix C.3).


W2. Reference children achieve positive mean reward gains over their corresponding no-mix parents at all four tested inversion settings. The same 30 AID images are evaluated at four settings:

Table Z2. Inversion-level sensitivity; 30 AID images.

SettingLPIPS ↓PSNR ↑SSIM ↑Child rewardParent gain
0.150.12029.500.8800.448+0.031
0.250.18227.090.8370.459+0.045
0.400.30323.370.7420.451+0.040
0.600.45019.800.6100.418+0.022

Our setting of 0.25 gives the highest measured candidate reward and mean parent-relative gain among the four settings. At 0.60, reconstruction quality and child reward decline, but the mean gain over the corresponding no-mix parent remains positive (+0.022). These measurements quantify the sensitivity while showing a candidate-level benefit across the tested settings. The lowest reconstruction error occurs at 0.15, indicating that reconstruction fidelity alone does not determine exploration quality. Varying the setting changes both the exploration center and inversion error. Sec. 4.6 reports the sensitivity result and its scope; Appendix C.3 describes the inversion procedure, and Appendix A.3 retains inversion error in the conditional benefit bound.

Comments on the paper

2 comments
Area_Chair_NyC6OpenReview ↗

Reviewer-author discussion kickoff

Dear Reviewers,

Thank you for your review. The authors have now provided their rebuttal. We would appreciate it if you could take some time to read their responses and engage in the discussion, particularly on whether your concerns have been adequately addressed.

Thank you for your time and contributions to the review process!

AuthorsOpenReview ↗

Global Response: Summary of Rebuttal and Manuscript Revisions

We thank the Area Chair for coordinating the discussion and all three reviewers for their constructive feedback. We have provided point-by-point responses and revised the manuscript, with changes marked in blue. Below we summarize the main additional evidence and the corresponding locations in the revised PDF.


1. Learned candidate construction improves reward beyond frozen and fixed-coefficient controls. On 300 AID contexts, with the context bank and continuation generator held fixed, the final predictor improves pooled child reward by +0.0051 over frozen initialization (95% CI [0.0031, 0.0072]) and +0.0063 over the tuned fixed coefficient (95% CI [0.0035, 0.0092]). This directly evaluates the learning benefit in candidate construction.

Revision: Sec. 4.6, Table 6(b) (p. 10); Appendix C.4, Table 12 (p. 28), including checkpoint-level reward, parent-relative gain, and win rate.

2. Matched reference-aware controls support an additional benefit from entry-wise construction. On the same 120-image AID test bank, three reference-aware constructions share the coefficient initialization of 1. Across three seeds, entry-wise construction improves the mean score by approximately +0.017 over the fixed control and +0.0083 over the learned-shared control, with positive differences in every seed. These controls are reported separately from the original benchmark runs.

Revision: Sec. 4.6, Table 6(a) (pp. 9-10). Our response to Reviewer 2S4z, Part 2, Q3 also reports the additional paired comparison against DiffusionNFT.

3. Measured MRI resources quantify the training cost at a matched scored-candidate budget. For three recorded jobs per method, each with 600 updates and 4,800 scored training candidates, mean A100 GPUh are 5.899 / 5.102 / 4.871 for DiffusionNFT / RACE / RACE+, respectively. Sampled peak memory is 44.2 / 44.5 / 44.6 GB. The predictor has 19,024 parameters and is unused at inference.

Revision: Appendix C.1, Table 10 (p. 25); Appendix C.2 (p. 26) specifies the implementation underlying these measurements. Additional validation-quality-versus-GPUh comparisons appear in the responses to cSEM (Part 2, Q3) and 2S4z (Part 2, Q2).

4. The inversion diagnostic measures reconstruction quality together with candidate reward. Four settings are evaluated on the same 30 AID images. Reference children have positive mean gains over their corresponding no-mix parents at all four settings. The setting 0.25 gives the highest measured child reward and parent-relative gain; 0.15 gives the lowest reconstruction error. The revised discussion clarifies that changing the inversion level affects both the exploration center and inversion error.

Revision: Sec. 4.6, Table 7 (p. 10); Appendix C.3 (pp. 26-27) describes the inversion procedure.

5. The problem formulation and gradient assumptions are now explicit. Sec. 3 begins with a self-contained statement of the task, training resources, scored-candidate budget, and held-out generation objective. We clarify the input-gradient conditions behind equal-field cancellation and distinguish the analyzed gradient graph from the implementation underlying the recorded MRI resources. The empirical evidence is presented separately from the sufficient conditions in the theory.

Revision: Sec. 3 (p. 3), Sec. 3.4 (p. 6), Appendix A.4, and Appendix C.2 (pp. 25-26).


Detailed responses are provided under the reviews of ztf1, cSEM (Parts 1-2), and 2S4z (Parts 1-2), including the additional endpoint-discrepancy and gradient diagnostics. We hope this overview helps locate the revisions and assess the added evidence, and welcome further questions during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.