acceptodds
Under review as a conference paper at ICLR 2027

ReCAT: Recurrent Composable Activation Transition for Test-Time Scaling of Diffusion Models

10 views

Abstract

Diffusion test-time scaling typically spends additional compute on generating, evaluating, or revising multiple candidates. We introduce ReCAT, a sequential scaling method that instead refines a single evolving internal state within one sampling trajectory. At a fixed intervention point, the frozen generator computes an intermediate activation once; a lightweight prompt- and state-conditioned controller then recurrently applies the same learned transition for a number of steps chosen at inference before generation resumes. Each additional refinement step requires only one controller call and no online reward queries. Reward-Ascent Matching (RAM) learns this transition from black-box rewards by fitting reward changes to activation displacements within sample groups and distilling the resulting secant directions across prompts. The resulting controller can be used directly; an optional second stage, Reward-Flow Calibration (RFC), further calibrates it on states reached by the controller's own updates using antithetic continuation probes. Neither stage backpropagates through the generator or reward model. Across three diffusion backbones and four target rewards, RAM alone yields consistent reward improvements, RFC provides additional gains, and reward gains increase with the number of refinement steps throughout the calibrated range. ReCAT also composes with search-based test-time scaling methods, improving Best-of-N search and gradient-based noise search.

Primary area:
generative models
Keywords:
Diffusion Models, Text to Image Synthesis, Test Time Scaling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_9aCqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReCAT, a test-time scaling method for diffusion models that recurrently refines intermediate activations using a lightweight controller. Reward-Ascent Matching (RAM) estimates reward-ascent directions from activation displacements and reward differences, while Reward-Flow Calibration (RFC) further calibrates these directions during recurrent refinement. Experiments demonstrate consistent improvements across multiple diffusion backbones and reward functions.

Critical strengths
  • Efficient test-time scaling: ReCAT improves generation quality through lightweight controller evaluations without requiring additional denoising steps or online reward queries.

  • Consistent empirical improvements: The method achieves improvements across three diffusion backbones and four reward functions. It also provides additional gains when combined with Best-of-N and ReNO.

Critical weaknesses
  • High offline training cost: Although inference is efficient, RAM and RFC require numerous generator forward passes and reward evaluations. A quantitative analysis of the overall training cost is insufficient.

  • Uncertain accuracy of RAM's gradient approximation: RAM estimates reward-ascent directions from a limited number of samples in a high-dimensional activation space. It remains unclear whether the local linearity assumption is sufficiently accurate for activations generated from independent seeds.

  • Non-differentiable reward: Although the proposed method can accommodate non-differentiable reward functions, all reward functions evaluated in the paper are differentiable. The lack of experiments with non-differentiable rewards leaves this potential advantage insufficiently validated. Including such experiments would further strengthen the paper.

Questions for authors

Questions are in weakness section.

10 comments

We sincerely thank the reviewer for the constructive review, and for recognizing the efficiency of the method and the consistency of its gains. Each of the three weaknesses pointed to something the submission should have shown, and in response to each we have conducted new experiments or analyses, which we describe below. The corresponding changes are included in the uploaded revised PDF (changes in blue).

W1. High offline training cost and insufficient quantitative cost analysis.

We thank the reviewer for pointing this out. We agree that the offline training cost is substantial and that the submission did not quantify it; the revision now does (Appendix D, Table 12). The cost has three parts:

  1. Sample pool: 240,000 forward-only generations with reward scores per backbone (5,000 prompts × S=48S = 48 seeds), generated once and reused for every reward; generating and scoring it takes about 277 / 1,013 / 233 A10G-hours (SANA-1.5 / SD3.5-Medium / FLUX.2-klein; estimated from the single-image latency).
  2. RAM: no generator passes; closed-form S×SS \times S solves plus fitting a 9–13M-parameter controller, which takes 0.53 / 1.10 / 2.65 A10G-hours per reward (SANA-1.5 / SD3.5-Medium / FLUX.2-klein; mean over the four rewards). The RAM controller alone already improves all 12 backbone–reward pairs (Table 3); RFC is an optional calibration on top of it.
  3. RFC: the only stage with additional generator computation, namely forward-only rollouts of the frozen suffix from the edited step onward, with no backward pass through the generator; this takes 127 / 424 / 108 A10G-hours per reward (mean over the four rewards).

All three stages run on a single A10G (24 GB), the GPU used for inference. This cost is paid once and is not specific to ReCAT: every learned method has an offline training cost, and the same sample pool trains Probe-Select, the learned test-time baseline. What the offline cost buys is cheap inference: once trained, ReCAT adds 78 / 340 / 385 ms per image (SANA-1.5 / SD3.5-Medium / FLUX.2-klein) and queries no reward model, whereas the methods without training (Best-of-N, Zero-Order Search, ReNO) pay additional generations and reward evaluations for every image.

W2. Accuracy of RAM's gradient approximation in high-dimensional activation space with independently seeded samples.

This is an important question, and the reviewer's caution is warranted: across independent seeds, local linearity holds only approximately. RAM, however, does not require more than that. It requires a direction that points uphill, not an accurate regression. We tested this directly on SANA-1.5 before fitting any controller (Section 3.2; Appendix B.7, Table 8 in the revised version): stepping activations along the closed-form teacher direction raises all four rewards, whereas stepping against it lowers all four (200 prompts × 16 seeds, step size η=0.2\eta = 0.2; Δ/σ):

Step along the teacher, before controller fittingImageRewardAestheticHPSv2PickScore
+g^c⋆+\widehat g_c^\star direction+0.393+0.377+0.470+0.216
−g^c⋆-\widehat g_c^\star direction−0.361−0.775−0.749−0.645

With SS samples in pp dimensions (S=48S = 48 in our experiments, pp in the millions), RAM does not attempt to recover the full gradient. Instead, it estimates the gradient's projection onto the span of the SS centered displacements (Proposition 3, under a local affine model as ρ→0\rho\to0), that is, onto the directions along which samples of this prompt actually differ. The locality weights down-weight the Taylor remainders and cross-seed reward deviations of distant seeds, and the ridge term bounds how much the sampled geometry amplifies the remaining errors (Lemma 4). The controller then pools these per-prompt estimates over 5,000 prompts, and RFC recalibrates the predicted direction on the states the controller actually visits.

The ablation in Table 2 supports the same conclusion: on the same 48-seed groups, targets that simply pull toward high-reward samples fail (group best −1.24 to −2.39σ; reward-weighted mean −1.49 to −2.60σ), and the entropic OT barycenter gains only +0.01 to +0.08σ, whereas the secant regression yields +0.22 to +1.00σ.

W3. Lack of experiments using genuinely non-differentiable rewards.

We agree, and we thank the reviewer for a suggestion that strengthens the paper. The four rewards in the main benchmark are all differentiable; we chose them so that the gradient-based baselines ReNO and Noise Hypernet could be run, while ReCAT itself uses only their scalar scores. We have now run an experiment with a genuinely non-differentiable reward on SANA-1.5 with RAM only; it is added to the revision (Section 4.2; Appendix B.8, Table 9). Neither ReNO nor Noise Hypernet can be applied in this setting.

JPEG file size (the non-differentiable reward used in DDPO, Black et al., ICLR 2024). We re-scored the existing 240k pool, without any new generations, and evaluated on the 500 test prompts (Δ/σ):

MethodCompressibilityIncompressibility
Best-of-2+0.56+0.59
Best-of-32+1.66+2.45
ReCAT (RAM-only)+2.55+4.72

We thank the reviewer again for these suggestions; each of them led to a new result that makes the paper more complete. If they resolve the reviewer's concerns, we would be grateful if the reviewer would consider updating the score, and we would be glad to clarify any remaining points.

Most of my concerns have been adequately addressed. However, the provided table 12 has reinforced my concern about the substantial training cost. Compared with methods such as Best-of-N, which require neither a precomputed training sample pool nor additional training, the advantages of the proposed method appear limited. I will therefore maintain my negative rating.

Although the experiments conducted during the rebuttal period are limited, they provide preliminary evidence of the method's applicability to non-differentiable rewards. To further strengthen the work, I suggest including larger-scale experiments with such rewards.

Reviewer_EmNdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReCAT, a new method for reward alignment of text-to-image diffusion/flow models. ReCAT modifies hidden activations at one block in the middle of the network after around 20% of the denoising steps. A small network learns a reward-aligned direction to push these activations in, and they repeat this push 30 times at this 20% time-step. The network is trained without backpropagating through the generator or the reward model, in two stages. First (RAM), they generate 48 images per prompt and fit a local linear regression of reward on the differences between their activations, giving an estimated direction for each sample. Second (RFC), they generate two images, once nudged along the predicted direction and once nudged against it. Whichever version scores higher decides whether that direction is kept or flipped, and the network is retrained on these labels. ReCAT is evaluated on two multi-step models (SANA-1.5, SD3.5M) and one 4-step distilled model (FLUX.2-klein) with four rewards. They report gains in the training reward on all 12 model–reward pairs, also when ReCAT is combined with Best-of-N, Zero-Order Search, and ReNO.

Critical strengths
  • The training recipe is new. RAM learns activation edits from reward scores only, with no backprop through the generator or the reward model. RFC then calibrates on the controller’s own states, again without backprop. This works across three different backbones including few and multi-step methods. This is unique as most methods are only designed for either few or multi-step diffusion mdoels.
  • ReCAT composes with existing methods. ReCAT + Best-of-2 beats Best-of-2 on all twelve backbone–reward pairs. Same for ReCAT + ReNO. The gains hold across the full search budget sweep out to 48 generation-equivalents.
  • The ablations are useful and well-chosen. Naive targets (group-best, reward-weighted mean) produce negative gains. Block activations beat noise and latent interventions. RFC without RAM initialization does almost nothing. These controls justify the design.
Critical weaknesses
  1. Few-step vs multi-step is conflated. The main issue is the framing: the paper treats few-step (4) distilled generation (FLUX.2-klein) and multi-step (20/28) generation (SANA, SD3.5) as interchangeable settings, but they aren’t, and they require fundamentally different considerations. For ReCAT, editing one step means different things when that’s 25% of the trajectory vs 3.5%. The timestep sweep was only run on SANA-1.5 and reused for the other backbones without justification, and for FLUX with only 4 steps, “20% of steps” doesn’t clearly map to a step index. The role of CFG (present in SANA and SD3.5 but not FLUX) in how edits propagate is also not discussed. This also conflates the baselines: as far as I understand, ReNO and Noise Hypernet were designed for few-step (or one-step) models, so running them through 20- or 28-step chains puts them outside their intended setting. The results also show this, with Noise Hypernet collapsing to negative numbers on SD3.5M that look like training collapse, not a meaningful comparison. For multi-step models, more natural competitors would be guidance-based approaches, FK steering, or fine-tuning methods like Adjoint Matching. On FLUX, where Noise Hypernet is actually in-regime, the comparison is mixed (Noise Hypernet wins HPSv2 and PickScore, Aesthetic collapsed, ReCAT wins ImageReward), but this is never discussed. The same mismatch shows up in reverse in the fine-tuning comparison (B.4, Table 6). SFT and DPO are trained with their standard loss, which is designed for multi-step and not meant for step-distilled models, so it is no surprise that they break FLUX.2-klein (SFT lowers all four rewards, DPO three of four). On the multi-step models, where these methods are in their intended setting, DPO actually beats ReCAT on 3 of 4 rewards on SANA-1.5 and 2 of 4 on SD3.5M. This only appears in the appendix and is not reflected in the main claims. Overall, the paper would benefit from clearly separating and clarifying these two regimes and choosing baselines appropriate to each while also reflecting on which might be the more natural setting for ReCAT.
  2. No check for reward hacking. All results are evaluated with the training reward. There is no cross-reward evaluation (e.g., train on ImageReward, evaluate on HPSv2), or general benchmark (e.g. GenEval, UniGenBench) that is disentangled from the training reward. For a method that explicitly pushes activations toward higher reward scores, this is a significant omission and should definitely be added.
  3. Limited analysis of compute costs. Training requires 5000 prompts × 48 seeds generations for the first stage, plus RFC rounds with 2 decoded images at each step. While inference cost is provided in detail and analyzed, the training cost is not much discussed or clarified in detail and how it compares to fine-tuning/Noise Hypernet.
  4. Novelty is somewhat overstated. Asyrp [1] already learns a network to edit mid-layer activations for semantic editing, which is structurally quite similar and not cited. The “sequential scaling” framing is also undercut by Table 3: a single application of the controller captures most of the gain (e.g., +1.22 vs +1.39 for Aesthetic on SANA-1.5), so the recurrence story seems weaker than initially presented.

[1] Kwon et al. "Diffusion Models already have a Semantic Latent Space". ICLR 2023.

Questions for authors
  • What is the authors motivation for using ReCAT for multi-step diffusion models instead of LoRA-based fine-tuning methods like DPO/Adjoint Matching/DRaFT/DiffusionNFT? How does the compute/quality tradeoff look like? How about for few-step models and ReNO/Noise Hypernet?
  • How does ReCAT compare to these fine-tuning methods with LoRA, both in quality and compute? How expensive is the RAM training, how expensive is the RFC training comparatively?
  • Which CFG branch is edited for SANA/SD3.5M? Are both edited or only one of them?
  • Could one do "editing" through ReCAT by pushing activations based on a different prompt? So the prompt embedding for ReCAT computed from a different prompt than the generation one? E.g. a certain style?
11 comments

We sincerely thank the reviewer for the detailed and expert review. Following the reviewer's suggestions, the revision adds:

  • FK steering on all three backbones, alone and combined with ReCAT;
  • cross-reward, FID, and GenEval evaluation of every controller;
  • a training-cost comparison with fine-tuning and Noise Hypernet;
  • the regime of each baseline, with the in-regime comparisons discussed in the main text;
  • the citation of Asyrp.

We address each point below; the changes are marked in blue in the uploaded revised PDF.

W1. Separating few-step and multi-step regimes and ensuring appropriate baseline comparisons.

We agree that few-step and multi-step generation are different settings, and that the submission should have stated the regime of each backbone and baseline explicitly. The revision states in Appendix C which baselines run outside their intended regime and discusses the in-regime comparisons in Section 4.2. Because the two regimes differ, a method that works in both is informative, which the reviewer also lists among the strengths. Of the learned methods that change the generated image (ReCAT, Noise Hypernet, SFT, and DPO), ReCAT is the only one that improves all four rewards in both regimes, and it does so with the same recipe. We address the specific points below.

The meaning of "20%", and one edit being 25% vs. 3.5% of the trajectory. The edit is applied after the first k=round⁡(0.2T)k = \operatorname{round}(0.2 T) of the TT denoising steps, with T−kT - k steps remaining (Appendix D in the revised version):

BackboneSteps TTCFGSteps before the edit (kk)Steps after the edit (T−kT - k)
SANA-1.520yes416
SD3.5-Medium28yes622
FLUX.2-klein4no (distilled)13
  • Position and granularity. On FLUX.2-klein, "20%" maps to k=1k = 1: the edit is applied after the first of the 4 steps. As the reviewer notes, the granularity differs: a single step is 1 of 4 on FLUX.2-klein, against 1 of 20 and 1 of 28. In relative position, however, the three backbones are close: the edit is applied at one point, after 20%, 21%, and 25% of the denoising steps (SANA-1.5, SD3.5-Medium, FLUX.2-klein), and the edited state then passes through the remaining 80%, 79%, and 75%.
  • Timestep sweep. The reviewer is correct that the sweep over the intervention timestep was run only on SANA-1.5 (Fig. 10). We reuse its 20% setting on SD3.5-Medium and FLUX.2-klein, and for all four rewards, rather than tuning each case: the aim is to show that one recipe works across backbones, not to maximize each entry. The three backbones differ in the number of steps (20, 28, and 4) and in the use of CFG, and with this transferred setting ReCAT improves all four rewards on both. Even on SANA-1.5, the sweep gives Aesthetic its highest gain at 50%, yet we use 20% (Fig. 10).
  • CFG. On SANA-1.5 and SD3.5-Medium, only the conditional branch is edited. At that step, the edited conditional output and the unedited unconditional output are combined by the usual CFG formula, and all later steps run unmodified. FLUX.2-klein has no CFG.

ReNO and Noise Hypernet on multi-step models. We agree that these methods were designed for one- or few-step models, and that the Noise Hypernet result on SD3.5-Medium is an out-of-regime outcome rather than a meaningful comparison. For the multi-step backbones, the revision adds FK steering as a competitor.

The in-regime comparison on FLUX.2-klein. The reviewer's reading is correct, and the revised Section 4.2 now states in the main text that Noise Hypernet is stronger on HPSv2 and PickScore. The two methods act at different places: Noise Hypernet amortizes noise optimization into a learned map of the initial noise, trained with reward gradients through the sampling chain, whereas ReCAT edits an intermediate activation and is trained from scalar rewards. ReCAT composes with every noise-space method we combined it with: ReCAT + ReNO is higher than ReNO on all 12 backbone–reward pairs, as are ReCAT + Best-of-NN and ReCAT + Zero-Order Search over their counterparts (Table 4).

More natural multi-step competitors. Following the reviewer's suggestion, we ran FK steering on all three backbones (500 test prompts, two particles; Appendix B.9, Table 10 in the revised version). FK requires stochastic continuations, so this comparison alone uses a stochastic sampler. The table gives the generations and reward queries per prompt, and the gains (Δ/σ):

BackboneMethodGenerationsReward queriesImageRewardAestheticHPSv2PickScore
SANA-1.5FK steering (2 particles)210+0.364+0.336+0.149+0.624
SANA-1.5ReCAT10+0.263+1.393+0.783+0.442
SANA-1.5ReCAT + FK (2 particles)210+0.565+1.414+0.792+0.866
SD3.5-MediumFK steering (2 particles)210+0.510+0.206+0.428+0.333
SD3.5-MediumReCAT10+0.116+0.910+0.499+0.295
SD3.5-MediumReCAT + FK (2 particles)210+0.561+0.980+0.591+0.415
FLUX.2-kleinFK steering (2 particles)26+0.366+0.381+0.313+0.536
FLUX.2-kleinReCAT10+0.308+1.353+0.520+0.475
FLUX.2-kleinReCAT + FK (2 particles)26+0.740+1.766+0.922+0.967
  • With one generation and no reward queries, ReCAT outperforms two-particle FK on Aesthetic and HPSv2 and underperforms it on ImageReward and PickScore, on all three backbones.
  • ReCAT also composes with FK. At two particles, ReCAT + FK is higher than FK on all four rewards on all three backbones.

Comparison with fine-tuning. ReCAT is a test-time method, and the comparison the paper is built around is with the other amortized test-time methods on a frozen generator, Probe-Select and Noise Hypernet (Table 1). Fine-tuning changes the generator weights, so we treat it as a reference point rather than as a competitor, which is why the submission placed SFT and DPO in Appendix B.4 (Table 6). The two are also not exclusive: ReCAT needs only forward passes of the generator it is given and a reward that still varies across seeds, so the same procedure can be run unchanged on a fine-tuned checkpoint.

The numbers are as the reviewer reads them, and the revision reflects both observations. The revised Section 4.2 now says in the main text that full-parameter DPO exceeds ReCAT on some rewards on the multi-step backbones. Appendix B.4 now states that SFT and DPO use their standard objectives, which are designed for multi-step models, and that on step-distilled FLUX.2-klein they run outside that regime.

Which setting is more natural for ReCAT? Both: ReCAT places no requirement on the regime. Each alternative assumes something that ReCAT does not. The standard fine-tuning objectives are designed for multi-step models; ReNO and Noise Hypernet are designed for few-step models and require a differentiable reward; and fine-tuning and Noise Hypernet backpropagate through the generator, so their training needs more memory than inference, whereas ReCAT's training uses only forward passes of the generator and runs on the hardware used for inference. ReCAT is therefore most useful where one of these assumptions fails: on a step-distilled generator, with a reward that provides no gradient, or when training is limited to inference hardware.

W2. Missing cross-reward evaluation or a general benchmark independent of the training reward.

We agree and have carried out both evaluations; they are added to the revision (Section 4.2; Appendix B.6, Table 7). Below, each SANA-1.5 controller is evaluated on all four rewards (bold marks the training reward):

SANA-1.5 controllerImageReward (Δ/σ)Aesthetic (Δ/σ)HPSv2 (Δ/σ)PickScore (Δ/σ)
Base0.0000.0000.0000.000
ImageReward+0.263+0.089+0.298+0.099
Aesthetic (prompt-independent reward)+0.066+1.393−0.316−0.267
HPSv2+0.206+0.111+0.783+0.243
PickScore+0.094+0.464+0.306+0.442

As a benchmark independent of the training reward, we report official GenEval accuracy (553 prompts × 4 seeds) for every controller on all three backbones. No controller optimizes it; a controller that exploited its reward model would be expected to lose compositional accuracy:

ControllerSANA-1.5 GenEval ↑SD3.5-Medium GenEval ↑FLUX.2-klein GenEval ↑
Base0.7960.7010.825
ImageReward0.8010.7080.831
Aesthetic (prompt-independent reward)0.7700.6700.802
HPSv20.8050.7020.833
PickScore0.7970.7020.826

The three prompt-conditioned controllers (ImageReward, HPSv2, PickScore) improve the rewards they were not trained on: every one on SANA-1.5, and all but one across the three backbones (the ImageReward controller on PickScore on SD3.5-Medium, −0.006σ). None of them lowers GenEval on any backbone (+0.001 to +0.009 over the base). The Aesthetic controller, whose reward does not see the prompt, is the exception: it lowers most of the prompt-conditioned rewards and lowers GenEval on all three backbones.

When a single controller is instead trained on a weighted sum of all four normalized rewards, it improves all four (SANA-1.5, Δ/σ):

Training objectiveImageRewardAestheticHPSv2PickScore
Weighted combination+0.154+0.807+0.490+0.520
Reviewer_MmT1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ReCAT, a test-time scaling method for text-to-image diffusion models that refines intermediate block activations within a single sampling trajectory. At a fixed early denoising step (20% progress) and DiT block, a lightweight (9–13M parameter) ConvNeXt controller gθg_\theta is recurrently applied KK times to update the activation before the frozen generator completes the remaining denoising steps. Training proceeds in two gradient-free stages without backpropagating through the generator or reward model: (1) Reward-Ascent Matching (RAM), which generates groups of S=48S=48 samples per prompt and fits local ridge regressions of reward differences on activation displacements ("secant teachers") to supervise gθg_\theta, and (2) Reward-Flow Calibration (RFC), which unrolls the controller and uses two-sided finite-difference probes to retain or flip the predicted update direction on out-of-distribution unrolled states. Experiments across three backbones (SANA-1.5, SD3.5-M, FLUX.2-klein) and four scalar rewards show consistent improvements in normalized reward uplift (Δ/σ\Delta/\sigma) with minimal inference latency overhead.

Critical strengths
  • Practical Test-Time Scaling Axis: Moving test-time compute from parallel candidate generation (e.g., Best-of-NN) or full-trajectory regeneration to recurrently updating an intermediate activation is a compelling direction that adds very little wall-clock latency and requires no reward queries at inference.

  • Black-Box Training Efficiency: Learning the activation transition purely from forward evaluations avoids backpropagating through multi-billion-parameter diffusion chains and allows training with non-differentiable or black-box rewards on modest hardware.

  • Composability with Search Methods: The method demonstrates consistent positive reward uplift across three architectures.

Critical weaknesses
  • Opaque Presentation of Core Mechanisms (Secant Teachers & RFC): The exposition in Section 3 is unnecessarily dense and obscures relatively straightforward numerical ideas behind heavy terminology. Specifically, I understand that the "secant teacher" in RAM is simply a locally weighted ridge regression approximating the reward gradient via finite differences within a sample group, and the "antithetic continuation probe" is a two-point central difference check along ±g^θ\pm \hat{g}_\theta that only updates the sign (orientation) of the controller's existing prediction rather than discovering new ascent directions. Clearer, more intuitive explanations of how RAM and RFC operate and interact are needed in the main text.

  • Risk of Reward Gaming & Lack of FID / Cross-Reward Evaluation: Each ReCAT controller is trained on a single scalar reward and evaluated only on that exact same in-target reward via normalized uplift (Δ/σ\Delta/\sigma). It is unclear whether the controller is genuinely improving overall generation quality or simply gaming/over-optimizing the target reward model's biases. No FID scores are reported to verify that the base generator's distributional realism is preserved, nor are cross-reward evaluations provided.

Questions for authors
  • Because each controller is evaluated solely on the reward it was trained to maximize, how can we be sure ReCAT is not exploiting reward-model artifacts?
3 comments

We sincerely thank the reviewer for the thoughtful review, and for recognizing the practical value of this test-time scaling axis and of black-box training. Both of the reviewer's points led to concrete improvements in the paper. The revision states the reviewer's reading of RAM and RFC directly at the start of Sections 3.2 and 3.3, and adds cross-reward, FID, and GenEval evaluation of every controller. We address both points below; the changes are marked in blue in the uploaded revised PDF.

W1. Making secant teachers and RFC more intuitive and clarifying what the continuation probes actually learn.

The reviewer's summary is accurate, and clearer than our original text. The revised Sections 3.2 and 3.3 now state it directly at their start:

  • "Within each sample group, RAM fits a locally weighted ridge regression of reward differences on activation differences."
  • "RFC therefore calibrates the RAM controller along its KtrainK_{\mathrm{train}}-step unrolls …: a two-point central-difference probe along the current prediction decides whether RFC keeps or flips it."

As the reviewer notes, a probe only checks the orientation of the current prediction; it does not discover new directions. This is the intended division of labor: RAM finds the directions from grouped samples, and RFC checks them on the states that recurrence actually visits. RFC builds on RAM and makes recurrence more consistently beneficial: RFC started from a random controller gains almost nothing (+0.07σ vs. +0.78σ on HPSv2, Table 2), and without RFC, recurrence is worse than a single application on 3 of 12 backbone–reward pairs; after RFC, it is better on all 12 (Table 3).

W2 & Q1. Reward-model exploitation, cross-reward evaluation, and preservation of distributional realism, including FID.

We thank the reviewer for raising this important point, and we agree that this evaluation was missing from the submission. The revision adds it (Section 4.2; Appendix B.6, Table 7). Below, each SANA-1.5 controller is evaluated on all four rewards (500 test prompts; bold marks the training reward):

SANA-1.5 controllerImageReward (Δ/σ)Aesthetic (Δ/σ)HPSv2 (Δ/σ)PickScore (Δ/σ)
Base0.0000.0000.0000.000
ImageReward+0.263+0.089+0.298+0.099
Aesthetic (prompt-independent reward)+0.066+1.393−0.316−0.267
HPSv2+0.206+0.111+0.783+0.243
PickScore+0.094+0.464+0.306+0.442

To assess distributional fidelity and compositional accuracy, we also report FID on 5,000 held-out MS-COCO captions and official GenEval accuracy (553 prompts × 4 seeds) for every controller on all three backbones:

ControllerSANA-1.5 FID ↓SANA-1.5 GenEval ↑SD3.5-Medium FID ↓SD3.5-Medium GenEval ↑FLUX.2-klein FID ↓FLUX.2-klein GenEval ↑
Base34.480.79629.310.70130.810.825
ImageReward34.350.80128.870.70830.730.831
Aesthetic (prompt-independent reward)41.890.77030.050.67030.400.802
HPSv235.760.80528.780.70230.870.833
PickScore34.170.79728.480.70232.500.826

On SANA-1.5, the three prompt-conditioned controllers (ImageReward, HPSv2, PickScore) improve every reward they were not trained on; across the three backbones, they do so in 26 of 27 cases (3 controllers × 3 other rewards × 3 backbones), the one exception being the ImageReward controller on PickScore on SD3.5-Medium (−0.006σ). Relative to the base, their FID changes by −0.83 to +1.69 and is lower in 6 of the 9 cases, and their GenEval accuracy is higher in all 9, by 0.001 to 0.009. The Aesthetic controller, whose reward does not see the prompt, behaves differently: it lowers 7 of its 9 prompt-conditioned evaluations and lowers GenEval on all three backbones. We report this result in the revision.

When the aesthetic reward is instead combined with prompt-conditioned rewards, a single controller improves all four rewards (SANA-1.5, Δ/σ):

Training objectiveImageRewardAestheticHPSv2PickScore
Weighted combination+0.154+0.807+0.490+0.520

The objective is a weighted sum of the four normalized rewards, ∑kwkrk/σk\sum_k w_k r_k/\sigma_k, with weights 0.5 / 1.0 / 0.5 / 1.0 on ImageReward / Aesthetic / HPSv2 / PickScore.

We thank the reviewer again for these suggestions; the clearer exposition and the new evaluations have made the paper stronger. If they resolve the reviewer's concerns, we would be grateful if the reviewer would consider updating the score, and we would be glad to clarify any remaining points.

I thank the authors for their comprehensive experiments to address my questions and concerns.

After reading through the author responses accross all reviews,

  • I am more confident that the author's claimed results and methodology are sound.
  • I am also happy that the authors are taking seriously the raised concern that the text was too convoluted to follow, and
  • I hope that they will do additional polishing to ensure the method is clear and not left to interpretation.

On a side note, some reviewers raised concerns regarding the relevant literature and position of the paper all of which seem to have been sufficiently addressed by the authors.

As a result, I will change my score.

Thank you very much for taking the time to read through all the responses and for raising your score. Your feedback really helped us explain the method more clearly, and we will keep polishing the writing to make sure the final version is as clear and easy to follow as possible.

Reviewer_6sUWOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper tackles the task of test-time scaling of diffusion models and proposes ReCAT, a sequential scaling approach that trains a prompt-conditioned controller operating on a fixed diffusion timestep and a fixed set of intermediate network activations. The controller learns an update rule for these activations that points toward higher-reward regions. It is trained using Reward-Ascent Matching, which, to simplify, involves generating a group of samples and their corresponding rewards to obtain activation-reward pairs (ai,ri)(a_i, r_i). For each activation aia_i, the reward gradient is then approximated, up to a projection onto the displacement subspace, by solving a linear regression problem. The authors also propose a secondary post-training stage in which the controller is further fine-tuned on its own rollouts to better align with the true reward gradient field during long recurrences. The method is evaluated on text-to-image generation using several base models and reward functions, and is shown to considerably improve per-sample rewards at a negligible computational overhead, which is very promising.

Critical strengths

The core idea of the paper is quite beautiful and elegant. It is somewhat related to looped transformer models, and its application to diffusion models in this way is quite lovely to see. The paper itself is also well written, although some sections can be difficult to follow and would benefit from further explanation. The experimental evaluation is also impressive, with a large number of ablations and studies that help with a detailed understanding of the approach. The results themselves are also quite promising, especially considering the negligible computational overhead introduced by the method.

Critical weaknesses

Some parts of the main paper are quite technically dense, with little explanation, making them challenging to follow. Here are a few examples:

  • The paragraph on constructing teachers in Section 3.2 is a bit difficult to follow. I would suggest putting the linear system being solved in Equation (6) as well.
  • The introduction of κ(a)\kappa(a) could use some further explanation or intuition. What is the intuition or motivation behind the choice of a tanh function? Is it simply a heuristic choice with bounded outputs?
  • The setup in lines 324-326 is a bit confusing. Assuming (a,c)(a,c) are fixed, what exactly does "label" refer to? Is it χ\chi? Based on the definition in line 307, χ\chi is a function of aa and would be constant, which does not seem to make sense in this context. Also, if (a,c)(a,c) are fixed, what remains random in the collection process? Does this setup assume a stochastic diffusion sampler?
Questions for authors
  • According to Table 3, it seems that the relative improvements that recurrence gives with RFC diminish as the base model improves. Specifically, on FLUX.2-klein, recurrence seems to do very little. Is this a correct interpretation of the results? Does this suggest that, as base models improve, recurrence may no longer be worth the additional computational effort?
  • The results in Figure 5 are a bit confusing. As far as I understand, the step size is Δs=η/Ktrain\Delta s = \eta / K_{\text{train}}. Therefore, wouldn't increasing the number of integration steps KtrainK_{\text{train}} be similar to increasing the number of denoising steps during diffusion sampling, leading to a more accurate integration of the learned update field up to the horizon η\eta? However, the quality visibly starts deteriorating once the number of recurrence steps exceeds the maximum used during training. What explains this behavior?
5 comments

We sincerely thank the reviewer for the careful reading and the generous assessment of our work; we are glad that the reviewer found the core idea elegant and the evaluation thorough. The reviewer's comments on Section 3 were very helpful, and we have followed each of them. We address each point below and describe the corresponding revisions, which are included in the uploaded revised PDF (changes in blue).

W1. Clarifying teacher construction in Section 3.2 and the linear system in Equation (6).

We thank the reviewer for this suggestion and agree that the original presentation of Section 3.2 was too terse. Following the suggestion, Section 3.2 now states the regression objective explicitly, specifies the dimension of every matrix, and gives the linear system that is actually solved (Eq. 3):

gc⋆(ai)=(D~i⊤Wi2D~i+ρIp)−1D~i⊤Wi2y~i=D~i⊤αi,αi=Wi(WiK~(i)Wi+ρIS)−1Wiy~i,g_c^\star(a_i) = \big(\widetilde{D}_i^\top W_i^2 \widetilde{D}_i + \rho I_p\big)^{-1} \widetilde{D}_i^\top W_i^2 \widetilde{y}_i = \widetilde{D}_i^\top \alpha_i, \qquad \alpha_i = W_i\big(W_i \widetilde{K}^{(i)} W_i + \rho I_S\big)^{-1} W_i \widetilde{y}_i,

where D~i∈RS×p\widetilde{D}_i \in \mathbb{R}^{S \times p} stacks the centered activation displacements, y~i∈RS\widetilde{y}_i \in \mathbb{R}^S the centered rewards, Wi=diag⁡(wij)W_i = \operatorname{diag}(w_{ij}) the locality weights, and K~(i)=D~iD~i⊤∈RS×S\widetilde{K}^{(i)} = \widetilde{D}_i \widetilde{D}_i^\top \in \mathbb{R}^{S \times S}.

To make the construction easier to follow, we also open the section with a one-sentence summary: RAM is a locally weighted ridge regression of reward differences on activation differences among the SS samples of one prompt. Although the unknown has pp entries, with pp in the millions, the solution lies in the span of the SS centered displacements, so it suffices to solve an S×SS \times S system; the result estimates the reward gradient projected onto this span.

W2. Motivation for χ(a)\chi(a) and the choice of the tanh function.

Intuitively, the label χ(a)=tanh⁡((r+−r−)/(κσR))\chi(a) = \tanh \big((r^+ - r^-)/(\kappa\sigma_R)\big) acts as a soft confidence gate on each probe, where σR\sigma_R is the across-seed reward standard deviation and κ\kappa is a temperature. It enters the RFC loss (Eq. 7) in two ways: sign⁡(χ)\operatorname{sign}(\chi) decides whether the controller's direction is kept or flipped, and ∣χ∣\lvert \chi\rvert weights that state.

  • Small gaps are down-weighted. Small reward gaps receive small weights ∣χ∣\lvert \chi\rvert, reducing their contribution to the training loss while keeping the target direction at unit norm.
  • Large gaps are capped. Once the gap exceeds a few κσR\kappa\sigma_R, the weight saturates at one, so a few large gaps do not dominate the objective, similar to clipping.

The reviewer is right that tanh itself is a heuristic choice. Its role in the loss comes from these two properties of the weight ∣χ∣\lvert \chi\rvert. Other functions that vanish continuously at zero gap and saturate for large gaps, e.g., a clipped linear map, would play the same role. The revised Section 3.3 adds this explanation and states this point explicitly.

W3. Label definition and remaining randomness when (a, c) are fixed in lines 324–326.

The reviewer's reading is correct. The label is χ\chi. For a fixed state (a,c)(a,c) and collection checkpoint θˉ\bar\theta, the probe pair a±=a±h(a)g^θˉ(a,c)a^\pm = a \pm h(a) \widehat g_{\bar\theta}(a,c) is fixed. Since the frozen suffix and the reward models are both deterministic in RFC, χ\chi is then a fixed value. Nothing remains random at a fixed state, and no stochastic sampler is assumed.

The only randomness in the collection process is which states are visited, determined by the sampled prompts and initial noises; this is the expectation in Eq. (7). Eq. (8) was written with conditional expectations as the general form; in our deterministic setting it reduces to

g⋆(a,c)=sign⁡(χ(a,c))g^θˉ(a,c).g^\star(a,c) = \operatorname{sign}(\chi(a,c)) \widehat{g}_{\bar\theta}(a,c).

We thank the reviewer for flagging this passage. Since RFC is trained in this deterministic setting, the revised Section 3.3 states this form directly and mentions the general form only in a brief remark. The experimental protocol (Appendix C) also states explicitly that the default samplers are deterministic.

Q1. Diminishing recurrence gains on stronger backbones, particularly FLUX.2-klein.

We thank the reviewer for this careful reading of Table 3. The interpretation holds for Aesthetic on FLUX.2-klein, but not for the other three rewards. Table 3 compares the recurrent unroll with a single application of the same RFC checkpoint. Below, each cell gives the increment from recurrence, with the single-application and recurrent gains (Δ/σ) in parentheses:

BackboneImageRewardAestheticHPSv2PickScore
SANA-1.5+0.12 (+0.14 → +0.26)+0.17 (+1.22 → +1.39)+0.06 (+0.72 → +0.78)+0.05 (+0.39 → +0.44)
SD3.5-Medium+0.05 (+0.07 → +0.12)+0.12 (+0.79 → +0.91)+0.06 (+0.44 → +0.50)+0.03 (+0.27 → +0.30)
FLUX.2-klein+0.05 (+0.26 → +0.31)+0.03 (+1.32 → +1.35)+0.07 (+0.45 → +0.52)+0.07 (+0.41 → +0.48)

On FLUX.2-klein, the increments on the other three rewards are comparable to those on the other two backbones. The increments also do not shrink as the base model gets stronger: SD3.5-Medium is the weakest base model on all four rewards and FLUX.2-klein the strongest (Table 7 in the submission; Table 11 in the revised version), yet SD3.5-Medium does not have the largest increment on any reward, and FLUX.2-klein has the largest on HPSv2 and PickScore.

The number of recurrent steps is chosen at inference on the same checkpoint, and its cost is small: on FLUX.2-klein, the 30 steps add 385 ms to a 3.35 s generation (Table 1). We therefore enable recurrence by default.

Q2. Why does quality deteriorate beyond the maximum recurrence length used during training in Figure 5?

We thank the reviewer for this question. For K>Ktrain=30K \gt K_{\mathrm{train}} = 30, the controller is applied to states outside the distribution seen during RFC training, so training and inference no longer match. The controller extrapolates to some extent, but this is not guaranteed: in Figure 5 the gain rises through K≤KtrainK \le K_{\mathrm{train}} on every backbone–reward pair, while beyond KtrainK_{\mathrm{train}} it keeps rising in some cases and drops in others.

We do not select the best KK for each pair: all primary results are reported at K=KtrainK = K_{\mathrm{train}}, the number of steps used in training.

We thank the reviewer again for the encouraging and constructive feedback, which has made Section 3 considerably clearer. If the revisions resolve the reviewer's concerns, we would be grateful if the reviewer would consider updating the score, and we would be glad to clarify any remaining points.

We thank the reviewer again for the encouraging review and for the concrete suggestions, which made the revised paper clearer. As the discussion period draws to a close, we would like to ask whether our response and the revisions have addressed the reviewer's questions on the presentation and on the recurrence behavior. If so, we would be grateful if the reviewer would consider updating the score. We would be glad to clarify any remaining detail.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We are grateful to the Area Chair and all reviewers for their time and constructive feedback.

We thank Reviewers 6sUW, MmT1, and 9aCq for actively engaging with our responses during the discussion and for their encouraging remarks on our approach. All three raised their ratings and now support acceptance.

We thank Reviewer EmNd for the detailed review and many concrete suggestions, and we responded to each concern point by point. For example, we clarified which baselines suit few-step and multi-step models, added cross-reward evaluations, and compared ReCAT with fine-tuning. Adding RAM improved all four target rewards on each of the four fine-tuned checkpoints, showing that test-time activation refinement can complement fine-tuning. The discussion period is about to close, and we have not yet heard back from the reviewer. We hope the additional experiments and clarifications address their concerns.

Again, we appreciate all reviewers’ suggestions, which helped us strengthen the evaluation and make the paper clearer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.