- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ReCAT, a test-time scaling method for diffusion models that recurrently refines intermediate activations using a lightweight controller. Reward-Ascent Matching (RAM) estimates reward-ascent directions from activation displacements and reward differences, while Reward-Flow Calibration (RFC) further calibrates these directions during recurrent refinement. Experiments demonstrate consistent improvements across multiple diffusion backbones and reward functions.
Critical strengths
-
Efficient test-time scaling: ReCAT improves generation quality through lightweight controller evaluations without requiring additional denoising steps or online reward queries.
-
Consistent empirical improvements: The method achieves improvements across three diffusion backbones and four reward functions. It also provides additional gains when combined with Best-of-N and ReNO.
Critical weaknesses
-
High offline training cost: Although inference is efficient, RAM and RFC require numerous generator forward passes and reward evaluations. A quantitative analysis of the overall training cost is insufficient.
-
Uncertain accuracy of RAM's gradient approximation: RAM estimates reward-ascent directions from a limited number of samples in a high-dimensional activation space. It remains unclear whether the local linearity assumption is sufficiently accurate for activations generated from independent seeds.
-
Non-differentiable reward: Although the proposed method can accommodate non-differentiable reward functions, all reward functions evaluated in the paper are differentiable. The lack of experiments with non-differentiable rewards leaves this potential advantage insufficiently validated. Including such experiments would further strengthen the paper.
Questions for authors
Questions are in weakness section.
10 comments
W3. Lack of experiments using genuinely non-differentiable rewards.
We agree, and we thank the reviewer for a suggestion that strengthens the paper. The four rewards in the main benchmark are all differentiable; we chose them so that the gradient-based baselines ReNO and Noise Hypernet could be run, while ReCAT itself uses only their scalar scores. We have now run an experiment with a genuinely non-differentiable reward on SANA-1.5 with RAM only; it is added to the revision (Section 4.2; Appendix B.8, Table 9). Neither ReNO nor Noise Hypernet can be applied in this setting.
JPEG file size (the non-differentiable reward used in DDPO, Black et al., ICLR 2024). We re-scored the existing 240k pool, without any new generations, and evaluated on the 500 test prompts (Δ/σ):
| Method | Compressibility | Incompressibility |
|---|---|---|
| Best-of-2 | +0.56 | +0.59 |
| Best-of-32 | +1.66 | +2.45 |
| ReCAT (RAM-only) | +2.55 | +4.72 |
We thank the reviewer again for these suggestions; each of them led to a new result that makes the paper more complete. If they resolve the reviewer's concerns, we would be grateful if the reviewer would consider updating the score, and we would be glad to clarify any remaining points.
Most of my concerns have been adequately addressed. However, the provided table 12 has reinforced my concern about the substantial training cost. Compared with methods such as Best-of-N, which require neither a precomputed training sample pool nor additional training, the advantages of the proposed method appear limited. I will therefore maintain my negative rating.
Although the experiments conducted during the rebuttal period are limited, they provide preliminary evidence of the method's applicability to non-differentiable rewards. To further strengthen the work, I suggest including larger-scale experiments with such rewards.
We sincerely thank the reviewer for the constructive review, and for recognizing the efficiency of the method and the consistency of its gains. Each of the three weaknesses pointed to something the submission should have shown, and in response to each we have conducted new experiments or analyses, which we describe below. The corresponding changes are included in the uploaded revised PDF (changes in blue).
We thank the reviewer for pointing this out. We agree that the offline training cost is substantial and that the submission did not quantify it; the revision now does (Appendix D, Table 12). The cost has three parts:
All three stages run on a single A10G (24 GB), the GPU used for inference. This cost is paid once and is not specific to ReCAT: every learned method has an offline training cost, and the same sample pool trains Probe-Select, the learned test-time baseline. What the offline cost buys is cheap inference: once trained, ReCAT adds 78 / 340 / 385 ms per image (SANA-1.5 / SD3.5-Medium / FLUX.2-klein) and queries no reward model, whereas the methods without training (Best-of-N, Zero-Order Search, ReNO) pay additional generations and reward evaluations for every image.
This is an important question, and the reviewer's caution is warranted: across independent seeds, local linearity holds only approximately. RAM, however, does not require more than that. It requires a direction that points uphill, not an accurate regression. We tested this directly on SANA-1.5 before fitting any controller (Section 3.2; Appendix B.7, Table 8 in the revised version): stepping activations along the closed-form teacher direction raises all four rewards, whereas stepping against it lowers all four (200 prompts × 16 seeds, step size η=0.2; Δ/σ):
With S samples in p dimensions (S=48 in our experiments, p in the millions), RAM does not attempt to recover the full gradient. Instead, it estimates the gradient's projection onto the span of the S centered displacements (Proposition 3, under a local affine model as ρ→0), that is, onto the directions along which samples of this prompt actually differ. The locality weights down-weight the Taylor remainders and cross-seed reward deviations of distant seeds, and the ridge term bounds how much the sampled geometry amplifies the remaining errors (Lemma 4). The controller then pools these per-prompt estimates over 5,000 prompts, and RFC recalibrates the predicted direction on the states the controller actually visits.
The ablation in Table 2 supports the same conclusion: on the same 48-seed groups, targets that simply pull toward high-reward samples fail (group best −1.24 to −2.39σ; reward-weighted mean −1.49 to −2.60σ), and the entropic OT barycenter gains only +0.01 to +0.08σ, whereas the secant regression yields +0.22 to +1.00σ.