acceptodds
Under review as a conference paper at ICLR 2027

Bounding the Initial-Noise Density Ratio Keeps Every Image under Reward Alignment

Abstract

Reward fine-tuning raises the score of text-to-image models under a reward model but makes many correct images of a prompt hard to sample: on Stable Diffusion 3.5-Medium, DRaFT raises PickScore by 1.41 while the share of base-model images its outputs cover falls from 94.8% to 34.7%. Penalties on KL divergence, diversity, or update size control averages, which a rare composition barely moves. We instead require every set of images for a prompt to keep its probability within fixed factors of the base model. Our key observation is that a frozen sampler maps each initial noise to one image, so the probability of an image set is the Gaussian mass of its noise preimage, and bounding the density ratio of the initial noise bounds every image set at once. RATIO learns a prompt-conditioned invertible noise transform whose density ratio stays in at every point, for any latent dimension, throughout training: Gaussian-preserving rotations choose directions at no cost, and monotone maps in Gaussian-CDF coordinates move probability under one budget shared by all coordinates. At the same PickScore gain, RATIO keeps 85.7% recall against 69.2–80.6% for seven fine-tuning and noise-space methods, and tests on 5,051 GenEval image events find no violation of its bounds, against 179–386 for the controls. Across three generators and three rewards, the bound fixes how much probability moves, and the reward decides which images receive it.

Primary area:
generative models
Keywords:
reward alignment, image generation, probability bounds, generative models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jjHyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies coverage loss during reward alignment of text-to-image generators. It proposes preserving every measurable image event within fixed probability factors of the base distribution. With a frozen generator, this guarantee follows from a bounded density ratio in initial-noise space. The proposed method, RATIO, combines Gaussian-preserving conditional rotations with monotone transformations in Gaussian-CDF coordinates, using a shared coefficient budget to obtain dimension-independent bounds. The paper also characterizes the optimal reward under these constraints and supports differentiable and black-box reward optimization. Experiments cover three generators and three rewards, including matched-reward coverage comparisons, event-level tests, architectural ablations, and human evaluations.

Critical strengths
  1. A meaningful guarantee enforced by construction. The paper clearly distinguishes average divergence control from multiplicative protection of individual events. The main technical contribution is the tractable transform that enforces this stronger requirement: conditional rotations preserve the Gaussian reference, while CDF transformations and a shared coefficient budget control the density ratio throughout training. The composition argument correctly handles non-Gaussian intermediate distributions. I did not identify a substantive error in the central exact-arithmetic proofs.

  2. Substantial empirical support for the reward–coverage trade-off. The evaluation goes beyond reporting reward improvements. At the matched PickScore gain, RATIO achieves 85.7% recall versus 69.2–80.6% for the external controls. The Affine+KL comparison is particularly informative because it uses the same conditioning interface and nearly matches the mean noise KL, yet exhibits substantially more event-bound violations. Cross-generator experiments, black-box reward training, and rotation ablations further support the usefulness of the construction.

Critical weaknesses
  1. The experiments only partially validate the motivating claim about rare, within-prompt alternatives. PRDC recall pools features across prompts, so it does not directly measure conditional coverage. Moreover, only 1,085 of the 5,051 events are deletion-detectable at the reported sample sizes (Table 9); the appendix acknowledges that rare-event deletions are largely undetectable. The tests also provide per-event rather than family-wide error control. Thus, zero rejections should not be interpreted as strong empirical verification of rare-event preservation, although this limitation does not invalidate the theorem.

  2. The matched-reward comparison does not fully establish the achievable performance frontier. Baseline metrics are linearly interpolated between relatively coarse post-training strength settings, whereas RATIO uses separately trained budget settings. Interpolated PRDC recall need not equal the recall of an actual sampler at the target reward, and scaling an already trained update may underperform training directly for that operating point. Uncertainty for these comparisons is also missing, which matters where the reported advantages are small.

Questions for authors
  1. Comparison validation: Can you evaluate actual baseline settings at RATIO’s target reward and provide paired uncertainty estimates? For a representative strong baseline, does training or selecting checkpoints directly for this reward level change the coverage comparison?

  2. Event-test interpretation: How many baseline violations remain after an appropriate multiple-testing correction? Can you provide a targeted evaluation of low-probability, prompt-valid alternatives with sufficient samples to detect their deletion, and report within-prompt coverage separately from pooled PRDC recall?

  3. Implementation clarification: Appendix D.4 specifies BF16 generator/conditioner computations and FP64 CDF/density calculations. Which precision is used for rotations and intermediate transformed latents? Can you report forward–inverse consistency and density-ratio stress tests, particularly in the tails, to quantify the gap between the exact-arithmetic guarantee and the implemented sampler?

4 comments
AuthorsOpenReview ↗

Response to Reviewer jjHy (1/3): matched-gain comparison at directly evaluated strengths

(1/3) We thank you for a review that engages with exactly the parts of the paper we care most about. Your summary of the strengths is how we would put it ourselves: the paper separates average divergence control from multiplicative protection of each event, and the contribution is a transform that enforces the stronger requirement at every training step. It meant a lot to us that you checked the composition argument for non-Gaussian intermediate latents and found no substantive error in the exact-arithmetic proofs. Your questions are precise and testable, so we answered each with an experiment or a re-analysis; all are in the new Appendix G (blue), and this reply covers Q1. Parts 2 and 3 cover the event tests and the numerics.

Q1 / W2. Are the matched-gain results an artifact of interpolation?

No; the comparison holds when every control is evaluated at its own matched strength, with the caveat on DrawBench below. Appendix G.1 (Table 15) evaluates each control directly at the strength s∗s^* that attains RATIO's PickScore, found by five bisection steps on PickScore alone, using initial noises disjoint from the reported images. Recall and CLIPScore never enter the choice. For the two knobs that are separate fits (λ\lambda for HyperNoise-L2 and Affine+KL), we retrain at λ∗=1.19\lambda^*=1.19. Intervals are paired 95% bootstrap intervals that resample prompts (same indices for both methods) and each method's three training runs, 10,000 times.

Pick-a-Pic, target 22.58Interp.DirectRATIO − control [95% CI]
DRaFT69.269.516.2 [14.6, 17.8]
Flow-GRPO75.675.89.9 [8.7, 11.2]
DiverseGRPO78.078.67.1 [5.6, 8.5]
GRPO-Guard80.681.44.3 [2.8, 5.9]
Affine+KL (refit at λ∗\lambda^*)75.075.99.8 [8.1, 11.4]
RATIO85.785.7

The remaining four rows are in Table 15. Two observations. First, RSA-FT and DiffusionNFT sit almost exactly at an evaluated strength (w=0.0010w=0.0010 and 0.99410.9941), so their direct values must match the interpolation, and they do (+0.0 and -0.1 points); this checks the protocol. Second, interpolation did understate the controls slightly (mean +0.4, at most +0.9 points), most where the target lies well inside a bracket; HyperNoise, entered as the envelope of its two knobs, lands 0.3 and 1.7 points below it. The smallest Pick-a-Pic gap shrinks from 5.1 to 4.3 points, and every Pick-a-Pic interval excludes zero.

You were right that uncertainty matters where differences are small. On DrawBench (200 prompts) RATIO still has the highest recall, but its differences to DiverseGRPO (0.7, CI [-1.2, 2.5]) and GRPO-Guard (1.5, CI [-0.5, 3.4]) are inside the intervals. Section 5.1 now says so, and we no longer count them as advantages.

Does training a control directly for the target reward change the picture?

It narrows the gap but does not close it (Appendix G.2, Table 16). For DRaFT, which shares RATIO's 16,000-image exposure, and GRPO-Guard, the strongest control, we select the first checkpoint whose validation gain reaches 0.596; for DRaFT we also retrain with a smaller learning rate tuned to end at that gain. Selection uses validation captions only; three runs per variant.

VariantRecallRATIO − variant [95% CI]
DRaFT, checkpoint selected70.814.9 [13.3, 16.4]
DRaFT, reduced LR72.413.3 [11.8, 14.9]
GRPO-Guard, checkpoint selected80.94.8 [3.2, 6.3]

Your intuition was right that a dedicated run beats a scaled update: the reduced learning rate gains DRaFT about three points. Checkpoint selection changes little for either control, because early checkpoints of a full-strength run already concentrate on the same images. A GRPO-Guard retrained with a larger KL coefficient needs a full reinforcement-learning run per seed, so we covered the strongest control through checkpoint selection, which answers the same question; we will add the retrained variant to the final version. The retention advantage does not come from how the controls were scaled.

Paper changes: Section 5 Protocol paragraph and Section 5.1 (blue); Appendix G.1–G.2, Tables 15–16.

AuthorsOpenReview ↗

Response to Reviewer jjHy (2/3): multiplicity, rare prompt-valid events, and within-prompt coverage

(2/3) This part answers W1 and Q2. We agree with the reading in your review: zero rejections at 128 draws are a falsification check, and the guarantee for rare events rests on the theorem. Section 5.2 now says this in those terms. We then added the three analyses you asked for, so that the empirical side no longer depends on that caveat alone.

How many violations survive a multiple-testing correction?

Many, for every control. Appendix G.3 (Table 17) treats each method's 10,102 directional tests (two per event, including the ones that cannot reject at these sample sizes) as one family. Holm at family-wise level 5% is valid under any dependence; we add Benjamini–Hochberg at FDR 5% as a descriptive count, since events of one prompt share images.

MethodPer test (Table 2)Holmof which prompt-valid deletions
DRaFT34310168
GRPO-Guard180178
HyperNoise3508158
Affine+KL38611476
RATIO ρ=2\rho=2 / ρ=3\rho=30 / 00 / 00 / 0

All eight controls keep between 17 and 114 certified violations. The comparison you found most informative survives: at the same mean noise KL, Affine+KL keeps 76 certified prompt-valid deletions and RATIO none. Holm's first threshold is 4.95×10−64.95\times10^{-6}, which only large drops of frequent events reach (removing every occurrence of a base-frequency-0.5 event gives p=3.2×10−4p=3.2\times10^{-4}, of a 0.9 event p=3.7×10−8p=3.7\times10^{-8}). The corrected counts therefore measure what these sample sizes can certify, not how many violations occur; we state this in Appendix G.3.

A targeted test of rare, prompt-valid alternatives

Appendix G.4 (Table 18) does what you proposed. From the existing 256-image base pool we select the prompt-valid events seen in 2–38 images (about 0.8–15%), merging events identical to their prompt's correctness event: 241 events from 162 prompts, none detectable with 128 draws. We then draw fresh pools of 4,096 base and 2,048 method images per prompt and test deletion only. At these sizes deletion is detectable above a base frequency of about 1.7%, which holds for 191 events. The test covers RATIO and three controls chosen for their role: DRaFT (same training exposure as RATIO), GRPO-Guard (strongest control), and Affine+KL (equal mean noise KL).

These three controls delete 12–52 of these events per test and 8–43 after Holm within the family; Affine+KL deletes the most (52), DRaFT 47, GRPO-Guard the fewest (12). RATIO deletes none. Because the selection uses only the old pool and every test uses new images, the family is fixed before any test outcome is seen.

Within-prompt coverage, reported separately from pooled recall

Appendix G.5 (Table 19) computes PRDC recall separately for each prompt and averages over prompts. It reuses the evaluation images of Table 15 (ten per prompt from each of three runs, so 30 per prompt) and compares them with 30 independent base images per prompt, with the same k=5k=5 and DINOv2 features.

Within-prompt recallPick-a-PicDrawBench
Base, independent pool90.891.5
RATIO80.285.9
Best control (DiverseGRPO)72.884.6
GRPO-Guard72.581.0
DRaFT57.670.1

Pooling did hide part of the loss: every control's gap to RATIO is larger within prompts than pooled (for GRPO-Guard on Pick-a-Pic, 4.3 pooled vs 7.7 within). DiverseGRPO widens least, which fits how it works: its diversity reward is computed within groups of images of one prompt, so it becomes the closest control under this metric (gap 7.4 and 1.3). Within-prompt recall matches the per-prompt form of our bound, so we now present it next to the pooled metric.

Paper changes: Section 5.2 (blue), limitation (iv) in Section 6, Appendix G.3–G.5, Tables 17–19.

AuthorsOpenReview ↗

Response to Reviewer jjHy (3/3): precision of the implementation and tail stress tests

(3/3) This part answers Q3, which asks how close the implemented sampler is to the exact-arithmetic guarantee. It is a question we should have answered in the submission.

Which precision is used for rotations and intermediate latents?

Appendix D.4 now lists every computation. The generator and the conditioner run in BF16. The Givens angles returned by the conditioner are cast to FP64; cos⁡θ\cos\theta, sin⁡θ\sin\theta, and the rotations are evaluated in FP64; intermediate latents between blocks are stored in FP32. The coefficient normalization (Eq. 4), the Gaussian CDF and its inverse, the sine maps, and all log-density terms are computed in FP64, with tail-stable log-CDF and log-survival evaluations and a residual tolerance of 10−1010^{-10} for the inverse.

This split matters for the guarantee. The bound depends on three things: the normalization, the CDF maps, and the orthogonality of each rotation. The first two are computed in FP64 on whatever raw outputs the BF16 conditioner returns, so conditioner rounding changes which bounded transform is applied, not whether its coefficients satisfy ∑j,k∣ωmjk∣<αm\sum_{j,k}|\omega_{mjk}|<\alpha_m. The rotations are orthogonal to FP64 round-off. What remains is the FP32 storage of latents between blocks.

Forward–inverse consistency and density-ratio stress tests

Appendix G.6 (Table 20) runs each of the three RATIO runs (ρ=2\rho=2) on 10,000 standard Gaussian inputs, and on stress inputs where a random 1% of the 65,536 coordinates are set to ±6\pm6, ±12\pm12, or ±30\pm30 (30,000 draws per regime).

InputMax round-trip error (10−610^{-6})Max log-density mismatch (10−710^{-7} nats)Realized log⁡Λ\log\Lambda (band [−1.77,2][-1.77, 2])Outside band
Standard3.62.9-1.08 / 1.370
±6\pm66.33.8-1.15 / 1.440
±12\pm1213.86.1-1.17 / 1.460
±30\pm3032.66.4-1.17 / 1.460

No draw leaves the band in any regime, and the realized log ratio stays at least 0.54 nats inside the nearer edge. Two built-in checks pass: the sample mean of log⁡Λϕ\log\Lambda_\phi under standard draws is 0.2884 nats, within Monte Carlo error of the measured mean noise KL (0.2896), as Eqlog⁡Λϕ=KL(qϕ∥γ)\mathbb E_q\log\Lambda_\phi=\mathrm{KL}(q_\phi\|\gamma) requires; and the summed rotation log-determinants stay below 10−1310^{-13}. The tails behave as the construction predicts. Beyond about ∣b~j∣=8|\tilde b_j|=8, Smj′S'_{mj} equals its endpoint value to within 10−1410^{-14} (its derivative vanishes at both endpoints), so a coordinate pushed to ±12\pm12 or ±30\pm30 contributes the fixed factor 1/Smj′(0)1/S'_{mj}(0) or 1/Smj′(1)1/S'_{mj}(1), which is inside the per-coordinate band; that is why the ±12\pm12 and ±30\pm30 rows agree. The gap between the exact-arithmetic statement and the implemented sampler is several orders of magnitude below the width of the band. Limitation (iii) in Section 6 states that the guarantee is proved in exact arithmetic and checked numerically.

For the AC discussion, in short

  • The comparison survives direct evaluation, training for the target, and paired intervals on Pick-a-Pic; on DrawBench we now claim only that RATIO is not exceeded.
  • Under Holm, every control keeps certified violations and RATIO none; on rare prompt-valid events with adequate power, the controls delete 12–52 and RATIO none.
  • The implemented transform stays inside its band under tail stress.

These analyses follow your suggestions closely. If they settle your soundness concerns, we would be grateful if you would consider raising your score and supporting the paper in the discussion. Your review made the empirical side of the paper much stronger, and we thank you for it.

Reviewer_jjHyOpenReview ↗

concerns resolved

Thanks to the authors' rebuttal. My concerns have been resolved. I will keep my score as 3, weak acceptance.

Reviewer_4dXQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper addresses the loss of image diversity when text-to-image diffusion and flow models are fine-tuned toward a reward. Instead of fine-tuning the generator, the authors keep it frozen and learn a prompt-conditioned invertible transform of the initial noise. The transform is constructed so that every set of images keeps its probability within fixed factors of the base model. They evaluate the method with several reward models on three text-to-image generators.

Critical strengths
  • The paper addresses an important and practical problem: reward alignment reduces the diversity of text-to-image models.
  • The idea of controlling alignment through the initial noise while keeping the generator frozen is interesting and well motivated.
  • The paper offers a theoretical perspective on how much the aligned model may deviate from the base model.
  • The experiments cover several generators and reward models.
Critical weaknesses
  • The paper is badly written, and the writing hides a simple idea. The main contribution fits in one paragraph. The authors freeze the generator and transform only the initial noise. Then they bound the ratio of the transformed noise density to the standard Gaussian at every point. The text does not state this idea clearly, and makes the reader reconstruct it. The paper needs a substantial revision of its writing to make its ideas clear and its contributions stand out. In its current form, the reader must work hard to separate the main idea from the details, and the contributions are lost among dense text, appendix references, and illustrative figures. A clear explanation of the problem, the key observation, and the method in the main text would make the paper much stronger.

  • The paper uses key quantities before it defines them, so the main claim is difficult to follow. The abstract refers to "the density ratio of the initial noise" and to "image events", but it does not first give a learned noise distribution or define an event. The introduction uses D_KL(Q ‖ P) before it defines P and Q, and it uses \gamma(z) before Section 3, which gives the first definition. The paper does not explain "Gaussian-CDF coordinates" until Section 4.1. The paper uses phrases without justification, such as "the bound fixes how much probability moves, and the reward decides which images receive it", in place of explanations.

  • The main text does not stand on its own, and the figures do not support the claims. Reading Section 5 requires the matched-gain protocol, the event tests, and the Affine+KL control, all of which are in the appendix. Many main-text claims also cite appendix tables. Table captions read like internal notes (e.g., “Do not average category columns to reconstruct Overall”). The conclusion is one sentence, and there is no limitations section.

Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4dXQ (1/2): the idea now stated in one paragraph, and every term defined before use

(1/2) Thank you for a review that is direct and, on reflection, correct. Your summary of the contribution is the clearest one we have seen, clearer than our own submission: freeze the generator, transform only the initial noise, and bound the ratio of the transformed noise density to the standard Gaussian at every point. You also took the time to list the specific places where the text made the reader reconstruct this idea. We used that list as a checklist, item by item, and the revised PDF is a better paper because of it. We are also glad that you found the problem important, the noise-space approach interesting and well motivated, and the paper technically sound.

Every change below is marked in blue in the revision.

"The writing hides a simple idea; the main contribution fits in one paragraph."

You are right, and we now say it in one paragraph. Section 1 has a new paragraph titled The idea in one paragraph, placed right after the problem statement:

RATIO freezes the generator and changes only its input. With a fixed deterministic sampler and decoder, SD3.5 turns each initial noise zz into one image G(z,c)G(z,c), so the probability of a set of images is the probability of the noise vectors that produce it. The base model draws zz from the standard Gaussian, whose density we write γ(z)\gamma(z); RATIO draws it from a learned density q(z∣c)q(z\mid c) instead. If ℓ≤q(z∣c)/γ(z)≤u\ell\le q(z\mid c)/\gamma(z)\le u at every zz, then every set of images, including sets nobody specified, keeps its probability between ℓ\ell and uu times its base value. [...]

It is essentially your sentence, and it is the first place in the paper where the method appears. The introduction then gives the three design choices in plain terms (map each coordinate to [0,1][0,1] with the Gaussian CDF and reweight it there; one budget shared by all coordinates; rotations that leave the Gaussian unchanged) and ends with a short roadmap of Sections 3–5. Section 4 keeps the same order, with one requirement per subsection, so the reader can match each design choice to the problem it solves.

"Key quantities are used before they are defined."

We went through every term you listed and checked the order in which a reader meets it.

  • "The density ratio of the initial noise" (abstract). The abstract now says "the ratio between the learned noise density and the standard Gaussian density at every point".
  • "Image events" (abstract). An event is now defined where it first appears: "every set of images for a prompt (an event, such as all correct overhead views)". Section 3 gives the formal definition with examples of a composition, a count, and a local attribute.
  • DKL(Q∥P)D_{\mathrm{KL}}(Q\|P) in the introduction. The symbol is gone; the sentence now reads "the KL divergence of the aligned image distribution from the base distribution". PcP_c and QcQ_c are introduced in words before any formula uses them.
  • γ(z)\gamma(z) before Section 3. It is now defined in the new idea paragraph ("the standard Gaussian, whose density we write γ(z)\gamma(z)"), before any use.
  • "Gaussian-CDF coordinates". Both the abstract and the introduction now explain the operation: the Gaussian CDF Φ\Phi sends each coordinate to [0,1][0,1], where it is uniform, an increasing map reweights it there, and the density ratio is the inverse derivative of that map. Section 4.1 gives the derivation.

"Phrases used in place of explanations."

We agree that "the bound fixes how much probability moves, and the reward decides which images receive it" read as a slogan. It is meant as shorthand for Proposition 2: within the bounds, the reward-optimal distribution is Qc⋆=ℓPc+(1−ℓ)PcτQ^\star_c=\ell P_c+(1-\ell)P^\tau_c, so the bounds fix the amount 1−ℓ1-\ell of mass that moves and the top fraction τ\tau that receives it, while the reward's ranking decides which images make up that top fraction. The abstract now states the claim concretely ("the bound limits how far the probability of any image set can move, and the reward determines which images gain probability"), and Section 4.5 uses the phrase only after the formula that justifies it. We removed the similar opener of Section 5.3 and replaced it with what it meant: every ablated variant satisfies the same bound (Eq. 5), and they differ in how much reward the transform obtains within it.

Part 2 covers the experiments section, the captions, the figures, and the conclusion.

AuthorsOpenReview ↗

Response to Reviewer 4dXQ (2/2): a self-contained Section 5, rewritten captions, and a conclusion with limitations

(2/2) This part covers your third point, that the main text did not stand on its own.

"Reading Section 5 requires the matched-gain protocol, the event tests, and the Affine+KL control, all in the appendix."

You are right, and Section 5 now defines all three before using them. A new Protocol paragraph at the start of Section 5 explains, in the main text:

  • Matched gain. Every method has one strength knob (ρ\rho for RATIO, the update scale ss or penalty weight λ\lambda for the controls), and a matched-gain comparison places all methods at RATIO's PickScore. The paragraph says how Table 1 obtains these values and where they are evaluated directly.
  • Event test. An event test checks one image set of one prompt by comparing its frequency among 256 base and 128 method images with the envelope of Theorem 1, using one-sided Clopper–Pearson bounds.
  • Affine+KL. The Setup paragraph now says what this control is and why it is there: it is our own control, keeps RATIO's conditioner and noise interface, and replaces the CDF maps with a conditional affine map under a KL penalty, so that it isolates the pointwise bound from the noise interface.

The numbers that carry each claim are stated in the text of Section 5, and Tables 1–5 and Figures 4–5 are in the main text; appendix tables now hold details (per-category scores, interpolation weights, resources) and the added analyses.

"Table captions read like internal notes."

We agree; the example you quoted was ours and was not written for a reader. The DPG-Bench caption (Table 12) now explains the reason instead of giving an instruction: "Because the two aggregations differ, Overall is not the mean of the category columns." We reread every caption for the same problem. The resources caption (Table 6) was the other such case, and it now says that Flow-GRPO is omitted because we use the models released by its authors, whose training cost is not known to us.

"The figures do not support the claims."

Figure 1 is meant to show the problem (one prompt has many correct images), not to prove a result, and its caption used to suggest otherwise. The caption now says that the figure illustrates the problem and that Table 1 and Section 5.2 measure how much of this variation each method keeps. The figures that carry evidence are Figure 4 (event margins of RATIO and Affine+KL at the same mean noise KL, the source of the 232 deletions) and Figure 5 (Aesthetic reward: text fidelity and coverage along both methods' strength paths). Figure 3 shows Proposition 2 on real images, and Figure 2 is the architecture.

"The conclusion is one sentence, and there is no limitations section."

Section 6 is now Conclusion and limitations. The conclusion summarizes the reduction, the evidence on three generators, and the role of ρ\rho. The limitations are stated plainly: (i) RATIO only reweights images the base model already draws, so its gain is capped (position correctness cannot exceed 86.9% at ρ=2\rho=2, while fine-tuned controls reach about 95%); (ii) the lower factor also protects incorrect images; (iii) the guarantee is proved in exact arithmetic and checked numerically; (iv) finite-sample tests have limited power for rare events, which the new targeted test addresses; and (v) the learned transform realizes 60–74% of the attainable gain.

In summary

The revision keeps the method, the theorems, and the experiments, and changes how they are presented, which is what your review asked for. The page budget made this possible: the main text now uses the 10-page allowance, and the additional page went into the idea paragraph, the protocol paragraph, and the limitations. We also added the statistical analyses requested by the other reviewers (Appendix G), which you may find useful as further support for the soundness you already credited.

We would be very grateful if you would look at Sections 1, 3, 5, and 6 of the revision. If they now make the idea and the contributions clear, we would appreciate it if you would reconsider your presentation and overall scores. Any remaining passage that you find hard to follow, we will gladly rewrite during the discussion.

I thank the authors for their response and appreciate their effort to incorporate my suggestions. However, even after reviewing the changes to the main text, my concern remains. Clarifying jargons and technical terms before their appearance, along with adding a paragraph highlighting the contributions, certainly helps. But my concerns were not about specific missing definitions -- those were only examples in my review. It's about the language and presentation of the paper as a whole. Even after the edits, the paper’s presentation and writing remain quite difficult to follow. I maintain my stance that the convoluted and dense writing obscures the method, and that the paper needs a substantial rewrite in clear and plain language before it's ready for publication. I therefore still maintain my original recommendation and score.

Reviewer_8oVoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how reward alignment can improve average scores while reducing coverage of a pretrained text-to-image model’s outputs. The authors propose RATIO, a prompt-conditioned invertible initial-noise transform with a dimension-independent density-ratio bound. For a frozen generator, this bound transfers to every measurable image event. The construction combines Gaussian-preserving conditional rotations with monotone Gaussian-CDF maps under a shared coefficient budget. Experiments span three generators and three rewards, including structural ablations and event-level evaluation. On SD3.5-Medium, RATIO achieves 85.7% PRDC recall at a PickScore gain of approximately 0.60, compared with interpolated baseline results of 69.2–80.6%.

Critical strengths
  1. A constructive guarantee for preserving output probabilities. RATIO enforces a density-ratio bound through its architecture and transfers it to every image event under a frozen generator. The dimension-independent coefficient budget makes this guarantee applicable to realistic latent spaces, connecting a precise distributional constraint to a trainable method.
  2. Targeted empirical support for the proposed mechanism. The experiments combine cross-generator evaluation, differentiable and black-box rewards, structural ablations, and an affine control with similar mean noise KL. These comparisons provide useful evidence that the bounded transform improves reward–retention trade-offs.
Critical weaknesses
  1. Matched-gain comparisons lack direct validation and uncertainty estimates. The headline comparison linearly interpolates metrics between evaluated strengths and reports point estimates. Since PRDC recall is nonlinear in the generated sample distribution, directly evaluated matched-gain settings and paired confidence intervals would strengthen the claimed advantage, particularly for smaller differences.
  2. Event-level evidence requires careful statistical interpretation. Tests control error per event rather than across all 5,051 events, and only 1,085 events are deletion-detectable at the stated sample sizes. Although these limits are discussed, multiplicity-adjusted results would clarify how many specific violations are supported and how strongly the experiments assess rare-event retention.
Questions for authors
  1. Clarification and additional validation. Are directly evaluated matched-gain results available, rather than interpolated metrics? Could you provide paired confidence intervals for the recall differences, accounting for training runs and evaluation sampling?
  2. Clarification of event testing. How many deletion and amplification rejections remain after an appropriate multiple-testing correction? Please distinguish these results from the reported per-event counts. This requests analysis of existing measurements rather than a new benchmark.
  3. Clarification of human evaluation. Please describe judge recruitment, informed consent, compensation where applicable, and the ethics-review or exemption status of the human studies.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 8oVo (1/2): direct matched-gain evaluation with paired intervals, and multiplicity-corrected event counts

(1/2) Thank you for a careful and constructive review. You put the contribution in the terms we had hoped a reader would take away: a constructive guarantee, enforced by the architecture, that transfers to every image event under a frozen generator, with a dimension-free budget that makes it usable in real latent spaces. We also appreciated that you read the Affine+KL control as evidence for the mechanism rather than as one more baseline. Both of your requests are well posed, and both are now answered with new results in Appendix G of the revision (blue).

Q1. Directly evaluated matched-gain results, with paired intervals

Yes. Appendix G.1 (Table 15) evaluates every control at the strength that attains RATIO's PickScore instead of interpolating. For update-scale knobs, s∗s^* comes from five bisection steps on PickScore alone, on initial noises disjoint from the reported images; for the two penalty knobs we refit at λ∗=1.19\lambda^*=1.19. The 95% intervals for RATIO minus each control are paired: 10,000 resamples of the prompts (same indices for both methods) and, independently, of each method's three training runs, so they account for both training runs and evaluation sampling, as you asked.

Recall at matched gainPick-a-Pic: control → RATIO − control [95% CI]DrawBench: control → difference [95% CI]
DRaFT69.5 → 16.2 [14.6, 17.8]78.7 → 11.4 [9.3, 13.6]
Flow-GRPO75.8 → 9.9 [8.7, 11.2]86.5 → 3.6 [1.9, 5.4]
DiverseGRPO78.6 → 7.1 [5.6, 8.5]89.4 → 0.7 [-1.2, 2.5]
GRPO-Guard81.4 → 4.3 [2.8, 5.9]88.6 → 1.5 [-0.5, 3.4]
Affine+KL75.9 → 9.8 [8.1, 11.4]84.6 → 5.5 [3.4, 7.6]

RATIO's recall is 85.7 (Pick-a-Pic) and 90.1 (DrawBench); the other four control configurations are in Table 15 (HyperNoise appears with two knobs, so nine configurations of eight controls). You anticipated the outcome correctly: direct evaluation shifts the interpolated controls by -0.1 to +0.9 points on Pick-a-Pic (HyperNoise, entered as the envelope of its two knobs, lands 0.3 and 1.7 points below it), and the smaller differences are the ones that need intervals. On Pick-a-Pic all nine intervals exclude zero, with the smallest gap 4.3 points (5.1 under interpolation). On DrawBench (200 prompts) the differences to DiverseGRPO and GRPO-Guard lie inside their intervals, and Section 5.1 now says that RATIO is not exceeded there, rather than that it leads. Appendix G.2 also checkpoint-selects DRaFT and GRPO-Guard for the target reward and retrains DRaFT for it; this narrows the gap to at least 4.8 points, still with an interval above zero.

Q2. Deletion and amplification rejections after multiplicity correction

Appendix G.3 (Table 17) re-analyzes the existing images; nothing new is sampled. For each method, the family is all 10,102 directional tests, two per event, including those that cannot reject at these sample sizes. We report Holm (family-wise 5%, valid under any dependence) as the primary correction and Benjamini–Hochberg (FDR 5%) as a descriptive count, since events of one prompt share images. Deletions and amplifications are reported separately:

MethodDeletions: per test → Holm → BHAmplifications: per test → Holm → BH
DRaFT223 → 74 → 141120 → 27 → 58
Flow-GRPO160 → 41 → 92105 → 14 → 41
GRPO-Guard78 → 9 → 33102 → 8 → 30
Affine+KL262 → 83 → 167124 → 31 → 61
RATIO (ρ=2\rho=2, ρ=3\rho=3)0 → 0 → 00 → 0 → 0

Every control keeps certified violations under Holm (17–114 in total), and at equal mean noise KL Affine+KL keeps 76 certified prompt-valid deletions against RATIO's zero. Table 2 and Section 5.2 now label the original counts as per-test and point to the corrected ones. Holm's first threshold (4.95×10−64.95\times10^{-6}) can only be reached by large drops of frequent events, so the corrected counts show how many violations these samples can certify, not how many occur; Part 2 describes the test we added for rare events.

AuthorsOpenReview ↗

Response to Reviewer 8oVo (2/2): how strongly rare-event retention is tested, and the human studies

(2/2) Your second weakness also asked how strongly the experiments assess rare-event retention. We agree that the 128-draw tests say little about rare events, and Section 5.2 now states this directly. Two new analyses address it.

A test with the power to detect rare deletions (Appendix G.4, Table 18)

From the existing 256-image base pool we select prompt-valid events seen in 2–38 images (about 0.8–15%), which gives 241 events from 162 prompts; none of them is deletion-detectable with 128 draws. Every test then uses fresh pools of 4,096 base and 2,048 method images per prompt, so the selection cannot bias the outcome. At these sizes deletion is detectable above a base frequency of about 1.7%, which covers 191 events. The test covers RATIO and three representative controls (DRaFT, GRPO-Guard, Affine+KL). They delete 12–52 of them per test and 8–43 after Holm within the family, in the same order as the prompt-valid deletions of Table 2 (Affine+KL most, GRPO-Guard fewest). RATIO deletes none.

Coverage within each prompt (Appendix G.5, Table 19)

Because pooled PRDC recall lets an output of one prompt cover a base image of another, we also compute recall within each prompt. RATIO remains above every control on both prompt sets (80.2 vs at most 72.8 on Pick-a-Pic; 85.9 vs at most 84.6 on DrawBench), and every gap is larger than in the pooled metric.

Q3. Human evaluation: recruitment, consent, compensation, and ethics

Thank you for asking; this information belonged in the paper and is now in Appendix E.4 (Judges and ethics). Judges were adult crowdworkers recruited through an online crowdsourcing platform. Before the task they were told that all images are machine-generated and that they could stop at any time, and they were paid for their work. The task carries minimal risk: judges saw only generated images and public benchmark prompts (DrawBench and the SFW Pick-a-Pic captions), gave only judgments about the images, and no personal data beyond the judgments and the platform's anonymous worker identifiers were collected. The protocols of the three tasks (pairwise preference, set diversity, overprocessing) are unchanged and described in the same appendix.

For the AC discussion, in short

  • Guarantee. The bound holds by construction for every parameter value; Appendix G.6 confirms that the implemented transform stays inside it, including under tail stress.
  • Matched gain. With direct evaluation and paired intervals, RATIO's advantage holds on Pick-a-Pic against all nine control configurations (eight methods; HyperNoise is evaluated with two knobs) (smallest gap 4.3 points); on DrawBench no control exceeds it.
  • Events. After Holm correction, every control keeps certified violations and RATIO has none; with adequate power for rare prompt-valid events, the controls delete 12–52 and RATIO none.
  • All additions validate claims that were already in the paper; no claim was added.

We tried to make each answer checkable from the tables alone. If the new results resolve the concerns behind your soundness score, we would be grateful if you would consider raising your rating and supporting the paper in the discussion. We thank you again for questions that made the empirical part of the paper considerably stronger.

Thank you for the direct matched-gain evaluations, paired confidence intervals, and multiplicity-corrected event tests. These substantially address my main empirical concerns. The additional rare-event analysis and human-study details also strengthen the response. I remain positive about the architectural guarantee and supporting evidence, and maintain my original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: summary of the revision and of the added analyses

We are grateful to all three reviewers for careful and generous reviews. Each review changed the paper in a specific place, and we have uploaded a revision in which the new and revised passages are printed in blue.

What the reviewers agree on

  • The guarantee is meaningful and holds by construction: Reviewer jjHy calls it "a meaningful guarantee enforced by construction" and found no substantive error in the exact-arithmetic proofs; Reviewer 8oVo describes "a constructive guarantee for preserving output probabilities" that connects "a precise distributional constraint to a trainable method".
  • The problem is important and the noise-space approach is well motivated (Reviewer 4dXQ), and the experiments cover several generators and rewards.
  • The Affine+KL control, which matches RATIO's mean noise KL and still deletes many events, is "particularly informative" (Reviewer jjHy) and "useful evidence" (Reviewer 8oVo).

What we changed (in blue)

  1. Writing (Reviewer 4dXQ). Section 1 now states the idea in one paragraph: freeze the generator, change only its input, and bound the ratio of the learned noise density to the standard Gaussian at every point. Every quantity (event, density ratio, γ\gamma, Gaussian-CDF coordinates) is defined before it is used; the abstract defines events and the density ratio in words. Section 5 opens with a self-contained Protocol paragraph (matched gain, event test, Affine+KL). The conclusion is now Conclusion and limitations (Section 6) with five explicit limitations. Captions that read like notes were rewritten.
  2. Matched-gain validation (Reviewers jjHy, 8oVo). Appendix G.1 evaluates every control directly at RATIO's reward with paired 95% intervals; Appendix G.2 checkpoint-selects DRaFT and GRPO-Guard for that reward and retrains DRaFT for it.
  3. Event-test statistics and rare events (Reviewers jjHy, 8oVo). Appendix G.3 applies Holm and Benjamini–Hochberg corrections over all 10,102 directional tests; Appendix G.4 tests 241 rare prompt-valid events with pools large enough to detect their deletion; Appendix G.5 reports coverage within each prompt.
  4. Numerics (Reviewer jjHy). Appendix D.4 states the precision of every computation; Appendix G.6 reports forward–inverse consistency and the realized density ratio under tail stress.
  5. Human studies (Reviewer 8oVo). Appendix E.4 describes recruitment, consent, compensation, and why the task carries minimal risk.

What the added analyses show (SD3.5-M, PickScore; RATIO at ρ=2\rho=2)

QuestionResultWhere
Does interpolation favor RATIO?Direct evaluation moves the interpolated controls' Pick-a-Pic recall by -0.1 to +0.9 points (HyperNoise: 0.3 and 1.7 below its envelope); every Pick-a-Pic interval excludes zero; the smallest gap is 4.3 points (GRPO-Guard)Table 15
Does training or selecting a control for the target help?A little; the gap stays at least 4.8 points, interval above zeroTable 16
Do violations survive multiplicity correction?Holm leaves 17–114 per control; RATIO 0Table 17
Are rare valid alternatives deleted?12–52 of 241 rare events deleted by DRaFT, GRPO-Guard, Affine+KL; 0 by RATIOTable 18
Does pooled recall hide within-prompt loss?Within-prompt recall widens every gap; RATIO stays highestTable 19
Does floating point break the bound?No draw leaves the band, even with coordinates at ±30\pm30Table 20

Two results are less favorable to us and we report them as such: on DrawBench (200 prompts), RATIO's recall is the highest, but its differences to DiverseGRPO and GRPO-Guard lie inside the paired intervals; and DiverseGRPO, whose diversity reward acts within prompt groups, is the closest control when recall is measured within each prompt. The paper now says this in Section 5.1 and Appendix G.

All new experiments re-examine claims already in the submission; none adds a method or a new claim. The main text stays within 10 pages, the title is unchanged, and the abstract changed only where Reviewer 4dXQ found undefined terms. We also corrected transcription errors in a few table cells.

We reply to each review separately below and would be glad to clarify anything further during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.