- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies coverage loss during reward alignment of text-to-image generators. It proposes preserving every measurable image event within fixed probability factors of the base distribution. With a frozen generator, this guarantee follows from a bounded density ratio in initial-noise space. The proposed method, RATIO, combines Gaussian-preserving conditional rotations with monotone transformations in Gaussian-CDF coordinates, using a shared coefficient budget to obtain dimension-independent bounds. The paper also characterizes the optimal reward under these constraints and supports differentiable and black-box reward optimization. Experiments cover three generators and three rewards, including matched-reward coverage comparisons, event-level tests, architectural ablations, and human evaluations.
Critical strengths
-
A meaningful guarantee enforced by construction. The paper clearly distinguishes average divergence control from multiplicative protection of individual events. The main technical contribution is the tractable transform that enforces this stronger requirement: conditional rotations preserve the Gaussian reference, while CDF transformations and a shared coefficient budget control the density ratio throughout training. The composition argument correctly handles non-Gaussian intermediate distributions. I did not identify a substantive error in the central exact-arithmetic proofs.
-
Substantial empirical support for the reward–coverage trade-off. The evaluation goes beyond reporting reward improvements. At the matched PickScore gain, RATIO achieves 85.7% recall versus 69.2–80.6% for the external controls. The Affine+KL comparison is particularly informative because it uses the same conditioning interface and nearly matches the mean noise KL, yet exhibits substantially more event-bound violations. Cross-generator experiments, black-box reward training, and rotation ablations further support the usefulness of the construction.
Critical weaknesses
-
The experiments only partially validate the motivating claim about rare, within-prompt alternatives. PRDC recall pools features across prompts, so it does not directly measure conditional coverage. Moreover, only 1,085 of the 5,051 events are deletion-detectable at the reported sample sizes (Table 9); the appendix acknowledges that rare-event deletions are largely undetectable. The tests also provide per-event rather than family-wide error control. Thus, zero rejections should not be interpreted as strong empirical verification of rare-event preservation, although this limitation does not invalidate the theorem.
-
The matched-reward comparison does not fully establish the achievable performance frontier. Baseline metrics are linearly interpolated between relatively coarse post-training strength settings, whereas RATIO uses separately trained budget settings. Interpolated PRDC recall need not equal the recall of an actual sampler at the target reward, and scaling an already trained update may underperform training directly for that operating point. Uncertainty for these comparisons is also missing, which matters where the reported advantages are small.
Questions for authors
-
Comparison validation: Can you evaluate actual baseline settings at RATIO’s target reward and provide paired uncertainty estimates? For a representative strong baseline, does training or selecting checkpoints directly for this reward level change the coverage comparison?
-
Event-test interpretation: How many baseline violations remain after an appropriate multiple-testing correction? Can you provide a targeted evaluation of low-probability, prompt-valid alternatives with sufficient samples to detect their deletion, and report within-prompt coverage separately from pooled PRDC recall?
-
Implementation clarification: Appendix D.4 specifies BF16 generator/conditioner computations and FP64 CDF/density calculations. Which precision is used for rotations and intermediate transformed latents? Can you report forward–inverse consistency and density-ratio stress tests, particularly in the tails, to quantify the gap between the exact-arithmetic guarantee and the implemented sampler?
4 comments
Response to Reviewer jjHy (2/3): multiplicity, rare prompt-valid events, and within-prompt coverage
(2/3) This part answers W1 and Q2. We agree with the reading in your review: zero rejections at 128 draws are a falsification check, and the guarantee for rare events rests on the theorem. Section 5.2 now says this in those terms. We then added the three analyses you asked for, so that the empirical side no longer depends on that caveat alone.
How many violations survive a multiple-testing correction?
Many, for every control. Appendix G.3 (Table 17) treats each method's 10,102 directional tests (two per event, including the ones that cannot reject at these sample sizes) as one family. Holm at family-wise level 5% is valid under any dependence; we add Benjamini–Hochberg at FDR 5% as a descriptive count, since events of one prompt share images.
| Method | Per test (Table 2) | Holm | of which prompt-valid deletions |
|---|---|---|---|
| DRaFT | 343 | 101 | 68 |
| GRPO-Guard | 180 | 17 | 8 |
| HyperNoise | 350 | 81 | 58 |
| Affine+KL | 386 | 114 | 76 |
| RATIO / | 0 / 0 | 0 / 0 | 0 / 0 |
All eight controls keep between 17 and 114 certified violations. The comparison you found most informative survives: at the same mean noise KL, Affine+KL keeps 76 certified prompt-valid deletions and RATIO none. Holm's first threshold is , which only large drops of frequent events reach (removing every occurrence of a base-frequency-0.5 event gives , of a 0.9 event ). The corrected counts therefore measure what these sample sizes can certify, not how many violations occur; we state this in Appendix G.3.
A targeted test of rare, prompt-valid alternatives
Appendix G.4 (Table 18) does what you proposed. From the existing 256-image base pool we select the prompt-valid events seen in 2–38 images (about 0.8–15%), merging events identical to their prompt's correctness event: 241 events from 162 prompts, none detectable with 128 draws. We then draw fresh pools of 4,096 base and 2,048 method images per prompt and test deletion only. At these sizes deletion is detectable above a base frequency of about 1.7%, which holds for 191 events. The test covers RATIO and three controls chosen for their role: DRaFT (same training exposure as RATIO), GRPO-Guard (strongest control), and Affine+KL (equal mean noise KL).
These three controls delete 12–52 of these events per test and 8–43 after Holm within the family; Affine+KL deletes the most (52), DRaFT 47, GRPO-Guard the fewest (12). RATIO deletes none. Because the selection uses only the old pool and every test uses new images, the family is fixed before any test outcome is seen.
Within-prompt coverage, reported separately from pooled recall
Appendix G.5 (Table 19) computes PRDC recall separately for each prompt and averages over prompts. It reuses the evaluation images of Table 15 (ten per prompt from each of three runs, so 30 per prompt) and compares them with 30 independent base images per prompt, with the same and DINOv2 features.
| Within-prompt recall | Pick-a-Pic | DrawBench |
|---|---|---|
| Base, independent pool | 90.8 | 91.5 |
| RATIO | 80.2 | 85.9 |
| Best control (DiverseGRPO) | 72.8 | 84.6 |
| GRPO-Guard | 72.5 | 81.0 |
| DRaFT | 57.6 | 70.1 |
Pooling did hide part of the loss: every control's gap to RATIO is larger within prompts than pooled (for GRPO-Guard on Pick-a-Pic, 4.3 pooled vs 7.7 within). DiverseGRPO widens least, which fits how it works: its diversity reward is computed within groups of images of one prompt, so it becomes the closest control under this metric (gap 7.4 and 1.3). Within-prompt recall matches the per-prompt form of our bound, so we now present it next to the pooled metric.
Paper changes: Section 5.2 (blue), limitation (iv) in Section 6, Appendix G.3–G.5, Tables 17–19.
Response to Reviewer jjHy (3/3): precision of the implementation and tail stress tests
(3/3) This part answers Q3, which asks how close the implemented sampler is to the exact-arithmetic guarantee. It is a question we should have answered in the submission.
Which precision is used for rotations and intermediate latents?
Appendix D.4 now lists every computation. The generator and the conditioner run in BF16. The Givens angles returned by the conditioner are cast to FP64; , , and the rotations are evaluated in FP64; intermediate latents between blocks are stored in FP32. The coefficient normalization (Eq. 4), the Gaussian CDF and its inverse, the sine maps, and all log-density terms are computed in FP64, with tail-stable log-CDF and log-survival evaluations and a residual tolerance of for the inverse.
This split matters for the guarantee. The bound depends on three things: the normalization, the CDF maps, and the orthogonality of each rotation. The first two are computed in FP64 on whatever raw outputs the BF16 conditioner returns, so conditioner rounding changes which bounded transform is applied, not whether its coefficients satisfy . The rotations are orthogonal to FP64 round-off. What remains is the FP32 storage of latents between blocks.
Forward–inverse consistency and density-ratio stress tests
Appendix G.6 (Table 20) runs each of the three RATIO runs () on 10,000 standard Gaussian inputs, and on stress inputs where a random 1% of the 65,536 coordinates are set to , , or (30,000 draws per regime).
| Input | Max round-trip error () | Max log-density mismatch ( nats) | Realized (band ) | Outside band |
|---|---|---|---|---|
| Standard | 3.6 | 2.9 | -1.08 / 1.37 | 0 |
| 6.3 | 3.8 | -1.15 / 1.44 | 0 | |
| 13.8 | 6.1 | -1.17 / 1.46 | 0 | |
| 32.6 | 6.4 | -1.17 / 1.46 | 0 |
No draw leaves the band in any regime, and the realized log ratio stays at least 0.54 nats inside the nearer edge. Two built-in checks pass: the sample mean of under standard draws is 0.2884 nats, within Monte Carlo error of the measured mean noise KL (0.2896), as requires; and the summed rotation log-determinants stay below . The tails behave as the construction predicts. Beyond about , equals its endpoint value to within (its derivative vanishes at both endpoints), so a coordinate pushed to or contributes the fixed factor or , which is inside the per-coordinate band; that is why the and rows agree. The gap between the exact-arithmetic statement and the implemented sampler is several orders of magnitude below the width of the band. Limitation (iii) in Section 6 states that the guarantee is proved in exact arithmetic and checked numerically.
For the AC discussion, in short
- The comparison survives direct evaluation, training for the target, and paired intervals on Pick-a-Pic; on DrawBench we now claim only that RATIO is not exceeded.
- Under Holm, every control keeps certified violations and RATIO none; on rare prompt-valid events with adequate power, the controls delete 12–52 and RATIO none.
- The implemented transform stays inside its band under tail stress.
These analyses follow your suggestions closely. If they settle your soundness concerns, we would be grateful if you would consider raising your score and supporting the paper in the discussion. Your review made the empirical side of the paper much stronger, and we thank you for it.
concerns resolved
Thanks to the authors' rebuttal. My concerns have been resolved. I will keep my score as 3, weak acceptance.
Response to Reviewer jjHy (1/3): matched-gain comparison at directly evaluated strengths
(1/3) We thank you for a review that engages with exactly the parts of the paper we care most about. Your summary of the strengths is how we would put it ourselves: the paper separates average divergence control from multiplicative protection of each event, and the contribution is a transform that enforces the stronger requirement at every training step. It meant a lot to us that you checked the composition argument for non-Gaussian intermediate latents and found no substantive error in the exact-arithmetic proofs. Your questions are precise and testable, so we answered each with an experiment or a re-analysis; all are in the new Appendix G (blue), and this reply covers Q1. Parts 2 and 3 cover the event tests and the numerics.
Q1 / W2. Are the matched-gain results an artifact of interpolation?
No; the comparison holds when every control is evaluated at its own matched strength, with the caveat on DrawBench below. Appendix G.1 (Table 15) evaluates each control directly at the strength s∗ that attains RATIO's PickScore, found by five bisection steps on PickScore alone, using initial noises disjoint from the reported images. Recall and CLIPScore never enter the choice. For the two knobs that are separate fits (λ for HyperNoise-L2 and Affine+KL), we retrain at λ∗=1.19. Intervals are paired 95% bootstrap intervals that resample prompts (same indices for both methods) and each method's three training runs, 10,000 times.
The remaining four rows are in Table 15. Two observations. First, RSA-FT and DiffusionNFT sit almost exactly at an evaluated strength (w=0.0010 and 0.9941), so their direct values must match the interpolation, and they do (+0.0 and -0.1 points); this checks the protocol. Second, interpolation did understate the controls slightly (mean +0.4, at most +0.9 points), most where the target lies well inside a bracket; HyperNoise, entered as the envelope of its two knobs, lands 0.3 and 1.7 points below it. The smallest Pick-a-Pic gap shrinks from 5.1 to 4.3 points, and every Pick-a-Pic interval excludes zero.
You were right that uncertainty matters where differences are small. On DrawBench (200 prompts) RATIO still has the highest recall, but its differences to DiverseGRPO (0.7, CI [-1.2, 2.5]) and GRPO-Guard (1.5, CI [-0.5, 3.4]) are inside the intervals. Section 5.1 now says so, and we no longer count them as advantages.
Does training a control directly for the target reward change the picture?
It narrows the gap but does not close it (Appendix G.2, Table 16). For DRaFT, which shares RATIO's 16,000-image exposure, and GRPO-Guard, the strongest control, we select the first checkpoint whose validation gain reaches 0.596; for DRaFT we also retrain with a smaller learning rate tuned to end at that gain. Selection uses validation captions only; three runs per variant.
Your intuition was right that a dedicated run beats a scaled update: the reduced learning rate gains DRaFT about three points. Checkpoint selection changes little for either control, because early checkpoints of a full-strength run already concentrate on the same images. A GRPO-Guard retrained with a larger KL coefficient needs a full reinforcement-learning run per seed, so we covered the strongest control through checkpoint selection, which answers the same question; we will add the retrained variant to the final version. The retention advantage does not come from how the controls were scaled.
Paper changes: Section 5 Protocol paragraph and Section 5.1 (blue); Appendix G.1–G.2, Tables 15–16.