- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper revisits the advantage of confidence ranking over random positions in masked image generation. The authors extend the "implicit cooling" explanation to text-to-image models with CFG and establish a temperature rule: ranking helps only at temperatures above the random-position optimum. Validated on four models, same-temperature comparisons show ranking gains of up to 51.8 FID, but after each sampler calibrates its own temperature, the FID advantage is not significant in three model families, and the released ranking is actually worse at the best temperature (MMaDA 14.1). The paper also proposes a saturation-aware finite-gate approximation for recommending ranking strength, and points out that position scoring can change effective guidance (COND probe and Proposition 1). The conclusion is that evaluating sampling rules requires temperature-calibrated baselines and guidance controls.
Critical strengths
Rigorous evaluation design: The rule was developed on aMUSEd-256, after which settings, metrics, interval methods and decision rules were frozen before testing on other models; the MMaDA predictions were made before the results; tuning and evaluation use different prompts, paired bootstrap, matched per-step commit counts (removing the effect of BF16 ties), and a values-only control (+0.60 and −1.61 FID, both not significant), which separates "better positions vs. different values" fairly cleanly. "Random-position baselines that need temperature calibration" is a valuable corrective for the community. Theory and experiments corroborate each other, and negative results are reported honestly: The finite-gate approximation clearly beats the power form under dense commits; COND's effective guidance has an FP64 controlled verification and a +7.02 FID effect in full generation; the authors also explicitly acknowledge that the recommendation rule did not select the best strength, that plain guidance tuning is stronger, and that sampler equivalence is not established.
Critical weaknesses
The core conclusion rests on "not significant ≠ equivalent" and a limited temperature grid. The intervals in Table 2 are not narrow (Meissonic g=3: [−2.43, 0.89]; MMaDA: [−1.75, 1.38], [−2.85, 0.15]), so they neither prove no difference nor exclude real gains on the order of 1–2 FID; the authors also state that sampler equivalence is not established. s* is approximated from only a few tested temperatures, and both L1–L3 and "no gain after calibration" depend on that approximation; the prompts of the two MMaDA evaluations (MMaDA† and MMaDA∗) overlap; LaViDa-O is evaluated only on alignment metrics, and few-step settings lack independently calibrated random-position temperatures (acknowledged by the authors in the Limitations). "Ranking has no value" should therefore be restricted to "the tested models, released step counts, and the FID metric".
Questions for authors
[1] Can you redo the calibrated comparison with a finer or continuously optimized temperature search, and run an equivalence test (TOST) with a pre-specified margin (e.g., ±1 FID)? [2] The evaluation prompts of MMaDA† and MMaDA∗ overlap; can you provide results on a fully independent third evaluation set? [3] Under few-step settings (which the paper says lack independently calibrated random-position temperatures), does "no gain after calibration" still hold? [4] Beyond FID, what are the calibrated comparison results on GenEval, CLIP, or human-preference metrics? [5] How large is the deviation of the finite-gate "fixed threshold" assumption on LaViDa-O and MMaDA? Why does it recommend r=4 on MMaDA when the best is r=16?
2 comments
Follow-up to Reviewer GGWL
Dear Reviewer GGWL, thank you again for your detailed review. As the discussion period ends soon, we would be grateful to know whether our response addresses your questions. In brief, Table 2 now reports one-sided upper limits instead of "not significant", the TOST with your ±1 FID margin establishes equivalence on aMUSEd and MaskGIT, and a fully independent MMaDA set favors random positions (Appendix G.1). We would be glad to clarify any remaining point.
Response to Reviewer GGWL
Thank you for the detailed review, and for describing the evaluation design as rigorous and the calibrated random baseline as "a valuable corrective for the community". Following your suggestion, the revision restricts our conclusions to the tested models, released step counts and FID. In place of "not significant", Table 2 now reports one-sided 95% upper limits of ranking's gain, which are below 2 FID in all 12 held-out evaluations and below 1 FID in nine.
Q1. Finer temperature search and TOST. We have done both. With finer temperature grids, little FID gain remains. The TOST, with your ±1 FID margin and the criterion (90% interval inside the margin) fixed before generating new images, establishes equivalence on aMUSEd and MaskGIT at all four step counts (Appendix G.1, Table 19). Intervals in Q1 are 90%.
Q2. A fully independent MMaDA set. We added a third set of 1,400 COCO prompts that share no image with any tuning or earlier evaluation set, and fixed the arms and statistics of Table 2 before generation. Random positions have the lower FID. Ranking's gain is −1.32 with 95% interval [−3.16, −0.28], and even its one-sided 95% upper limit, −0.46, is negative. The MMaDA conclusion therefore does not rest on overlapping prompts. On the same images CLIP-L slightly favors ranking (see Q4).
Q3. Few-step settings. For aMUSEd and MaskGIT, yes: every interval lies below 1.4 FID. On MMaDA the intervals are wider. With both samplers tuned on separate prompts and evaluated on new ones (Appendix G.3), aMUSEd gives −0.33 [−1.68, 1.08] and +0.36 [−0.67, 1.35] at 4 and 8 steps, and MaskGIT at 4 steps is equivalent within your margin (Q1). On MMaDA at 8 steps the gain is +1.34 [−1.41, 3.45]. The largest gain, +2.96 [0.19, 4.90], came from a noise-matched ranking arm that we added after seeing this result, and fell to +0.89 [−0.72, 2.46] on 1,400 new prompts.
Q4. GenEval, CLIP and human preference. After calibration on FID, we detect no advantage for ranking on GenEval or a learned human-preference score (PickScore), and ranking keeps a small text-alignment advantage on CLIP-L (Table 29).
§7 now recommends calibrating the baseline on the target metric.
Q5. Fixed-threshold deviation and the r = 4 pick.
How large is the deviation? It is much smaller than the power form's. The table gives the mean total variation between the finite-gate law and Monte Carlo estimates of the committed-value distribution, with the power form's error in parentheses (Appendix G.6):
On LaViDa-O at a = 1.5, over committed fractions of 0.7–45%, it is at most 0.0004 (power form 0.001–0.14).
Why r = 4 when the best is r = 16? The pick comes from the entropy-matching step, not from the local law. Entropy matching placed r = 4 and r = 16 at matched temperatures 0.95 and 1.05, almost equally close to s* = 1.0, and picked r = 4 (§4.5). The resulting gap is 1.2 FID with an interval that includes zero, and the broader prediction fixed before evaluation, that the best r is 4 or 16, held. In the five tests fixed before generation, our recommendation is never farther from the best than the power form's, so we present it as a starting point for a strength search.