acceptodds
Under review as a conference paper at ICLR 2027

Learning to Generate in the Right Order

Abstract

We introduce PONDER, a method that learns in which order an image generator should fill in content and which of its earlier predictions to revise. Masked generative models construct a sample by repeatedly filling in subsets of missing variables. The order of these partial predictions determines the context available in subsequent steps and therefore affects the quality of the final sample. Common strategies are to choose the order randomly or based on local confidence estimates. Conditioned on the partially generated sample and the generator’s features, our policy chooses which positions to fill next and/or which previously filled positions to mask for later regeneration, a mechanism we call rethinking. In this way, the generation process is able to adapt to the evolving context and also to revise earlier predictions. We optimize the policy using group-relative policy gradients, with rewards that measure constraint satisfaction, human preference, or distribution quality. For the latter, we introduce FIDgrad, an image-level reward derived from the (first-order) change in Fréchet Inception Distance caused by adding a generated sample to the current feature distribution. With the generators frozen, learned ordering reduces MAR-B ImageNet FID from 4.09 to 2.78 and raises the accuracy on hard visual Sudokus from 3.9% to 56.6%, surpassing uncertainty-based greedy ordering (38.7%). Rethinking further improves these results to 2.70 FID and 96.7% accuracy, while increasing the rethinking budget of a fixed policy yields continued improvements at inference time.

Primary area:
generative models
Keywords:
Image Generation, Reinforcement Learning, Order Prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3DkgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PONDER trains a policy to choose which positions a frozen masked generator should fill or remove for later regeneration. Training uses terminal rewards and group-normalized policy gradients.

Critical strengths

1 The problem in this paper is clear, and paper is well motivated.

2 Freezing the generator clarifies PONDER’s contribution: Better inference decisions improve results without changing the generator.

3 Strong Sudoku results and ablations show that both generation order and learned revision locations are important

Critical weaknesses

1 typo: line 129 and 65 miss references.

2 The overall accuracy in the Sudoku input ablation is about 96.49% (table 5), while it is 98.2% (table1) in the main experiment. What settings are different between them?

3 Limited discussion of related work. The paper does not discuss its connection to InCoDe [1], which also learns an adaptive order for selecting informative semantic queries or image patches.

[1] InCoDe: Interpretable Compressed Descriptions For Image Generation

Questions for authors

See weaknesses

2 comments

Thank you for your constructive feedback. We incorporated it into our manuscript. If our answers are satisfying, we kindly ask to consider raising your score.

Q1. Missing references (lines 65 and 129)

Thank you, both are fixed in the revision: line 65 now cites GLM-Image and line 129 Hong et al. (ICLR 2026).

Q2. Sudoku input ablation (Table 5, 96.49%) vs. main table (Table 1, 98.2%)

Thank you for pointing out this inconsistency. The experiments in Table 5 were comparing sudoku policies after only 25k training iterations, whereas the policy in our main table was trained for 50k steps. We will update Table 5 with the numbers after 50k training steps.

Q3. Relation to InCoDe

Thank you for the pointer. We added a short discussion in the revised related work section. Both InCoDe and PONDER make a sequence of sample dependent choices for image generation, but for different purposes. InCoDe compresses an existing image into a short, human-readable description. From a fixed set of yes or no questions about the image (e.g., “is there a window?”), it learns which question to ask next, choosing at each step the one whose answer which is most informative about the remaining answers. A fine-tuned diffusion model then generates images that match these answers. The order thus decides which information about a given image to observe. The answers are read off that image or set by a user, and are not revised. PONDER has no image to observe. It decides in which order a frozen generator creates the image, and since the generator’s outputs can be wrong, it also learns which already generated patches to discard and regenerate. It is trained with reinforcement learning on the quality of the finished image (or the correctness of a Sudoku), not on information gain.

Thank you for your clarification and for addressing my concerns.

Reviewer_mqFwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PONDER, a policy that chooses which positions a frozen continuous-token image generator should fill and which previously generated positions to mask for later regeneration. It trains only the position-selection policy using group-relative policy gradients and terminal rewards. A second contribution, FIDgrad, derives a per-image reward from the first-order influence of a sample on population FID. Experiments cover visual Sudoku, class-conditional ImageNet generation with MAR-B, and text-to-image generation with MaskGen-KL-XL. Learned ordering improves performance without revision; a fixed policy can also use larger revision budgets at inference. The strongest result is hard-Sudoku accuracy, which increases from 56.6% with learned ordering alone to 96.7% with learned revision.

Critical strengths
  • The idea of learning order is interesting, espiaclly given that the image does not have a order.
  • Results on Sudoku directly support learning which predictions to revisit, beyond obtaining additional samples. On hard Sudoku, learned ordering reaches 56.6% accuracy versus 38.7% for greedy SRM.
  • The method improves frozen image generators without their output likelihoods. FIDgrad provides a principled distributional training signal: MAR-B FID improves from 4.09 to 2.78 without revision.
Critical weaknesses
  • The improvement on ImageNet is limited. Even with FID provided as the reward signal, it only optimizes FID from 4 to 2.78, which is a marginal improvement.
  • I would argue that a more convincing and impactful experiment than the marginal FID improvement from learned order at inference time in mask diffusion / MaskGIT would be the following: building an AR image generator on top of such a learned order — does it achieve better results than with a raster order or other orderings?
Questions for authors

See Weaknesses.

3 comments

Thank you for your constructive feedback. We incorporated it into our manuscript. If our answers are satisfying, we kindly ask to consider raising your score.

Q1 Marginal Gains:

We respectfully disagree that the FID gain from 4.08 to 2.78 is marginal. We also check that we don’t over-optimize on the FID reward: we observe significant improvements in FDr-5, which is not used during training. For reference, in the MAR paper (Li et al., 2024, without CFG), scaling the generator from MAR-B (208M parameters) to MAR-L (479M) improves FID by 0.88 (3.48 → 2.60), and scaling to MAR-H (943M) by 1.13 (→ 2.35). PONDER's improvement of 1.31 comes from a 43M-parameter policy and keeps the generator frozen.

Q2 Finetune MAR on policy orders:

Thank you for the suggestion. Since PONDER's orderings are sample dependent and adaptive depending on the generated content, there is no static learned order on which to train a raster-style AR model. The closest experiment is to collect orders in which the policy would generate ImageNet images without actually letting the generator fill in the patches, but let the ground truth be filled in (teacher forced). That way we can collect orders for ground truth images on which we can finetune MAR-B for 12.5k steps (20 epochs) with MAR's training recipe. We evaluate at R=0. Training on PONDER's orders improves FDr-5 from 13.28 to 12.70 and IS from 206.55 to 207.88 and FID marginally from 2.78 to 2.75. We think that this could be an interesting research direction for future work. The orders are adaptive to the generated content and depend on the learned generator. Therefore, learning an optimized order and training a generator on those orders at the same time will be a challenging problem to explore.

Thank you for the response. I acknowledge that, without using FID as a signal, going from 4 to 2.78 is a notable improvement.

However, this paper does use FID as a signal, and I would expect a larger improvement to show the effectiveness of the method.

Therefore, I maintain my original rating, but I lean more toward borderline now.

Thank you for the follow-up. Two clarifications for the discussion:

  1. PONDER only chooses at each step where the frozen MAR-B generates and revises. The generator weights don't change. Our FIDgrad reward turns the distributional FID metric into a per-sample reward, which lets us minimize it with RL and reduces FID by 32% (4.09 → 2.78). However, no reward can push the generator beyond what it can generate. As mentioned above the FID gains from optimizing the generation orders are larger than scaling MAR-B up to MAR-L or MAR-H. If you have a specific reference point in mind for the expected improvement, we would appreciate knowing it.
  2. Furthermore, PONDER is not an ImageNet specific method. We show a general framework on how to learn adaptive orders that optimize a chosen metric, evaluated on three tasks. It is interesting and applicable to other domains with unordered sequences as well, for example human motion diffusion (ActionPlan, Nazarenus et al. 2026). For text-to-image generation with MaskGen-KL-XL, PONDER improves HPSv2 from 24.5 to 26.5. On hard visual Sudoku, learned revision reaches 96.7% accuracy, compared with 38.7% for the ordering with SRM (Wewer et al., 2025). Setting ImageNet aside, the Sudoku result, which you highlight as the strongest, and the MaskGen result stand on their own to showcase the effectiveness of the method. We would therefore appreciate it if you would reconsider your score.
Reviewer_UjwcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PONDER, which learns which locations a frozen generator should fill or revisit. A separate policy observes generator features and the noise associated with the next fill, and is trained using terminal rewards. For image generation, the paper derives FIDgrad, a per-sample reward based on the first-order influence on FID. Experiments cover Sudoku, ImageNet, and text-to-image generation. The strongest evidence comes from the large Sudoku gains and the matched-budget comparison against random rethinking. The image results and evaluations outside the training-reward representation also support the approach, although the additional value and cost of rethinking on natural images need clearer quantification.

Critical strengths
  1. A clean position-policy interface for continuous generators. PONDER optimizes discrete position and rethinking actions while keeping the content generator frozen, avoiding the need to evaluate the likelihood of diffusion-head outputs. The FIDgrad derivation provides a principled route from a distribution-level image metric to a per-sample training signal. This combination is a useful methodological contribution.

  2. The main evidence includes a strong rethinking control and external metrics. In Table 1, learned rethinking improves hard Sudoku accuracy from 62.7% to 96.7% relative to random rethinking at the same removal budget. On ImageNet, MAR-B improves from FID 4.09 to 2.78 without rethinking, and FDr-5 evaluates representations outside the training FID space. These results support the value of the learned policy beyond simply allocating more generation steps or optimizing one representation.

Critical weaknesses
  1. The practical overhead is insufficiently quantified. Table 6 reports a 43.334M-parameter policy for the 207.925M-parameter MAR-B generator, while Appendix A.5 increases the number of MaskGen rounds from 32 to 96 at R=32. Parameter counts do not directly determine latency, but the paper does not provide sufficient runtime evidence for a minor-overhead claim. Training cost, measured inference latency, and quality-versus-time curves are important for judging when the learned policy is useful.

  2. The smaller natural-image rethinking gain and its source remain uncertain. In Table 2, learned versus random rethinking changes FID from 2.76 to 2.70 and FDr-5 from 12.68 to 12.47; evaluation seeds do not measure policy-training variation. In addition, the policy observes the noise of the next fill, allowing selection based on prospective randomness as well as context. Table 5 appropriately studies features-only policies, but the corresponding ablation is missing for the central MAR-B/FIDgrad setting. This limits attribution of the image gains, although it does not invalidate noise-conditioned scheduling as a useful method.

Questions for authors
  1. Cost clarification: What are the measured training and inference costs, and how does quality change with wall-clock time as the rethinking budget increases? Does the advantage persist against a simple alternative at the same runtime?
  2. Targeted validation: For MAR-B/FIDgrad, how much improvement remains without next-fill noise as a policy input? Is the incremental benefit of learned over random rethinking stable across policy-training runs? These checks would clarify the interpretation and practical strength of the image results.
2 comments

Thank you for your constructive feedback. We incorporated it into our manuscript. If our answers are satisfying, we kindly ask to consider raising your score.

Q1a Latency

We measured runtimes on one RTX 3090 Ti and will add the full tables to the appendix. Our policy is not optimized for inference speed and different design choices can speed up runtime significantly. For example our MaskGen policy only predicts one action per iteration following the fully sequential design of the generator. This means that each additional rethinking step currently needs 1 removal and 1 re-generation action and therefore runtime increases linear with R, which could still be heavily reduced with further optimization. Our MAR-B and Sudoku policies allow parallelization. Our MAR-B policy outputs 7 actions per iteration and with the parallelization schedule we used in the paper (see A.2), we obtain roughly constant speed up to R=8, because we do not have to run more generator rounds. For sudoku the latency stays relatively constant up to R=40.

MAR-B (ImageNet)

PolicyRRoundsms/sample ↓FID ↓
Random0642634.09
Random0722954.08
Random0853474.02
Random32763093.80
PONDER0642932.78
PONDER8642922.74
PONDER32763452.70

MaskGen-KL-XL

PolicyRRoundsms/sample ↓HPSv2 ↑
Random03247224.50
Random3296139525.58
PONDER03247926.50
PONDER44059526.95
PONDER3296141127.80

MAR-Sudoku

PolicyRRoundsms/sample ↓Accuracy (%) ↑
Random0815.6850.5
PONDER0816.2080.2
PONDER9816.4098.2
PONDER40816.5098.7
PONDER12816913.8898.9

Q1b: Training cost

Our training cost is roughly 30% of the MAR-B pretraining cost (1.4×10²⁰ vs 4.5×10²⁰ FLOPs) and took about 115 hours on four RTX PRO 6000 GPUs. Most of the compute is spent on collecting the trajectories with the generator (MAR-B). Our implementation is not tuned for efficiency: each batch of rollouts is used for one on-policy update and every round is replayed for the gradient. Reusing rollouts across several clipped updates, replaying a subset of rounds or a smaller policy would reduce this cost.

Q2a: training seed variability/significance

Thank you for the suggestion. We continued training the reported policy for 1k updates with three seeds and evaluated each at R=32 with learned and with random removals. Learned rethinking improves FDr-5, which the reward never sees, on every seed (−0.19 ± 0.02, paired t-test p = 0.005). The FID difference (−0.02 ± 0.05, p = 0.60) is within seed variation, which we will state in the paper. We want to emphasize that we already obtained large gains by optimizing the generation order (4.09 → 2.78) and further improvements through rethinking might be harder to obtain on this specific generator and reward. On Sudoku and MaskGen, learned rethinking clearly outperforms random rethinking at the same budget (hard-Sudoku accuracy 96.7% vs. 62.7%, HPSv2 26.95 vs. 26.40).

Q2b Noise conditioning

We are retraining the MAR-B/FIDgrad policy without noise input and will add it to Table 5. Meanwhile, we evaluated the reported policy at R=0 with independent noise as its input, while the generator keeps its own noise:

Policy noise input (R=0)FID ↓FDr-5 ↓IS ↑
Generator's noise2.7813.28206.55
Independent noise4.0913.72179.25
Random order4.0913.69181.22

With mismatched noise, the gain over random order disappears, so the policy uses the noise to choose the next patch. It is still order selection: each position's noise is drawn once and never redrawn. The policy therefore still only predicts the order, but is better informed about the change of the image. Since the policy never saw mismatched noise in training, this potentially overstates the noise's role.

Thank you for your response. I have no further questions. Best of luck!

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. They found the problem well motivated (3Dkg) and the idea interesting (mqFw), and highlighted that the policy improves a frozen generator (Ujwc, mqFw, 3Dkg), the principled FIDgrad reward (Ujwc, mqFw), the strong Sudoku results (Ujwc, mqFw, 3Dkg), and the image results, including evaluation outside the training reward (Ujwc).

Reviewer Ujwc asked about training and runtime cost, seed stability, and the role of the policy's noise input.

  • PONDER keeps its advantage at matched runtime (FID 2.78 vs. 4.08).
  • Learned rethinking improves the held-out FDr-5 on every seed (p = 0.005); the FID difference is within seed variation.
  • The policy uses the next-fill noise to choose positions but still only selects the order.

→ The reviewer confirmed that the concerns were addressed.

Reviewer 3Dkg asked about missing references, Table 5 vs. Table 1, and InCoDe. We fixed the references, updated Table 5 to 50k training steps as in Table 1, and added a discussion of InCoDe.

→ The reviewer confirmed that the concerns were addressed.

Reviewer mqFw questioned the size of the FID gain given the FID reward, and suggested training an AR generator on learned orders.

  • With the generator frozen, the FID gain (4.09 → 2.78) exceeds MAR's gains from scaling to MAR-L or MAR-H.
  • The reviewer acknowledged that without FID as a signal this is "a notable improvement" and now leans "more toward borderline".
  • Across tasks: hard Sudoku 3.9% → 96.7%; MaskGen ImageReward 0.44 → 0.68 (not used in training).
  • Fine-tuning MAR-B on PONDER's orders further improves FDr-5 (13.28 → 12.70).

We'd like to emphasise again that PONDER is a general framework to predict generation order that is applicable to a wide range of domains, as demonstrated by our three different setups.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.