- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Routing Policy Optimization (RPO), which introduces stochastic exploration through attention routing rather than noisy denoising transitions. Bernoulli masks derived from native query–key affinities define discrete actions, and two rollouts sharing the same generation context provide paired reward differences. Conditional mask-history likelihoods update parameters shared with dense attention. Experiments on Stable Diffusion 3.5 and Wan2.2 report improved dense-inference quality and lower training cost than sampler-space RL baselines.
Critical strengths
- The attention-routing policy offers an interesting way to optimize deterministic visual generators with black-box rewards, without adding a policy head or changing inference.
- Improvements persist after routing is removed, including gains across external DrawBench metrics and several semantic VBench dimensions. This supports the practical value of temporary routing for dense-model alignment.
Critical weaknesses
- The evaluation backends are not fully aligned. Appendix D reports that the same initial checkpoint scores 4.92 versus 5.53 under different implementations. Subtracting these initial scores does not establish that backend effects remain constant during training, making the convergence comparison difficult to interpret.
- The efficiency comparison uses different group sizes and training schedules, including a longer DanceGRPO run. The reported GPU-hour ratios therefore do not clearly establish an equivalent-quality efficiency advantage. Explicit checkpoint-selection rules and training budgets would make this comparison more convincing.
- The routing mechanism is insufficiently isolated. No ablations separate affinity-conditioned routing, matched pairing, and optimization safeguards. Appendix E also reports changing mask density and dense-gain fade under some schedules, leaving unclear which components support reliable transfer to dense inference.
Questions for authors
- Can the authors provide absolute scores through one shared evaluation pipeline and clarify the training budgets underlying Table 1?
- Are there existing ablations separating the effects of routing, matched pairing, mask-density changes, and optimizer safeguards?
3 comments
W2: The efficiency comparison uses different group sizes and training schedules, including a longer DanceGRPO run. The reported GPU-hour ratios therefore do not clearly establish an equivalent-quality efficiency advantage. Explicit checkpoint-selection rules and training budgets would make this comparison more convincing.
Q1 (budgets): Can the authors clarify the training budgets underlying Table 1?
Response to W2 and Q1 (budgets). We thank the reviewer for this suggestion. The revision now states the training budgets and the checkpoint-selection rule explicitly.
Training budgets (new Tables 4 and 5 in the manuscript).
| RPO | DanceGRPO | Flow-GRPO | |
|---|---|---|---|
| Group size, prompts per step | G = 2, 64 | G = 8, 64 | G = 8, 64 |
| Rollout videos per step | 128 | 512 | 512 |
| Replayed transitions per video | 1 | 2 | 2 |
| Launched schedule | 50 steps | 120 steps | 120 steps |
| Checkpoint (Table 1 in the manuscript) | step 50 | step 50 | step 40 |
| Rollout videos to that checkpoint | 6,400 | 25,600 | 20,480 |
| GPU-hours to that checkpoint (measured) | 25.4 | 102.3 | 75.4 |
GPU-hours. All three costs are measured from the training logs and include startup, validation, and checkpoint saves. The baselines roll out with the optimized vLLM-Omni backend. RPO needs FlexAttention for its routed step, which vLLM-Omni does not support, so it rolls out with a batched diffusers backend. The saving comes from the counts above: 4× fewer rollout videos and half the replay per video (Table 5 in the manuscript).
Checkpoint-selection rule.
- Within each run. Each run is evaluated at the end of the common 50-step window. Flow-GRPO saved checkpoints only at steps 40, 80, and 120, so step 40 is its last checkpoint inside that window. No checkpoint was chosen by its evaluation score. For example, Table 1 in the manuscript uses RPO at step 50 (82.00), not step 20 (82.03).
- Across runs. We selected the RPO configuration from the sweep of Table 7 in the manuscript. The reported run is the better of two seeds (+2.86 and +2.40). The baselines are single runs.
- Later baseline checkpoints. The revision adds Flow-GRPO's later checkpoints to the shared-renderer comparison and to VBench, so the comparison no longer stops at the 50-step window.
Equivalent-quality comparison. We agree that comparing costs at different quality levels is not enough. New Table 9 in the manuscript compares the budget each method needs to reach the same absolute reward under one shared renderer. RPO reaches its level at step 20.
| Method | First checkpoint at or above 7.87 (RPO, step 20) | Rollout videos |
|---|---|---|
| RPO | step 20 (7.87) | 2,560 |
| Flow-GRPO | step 80 (8.42). Step 40 is at 6.63. | 40,960 |
| DanceGRPO | not reached (5.32 at step 50) | – |
Flow-GRPO crosses RPO's level between its saved checkpoints at 8× and 16× RPO's rollout videos.
Final level. With a larger budget, Flow-GRPO surpasses RPO in HPSv3 (8.42 at step 80 vs. 7.87). This higher reward does not translate into better videos on VBench. Flow-GRPO only matches the RPO total at steps 80 and 120 (82.02 and 81.99 vs. 82.00), after 6.4 and 9.6 times as many rollout videos, and by step 120 its dynamic degree falls to 33.89, below the base model's 50.56. This pattern suggests that part of the extra reward comes from over-optimizing HPSv3 rather than from better generation. We therefore state the claim as the rollout budget needed for a given reward.
W3: The routing mechanism is insufficiently isolated. No ablations separate affinity-conditioned routing, matched pairing, and optimization safeguards. Appendix E also reports changing mask density and dense-gain fade under some schedules, leaving unclear which components support reliable transfer to dense inference.
Q2: Are there existing ablations separating the effects of routing, matched pairing, mask-density changes, and optimizer safeguards?
Response to W3 and Q2. We thank the reviewer for this question. We ran one ablation per component on Wan2.2 (Table 12 in the manuscript, App. F). All runs use seed 42 and change one setting of the reference configuration. Each entry is the dense-probe gain over that run's own step zero. The row labels match the paragraphs below.
| Row | Run | Removes | Keep ratio | Step 15 | Step 50 |
|---|---|---|---|---|---|
| (a) | RPO, reported run | – | 0.48 → 0.47 | +2.96 | +2.86 |
| (b) | RPO, permuted route logits | affinity pattern | 0.50 | +2.67 | +2.65 |
| (c) | RPO, independent content seeds | matched pairing | – | +2.44 | +1.88 |
| (d) | RPO, single fresh minibatch | stale update, ratio clipping | 0.41, then ≈0.60 | −7.05 | −13.81 |
| (e) | Flow-GRPO, G = 2, shared noise | routing (SDE instead) | – | +0.67 | +1.28 |
(b) Affinity pattern. Within each routing unit, the route logits are shuffled across candidate blocks. The density stays at 0.50, and only the affinity pattern is lost. The run plateaus near +2.6, about 0.3 below (a), so most of the gain does not depend on the affinity pattern.
(c) Matched pairing. With independent content seeds, RPO still learns, reaching +2.73 at step 20 and +1.88 at step 50. This variant also changes the probe noise (step zero 4.26 rather than 4.92), which is why every row reports gains. A per-pair gradient probe also finds no variance reduction from matching (App. F). Matching still makes each reward difference attributable to the route (Table 8 in the manuscript), and RPO learns without it, although with single runs we cannot attribute the step-50 gap between (a) and (c) to matching.
(d) Stale update and ratio clipping. With a single fresh minibatch per step, training diverges. The keep ratio first falls to 0.41 and then rises to about 0.60. This change also halves the number of optimizer steps. Row (d) therefore shows that the two-minibatch update is needed in this configuration, but it does not isolate the effect of clipping.
Advantage processing (Table 7 in the manuscript). The existing sweep, with two seeds per arm, covers this safeguard. With winsorization, all 8 runs improve, with a mean of +2.29. Without it, 3 of 8 collapse or drift negative.
(e) Routing vs. sampler-space exploration. At RPO's rollout budget, the sampler-space baseline learns more slowly (+0.67 at step 15) and ends at +1.28 at step 50, below the RPO runs (+1.88 to +2.86).
Mask density. In (a), the realized keep ratio is 0.48 at step 1 and 0.47 at step 50, against the nominal κ = 0.5, so density does not drift. Among the runs for which we recorded the keep ratio, a large density change appears only in the diverging run (d).
Dense-gain fade. The reported run (a) shows no fade. Its gain stays between +2.86 and +2.98 at every probe from step 15 to step 50, and its VBench totals at steps 20 and 50 agree (82.03 and 82.00). Both training seeds of the reference configuration end with a positive gain (+2.86 and +2.40). App. E lists the fade as an open limitation and no longer asserts a mechanism.
Which components support reliable transfer. These runs support the following conclusions:
- The gain does not require the affinity pattern (b) or matched seeds (c). Each conclusion rests on one run.
- The ratio-clipped two-minibatch update is required in this configuration (d).
- Winsorized advantages prevent collapse in the sweep (Table 7 in the manuscript).
- Both training seeds of the reference configuration improve the dense model at step 50, by +2.86 and +2.40.
We hope these ablations clarify the role of each component, and we would be glad to provide further details.
We thank Reviewer gCBz for the thoughtful review. The new results are in Tables 9 to 12 in the manuscript (changes in blue). Part 1 answers W1 and the shared-pipeline request in Q1, Part 2 answers W2 and the budgets in Q1, and Part 3 answers W3 and Q2.
Response to W1 and Q1 (shared pipeline). We agree with the reviewer on this point. The three original runs start at 4.92 (RPO), 5.53 (DanceGRPO), and 4.71 (Flow-GRPO). They differ in backend, precision, and validation seeding, so subtracting the initial score cannot rule out effects that change during training. The revision removes the constant-offset explanation (App. D) and adds two absolute evaluations through one shared pipeline.
Absolute reward through one renderer (new Table 9 in the manuscript). We re-rendered saved checkpoints of all three runs on the 61 probe prompts. One renderer, one HPSv3 scorer, and identical prompt-keyed noise are used throughout, so all runs share one step-zero score.
The ranking is unchanged under the shared renderer. RPO reaches its level within 20 steps. Flow-GRPO needs between 8 and 16 times as many rollout videos to reach it and then continues higher. DanceGRPO stays at the base level. The per-run baselines of Figure 4 in the manuscript had slightly favored Flow-GRPO, whose original gains are about 0.7 higher than under the shared renderer, against 0.3 to 0.4 for RPO.
Absolute video quality through one pipeline (Tables 6 and 10 in the manuscript). VBench renders all Wan2.2 checkpoints with one prompt set and shared initial noise. All 16 dimensions are now reported.
Within 50 steps, both RPO training seeds exceed the base model on the total, while DanceGRPO@50 and Flow-GRPO@40 fall below it. The seed-43 checkpoint comes from a separate training run of that configuration (App. F). A paired bootstrap over prompts gives +0.43 [+0.02, +0.85] for RPO's quality score and +0.39 [−0.01, +0.79] for the total. Flow-GRPO ties RPO's total at steps 80 and 120, after 6.4 and 9.6 times as many rollout videos. At that point it has a higher semantic score, and its dynamic degree drops to 33.89 at step 120. RPO's largest gains are object class (+4.40) and human action (+4.80), and its largest loss is appearance style (−3.14).