- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies step-level credit assignment in diffusion reinforcement learning, focusing on how terminal preference feedback can be attributed to individual denoising transitions. The authors argue that uniform allocation of terminal rewards across denoising steps fails to account for their unequal contributions and interactions. To address this issue, the paper proposes a counterfactual attribution framework that uses ODE/SDE interventions and Shapley values to estimate each transition's contribution to the final preference outcome. A two-stage trajectory evaluator is then trained to predict these credits from completed trajectories, which are used to reweight the terminal advantages during policy optimization. The paper further provides an error-sensitivity analysis and evaluates the proposed method on Stable Diffusion 3.5 Medium, reporting improvements in generation quality and credit prediction accuracy through baseline comparisons and ablation studies.
Critical strengths
S1. The paper proposes a concrete framework that integrates counterfactual credit assignment, learned trajectory evaluation, and step-wise advantage reweighting during policy optimization. It also includes controlled experiments to examine the contributions of the framework's individual components.
Critical weaknesses
W1. The paper lacks clarity and logical coherence in its presentation of the motivation and proposed methodology. The introduction introduces several technical concepts without sufficient background explanation or intuitive motivation, making it difficult to understand the problem being addressed. In particular, the role of Shapley-based credit assignment and its connection to terminal preference feedback are not clearly established before the formal methodology is presented. Although Sections 3 and 4 provide mathematical definitions of the counterfactual intervention game, credit prediction, and advantage reweighting, the connections between these components are not explained sufficiently clearly. A more intuitive overview of the complete framework, followed by a step-by-step explanation of its key components, would help readers better understand the proposed approach.
W2. The experimental evidence supporting the paper's core conclusions and contributions is insufficient. First, the baselines in Table 2 are not adequately specified. In particular, the architectures, input features, and training configurations of the "Endpoint-feature-matched" and "Diffusion-native temporal" baselines remain unclear, making it difficult to assess the fairness and reproducibility of the comparisons. Second, while the paper reports improvements on GenEval and provides qualitative examples of object counting, attribute binding, and spatial relationships, additional evaluation on compositional generation benchmarks such as T2I-CompBench++ would provide stronger evidence of these improvements across different compositional capabilities. Third, the experiments are limited to a single image generator, Stable Diffusion 3.5 Medium (SD3.5-M), leaving the method's generalizability across diffusion models insufficiently examined. Finally, the main numerical results are reported without standard deviations or confidence intervals, making it difficult to assess the stability and statistical reliability of the reported improvements.
W3. The significance of the paper's claimed contributions is not sufficiently established. The paper presents three main contributions: a replayable ODE/SDE counterfactual intervention game, a two-stage evaluator for step-level credit prediction, and a bounded advantage allocation mechanism with error-sensitivity analysis. However, the additional methodological insights and practical advantages of these components over existing approaches are not sufficiently established. In particular, the advantages of the proposed counterfactual attribution over alternative step-level credit assignment methods, the necessity of the two-stage evaluator, and the practical implications of the error-sensitivity analysis require further justification.
Questions for authors
Q1. What are the key methodological differences between the proposed approach and TurningPoint-GRPO, which also studies step-level credit assignment and the long-term effects of intermediate denoising decisions? What additional advantages does the proposed Shapley-based attribution mechanism offer, and why is TurningPoint-GRPO not discussed or compared against?
Q2. How stable are the reported improvements across different random seeds? Can the authors report results from multiple independent training runs, including standard deviations or confidence intervals?
Q3. How exactly are the baselines in Table 2 constructed? In particular, what are the architectures, input features, and training settings of the "Endpoint-feature-matched" and "Diffusion-native temporal" baselines? Please clarify whether these baselines are adapted from existing methods or newly constructed, and provide explicit implementation details to support the interpretation and reproducibility of the comparisons.
Q4. How does the proposed method perform on T2I-CompBench++, particularly on tasks involving object counting, attribute binding, and spatial relationships? Can the authors provide additional experimental results on this benchmark to further validate the reported improvements?
13 comments
Response to Reviewer kDVW (Part 2/5)
3. Necessity of Two-stage Learning and Post-rollout Advantage Reweighting (W3)
(1) Latent Preference Perception and Completed Trajectory Evaluation
The two stages use complementary supervision with markedly different construction costs: inexpensive, scalable noisy-latent preference pairs for perceptual adaptation, followed by more expensive, generator-specific counterfactual trajectories for step-credit learning.
Stage I — Latent Preference Perception. We encode existing clean image-preference pairs with the generator's VAE and apply forward noise according to its native noise schedule, reusing their pairwise preference labels at different noise levels. This does not require denoising rollouts or counterfactual coalitions, allowing low-cost construction of large-scale noisy-latent preference data. We train the reward-side latent encoder and adapter to map noisy latents and timestep information into VLM-compatible tokens, while keeping the original preference-VLM backbone and native output head frozen. The resulting latent RM is denoted .
Stage II — Completed Trajectory Evaluation. Unlike direct forward noising, this stage uses actual denoising trajectories sampled from the current generator. The replayable ODE/SDE intervention game (Sec. 3.1), downstream coalition replay, and frozen clean preference judge provide exact Shapley credits and terminal-gain labels for completed mixed/all-ODE trajectory pairs. These labels cost more to construct because they require additional counterfactual completions and preference evaluations, but capture generator-specific denoising dynamics and transition interactions. Starting from , Stage II further trains , , and language LoRA to obtain the trajectory RM , jointly predicting step credits and terminal gain from the completed pair's prefix/window/suffix evidence, timestep and ODE/SDE role information, and final images.
The complementarity is therefore explicit: Stage I learns noisy-latent preference perception from abundant low-cost data; Stage II learns step attribution from expensive, high-information counterfactual trajectories, improving the utilization of Stage II labels.
| Variant | Step MAE ↓ | Spearman ↑ | Largest-absolute-step accuracy ↑ |
|---|---|---|---|
| Stage II only | 0.028 | 0.51 | 62.2% |
| Stage I + II | 0.019 | 0.70 | 74.6% |
Under the same Stage II architecture and training configuration, Stage I reduces step MAE by 32.1%, improves Spearman from 0.51 to 0.70, and increases largest-absolute-step accuracy from 62.2% to 74.6%. The label-efficiency study (Fig. 5(b), Table 6) further shows that Stage I + II first reaches step MAE ≤ 0.025 with 5k labeled Stage II windows, whereas Stage II only requires 20k at the tested budget points. Thus, Stage I improves accuracy and reduces the amount of expensive Stage II supervision needed to reach the same error threshold; this is a label-efficiency comparison, not a claim that the two-stage pipeline always has lower total compute.
(2) Post-rollout Advantage Reweighting and Error Sensitivity
As defined in Sec. 4, the predicted terminal gain is group-standardized into the terminal group advantage; predicted step credits determine nonnegative, bounded weights allocating that advantage across SDE-window transitions. This preserves the scalar terminal advantage's sign at each step and its mean across the window, while allowing different update magnitudes. Eq. (11) bounds the allocation-weight discrepancy caused by credit-prediction error under fixed terminal advantage, temperature, and allocation parameters. The credit-rule ablations and learned-versus-exact results provide complementary evidence of the resulting generation quality.
Response to Reviewer kDVW (Part 3/5)
4. Baseline Construction, Fairness, and Reproducibility (W2, Q3)
We clarify the implementation of the Endpoint-feature-matched and Diffusion-native temporal controls specifically raised by the reviewer, as well as the other predictors in Table 2.
All Table 2 variants are initialized from the same image-preference-trained Qwen3.5-9B VLM, and use the same completed mixed/all-ODE final-image pair, offline exact Shapley targets, data split, and evaluation protocol. They differ primarily in the process evidence provided, how it is fused with the base VLM, and which additional modules are trained. Thus, they share an initial reward model and prediction target, not necessarily identical final parameters or architectures.
Table K2. Construction and attribution accuracy of Table 2 baselines.
| Variant | Additional process evidence | Architecture / fusion | Step MAE ↓ | Spearman ↑ |
|---|---|---|---|---|
| Final-image-only | None | Native VLM visual encoder | 0.034 | 0.28 |
| Endpoint-feature-matched | Intermediate ODE completions | VLM visual encoder + fusion | 0.031 | 0.36 |
| Diffusion-native temporal | SDE-window features | Feature projector + VLM | 0.027 | 0.49 |
| StitchVM-style + Shapley | 3 latent-pair + 1 terminal score | DiT–VLM stitching + 4-to-3 MLP | 0.025 | 0.55 |
| Ours Stage II only | Completed trajectory pair | + + VLM | 0.028 | 0.51 |
| Ours Stage I + II | Completed trajectory pair | Same trajectory RM; Stage I initialization | 0.019 | 0.70 |
Final-image-only (constructed control). Uses only the native VLM image representations of the two final images to predict the step/terminal array, without intermediate process evidence.
Endpoint-feature-matched (constructed control). Performs deterministic ODE completion from each intermediate SDE state in the attribution window. The resulting completed images are encoded by the original VLM visual encoder and fused with the mixed/all-ODE final-image representations to predict the same step/terminal array.
Diffusion-native temporal (constructed control). Extracts diffusion-process features directly from the SDE window, maps them into the VLM feature space through a trainable projector, and fuses them with the same final-image features to predict step credits. Thus, Endpoint-feature-matched tests image-space intermediate completion evidence, whereas Diffusion-native temporal tests direct diffusion-native features.
StitchVM-style + Shapley (StitchVM-inspired constructed control). We adapt the structural DiT–VLM stitching idea of StitchVM [3] to Shapley attribution prediction; this is not a direct reproduction of StitchVM's original value-prediction task. A stitching adapter inserts DiT features into a selected intermediate VLM layer, and downstream VLM layers are fine-tuned for latent-pair preference evaluation while retaining the original final-image preference path. We then freeze the adapted RM and extract three SDE/ODE latent-pair preference scores plus one terminal image-pair score. Only a 4-to-3 MLP is trained to regress the three Shapley credits from these four scalar scores; the terminal prediction comes from the preserved final-image path. This tests scalar intermediate reward compression followed by Shapley regression against our direct trajectory-evidence modeling.
Ours Stage II only vs. Stage I + II (method ablation). Both use the identical trajectory RM architecture and Stage II Shapley supervision, taking completed mixed/all-ODE prefix/window/suffix states, pre/post-state evidence, timestep/role information, and final images as input. The sole difference is Stage I latent preference perception, which initializes and before Stage II.
All variants are evaluated on the same step/terminal prediction task. Ours Stage I + II achieves the lowest step MAE (0.019) and highest within-window Spearman (0.70). We will add detailed baseline configurations and module-level training specifications to the revised Appendix.
Response to Reviewer kDVW (Part 1/5)
We thank the reviewer for the careful assessment and constructive feedback. We address the distinctions from TurningPoint-GRPO, the value of Shapley attribution and two-stage training, baseline reproducibility, and experimental reliability with additional controlled experiments and implementation details.
1. Contributions and Comparison with TurningPoint-GRPO (W1, W3, Q1)
As explained in C1–C2 of our General Response, we extend a terminal image-preference RM to jointly predict step credits and terminal gain from completed trajectories, while reducing repeated online process-reward computation. Two distinctions from TurningPoint-GRPO [1] are central.
(1) Counterfactual Shapley credits vs. turning-point aggregation. TurningPoint-GRPO evaluates ODE completions from intermediate states to obtain incremental rewards and aggregates long-term effects around turning points. Our replayable ODE/SDE intervention game (Sec. 3.1) instead fixes the policy snapshot and indexed noise tape, replays downstream trajectories under different intervention coalitions, and averages each transition's marginal contributions across coalitions using Shapley values [2]. This captures how a transition's contribution depends on which other stochastic transitions are retained.
(2) Online intermediate completions vs. learned trajectory evaluation. TurningPoint-GRPO uses multiple intermediate-state ODE completions and reward evaluations during online RL. Our trajectory RM (θ2), trained on offline exact Shapley labels, makes one autoregressive evaluation of the already collected completed mixed/all-ODE trajectory pair to predict the window's step credits and terminal gain, without additional online intermediate completions or coalition enumeration.
On SD3.5-M, ours attains 0.758 GenEval / 1.34 ImageReward, compared with 0.738 / 1.29 for TurningPoint-GRPO; end-to-end cost to each method's own convergence decreases from 624 to 324 GPU-hours (−48.1%). The distinction therefore concerns both credit definition and amortized online feedback. We will clarify these differences in the related work, Introduction, and Figs. 1–2.
2. Value of Shapley Attribution and Learned Prediction (W3, Q1)
We compare Chronological Marginal, Leave-one-out, Exact Shapley, and learned Shapley credits while holding the generator, terminal evaluator, SDE window, post-rollout reweighting, and RL configuration fixed. Only credit construction changes. TurningPoint-GRPO is a full-system comparison, not a single-factor ablation.
Table K1. Effectiveness and efficiency of credit assignment methods.
Three-seed means on SD3.5-M. E2E GPU-hours measure each method's own convergence endpoint, not a shared GenEval target; TurningPoint-GRPO is evaluated as a full method.
(1) Coalition averaging. Chronological Marginal follows a fixed temporal ordering; Leave-one-out removes one transition from the full SDE window. Both use sample-specific counterfactual information but limited coalition contexts. Exact Shapley averages across all coalition contexts and achieves 0.762 GenEval, exceeding Chronological Marginal (0.731; +0.031) and Leave-one-out (0.724; +0.038). This supports coalition averaging under the same advantage reweighting rule.
(2) Amortized prediction. Whereas Exact Shapley requires repeated counterfactual replay, our trajectory RM predicts step credits and terminal gain from offline supervision. It achieves 0.758 vs. 0.762 GenEval, with E2E cost reduced from 1,128 to 324 GPU-hours (−71.3%). Complete online RL iterations take 57 s (ours) versus 192 s (Exact Shapley). The 118 GPU-hours for Stage I, coalition labeling, and Stage II are included in our reported E2E cost. Detailed cost accounting will be provided in the revision.
These controls support the value of coalition-averaged attribution targets and amortized trajectory evaluation.