acceptodds
Under review as a conference paper at ICLR 2027

From Preference to Process: Counterfactual Trajectory Reward Modeling for Diffusion RL

Abstract

Terminal preference rewards guide diffusion reinforcement learning, but do not specify how gains should be allocated across interacting denoising transitions. We define counterfactual trajectory credit by selectively replacing stochastic transitions with deterministic updates and replaying their downstream continuations under a fixed policy and indexed noise tape. A frozen preference judge compares the resulting outcomes against a shared deterministic reference, and Shapley values average each intervention's marginal contribution across combinations of retained transitions. To avoid enumerating these outcomes during policy updates, we first adapt clean preference perception to noisy latents, then train the evaluator on offline counterfactual labels to predict step credits and terminal gain from completed trajectory pairs. The predicted credits determine bounded, nonnegative weights on a terminal group advantage. On Stable Diffusion 3.5 Medium, learned allocation improves GenEval from to while retaining the same learned terminal evaluator. Latent initialization reduces step-credit mean absolute error from to , and permutation controls support assigning credits to their corresponding transitions. The results support learned counterfactual credit as useful supervision for allocating terminal feedback across denoising decisions.

Primary area:
generative models
Keywords:
Diffusion reinforcement learning, Counterfactual credit assignment, Shapley values, Trajectory reward modeling, Preference alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kDVWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies step-level credit assignment in diffusion reinforcement learning, focusing on how terminal preference feedback can be attributed to individual denoising transitions. The authors argue that uniform allocation of terminal rewards across denoising steps fails to account for their unequal contributions and interactions. To address this issue, the paper proposes a counterfactual attribution framework that uses ODE/SDE interventions and Shapley values to estimate each transition's contribution to the final preference outcome. A two-stage trajectory evaluator is then trained to predict these credits from completed trajectories, which are used to reweight the terminal advantages during policy optimization. The paper further provides an error-sensitivity analysis and evaluates the proposed method on Stable Diffusion 3.5 Medium, reporting improvements in generation quality and credit prediction accuracy through baseline comparisons and ablation studies.

Critical strengths

S1. The paper proposes a concrete framework that integrates counterfactual credit assignment, learned trajectory evaluation, and step-wise advantage reweighting during policy optimization. It also includes controlled experiments to examine the contributions of the framework's individual components.

Critical weaknesses

W1. The paper lacks clarity and logical coherence in its presentation of the motivation and proposed methodology. The introduction introduces several technical concepts without sufficient background explanation or intuitive motivation, making it difficult to understand the problem being addressed. In particular, the role of Shapley-based credit assignment and its connection to terminal preference feedback are not clearly established before the formal methodology is presented. Although Sections 3 and 4 provide mathematical definitions of the counterfactual intervention game, credit prediction, and advantage reweighting, the connections between these components are not explained sufficiently clearly. A more intuitive overview of the complete framework, followed by a step-by-step explanation of its key components, would help readers better understand the proposed approach.

W2. The experimental evidence supporting the paper's core conclusions and contributions is insufficient. First, the baselines in Table 2 are not adequately specified. In particular, the architectures, input features, and training configurations of the "Endpoint-feature-matched" and "Diffusion-native temporal" baselines remain unclear, making it difficult to assess the fairness and reproducibility of the comparisons. Second, while the paper reports improvements on GenEval and provides qualitative examples of object counting, attribute binding, and spatial relationships, additional evaluation on compositional generation benchmarks such as T2I-CompBench++ would provide stronger evidence of these improvements across different compositional capabilities. Third, the experiments are limited to a single image generator, Stable Diffusion 3.5 Medium (SD3.5-M), leaving the method's generalizability across diffusion models insufficiently examined. Finally, the main numerical results are reported without standard deviations or confidence intervals, making it difficult to assess the stability and statistical reliability of the reported improvements.

W3. The significance of the paper's claimed contributions is not sufficiently established. The paper presents three main contributions: a replayable ODE/SDE counterfactual intervention game, a two-stage evaluator for step-level credit prediction, and a bounded advantage allocation mechanism with error-sensitivity analysis. However, the additional methodological insights and practical advantages of these components over existing approaches are not sufficiently established. In particular, the advantages of the proposed counterfactual attribution over alternative step-level credit assignment methods, the necessity of the two-stage evaluator, and the practical implications of the error-sensitivity analysis require further justification.

Questions for authors

Q1. What are the key methodological differences between the proposed approach and TurningPoint-GRPO, which also studies step-level credit assignment and the long-term effects of intermediate denoising decisions? What additional advantages does the proposed Shapley-based attribution mechanism offer, and why is TurningPoint-GRPO not discussed or compared against?

Q2. How stable are the reported improvements across different random seeds? Can the authors report results from multiple independent training runs, including standard deviations or confidence intervals?

Q3. How exactly are the baselines in Table 2 constructed? In particular, what are the architectures, input features, and training settings of the "Endpoint-feature-matched" and "Diffusion-native temporal" baselines? Please clarify whether these baselines are adapted from existing methods or newly constructed, and provide explicit implementation details to support the interpretation and reproducibility of the comparisons.

Q4. How does the proposed method perform on T2I-CompBench++, particularly on tasks involving object counting, attribute binding, and spatial relationships? Can the authors provide additional experimental results on this benchmark to further validate the reported improvements?

13 comments
AuthorsOpenReview ↗

Response to Reviewer kDVW (Part 1/5)

We thank the reviewer for the careful assessment and constructive feedback. We address the distinctions from TurningPoint-GRPO, the value of Shapley attribution and two-stage training, baseline reproducibility, and experimental reliability with additional controlled experiments and implementation details.

1. Contributions and Comparison with TurningPoint-GRPO (W1, W3, Q1)

As explained in C1–C2 of our General Response, we extend a terminal image-preference RM to jointly predict step credits and terminal gain from completed trajectories, while reducing repeated online process-reward computation. Two distinctions from TurningPoint-GRPO [1] are central.

(1) Counterfactual Shapley credits vs. turning-point aggregation. TurningPoint-GRPO evaluates ODE completions from intermediate states to obtain incremental rewards and aggregates long-term effects around turning points. Our replayable ODE/SDE intervention game (Sec. 3.1) instead fixes the policy snapshot and indexed noise tape, replays downstream trajectories under different intervention coalitions, and averages each transition's marginal contributions across coalitions using Shapley values [2]. This captures how a transition's contribution depends on which other stochastic transitions are retained.

(2) Online intermediate completions vs. learned trajectory evaluation. TurningPoint-GRPO uses multiple intermediate-state ODE completions and reward evaluations during online RL. Our trajectory RM (θ2\theta_2), trained on offline exact Shapley labels, makes one autoregressive evaluation of the already collected completed mixed/all-ODE trajectory pair to predict the window's step credits and terminal gain, without additional online intermediate completions or coalition enumeration.

On SD3.5-M, ours attains 0.758 GenEval / 1.34 ImageReward, compared with 0.738 / 1.29 for TurningPoint-GRPO; end-to-end cost to each method's own convergence decreases from 624 to 324 GPU-hours (−48.1%). The distinction therefore concerns both credit definition and amortized online feedback. We will clarify these differences in the related work, Introduction, and Figs. 1–2.

2. Value of Shapley Attribution and Learned Prediction (W3, Q1)

We compare Chronological Marginal, Leave-one-out, Exact Shapley, and learned Shapley credits while holding the generator, terminal evaluator, SDE window, post-rollout reweighting, and RL configuration fixed. Only credit construction changes. TurningPoint-GRPO is a full-system comparison, not a single-factor ablation.

Table K1. Effectiveness and efficiency of credit assignment methods.

Credit rule / methodGenEval ↑ImageReward ↑E2E GPU-h ↓
Uniform (λ=0\lambda=0)0.6981.15726
Chronological Marginal0.7311.21604
Leave-one-out0.7241.30612
TurningPoint-GRPO0.7381.29624
Ours (Learned Shapley Credits)0.7581.34324
Exact Shapley0.7621.321128

Three-seed means on SD3.5-M. E2E GPU-hours measure each method's own convergence endpoint, not a shared GenEval target; TurningPoint-GRPO is evaluated as a full method.

(1) Coalition averaging. Chronological Marginal follows a fixed temporal ordering; Leave-one-out removes one transition from the full SDE window. Both use sample-specific counterfactual information but limited coalition contexts. Exact Shapley averages across all coalition contexts and achieves 0.762 GenEval, exceeding Chronological Marginal (0.731; +0.031) and Leave-one-out (0.724; +0.038). This supports coalition averaging under the same advantage reweighting rule.

(2) Amortized prediction. Whereas Exact Shapley requires repeated counterfactual replay, our trajectory RM predicts step credits and terminal gain from offline supervision. It achieves 0.758 vs. 0.762 GenEval, with E2E cost reduced from 1,128 to 324 GPU-hours (−71.3%). Complete online RL iterations take 57 s (ours) versus 192 s (Exact Shapley). The 118 GPU-hours for Stage I, coalition labeling, and Stage II are included in our reported E2E cost. Detailed cost accounting will be provided in the revision.

These controls support the value of coalition-averaged attribution targets and amortized trajectory evaluation.

AuthorsOpenReview ↗

Response to Reviewer kDVW (Part 2/5)

3. Necessity of Two-stage Learning and Post-rollout Advantage Reweighting (W3)

(1) Latent Preference Perception and Completed Trajectory Evaluation

The two stages use complementary supervision with markedly different construction costs: inexpensive, scalable noisy-latent preference pairs for perceptual adaptation, followed by more expensive, generator-specific counterfactual trajectories for step-credit learning.

Stage I — Latent Preference Perception. We encode existing clean image-preference pairs with the generator's VAE and apply forward noise according to its native noise schedule, reusing their pairwise preference labels at different noise levels. This does not require denoising rollouts or counterfactual coalitions, allowing low-cost construction of large-scale noisy-latent preference data. We train the reward-side latent encoder EGE_G and adapter AψA_\psi to map noisy latents and timestep information into VLM-compatible tokens, while keeping the original preference-VLM backbone and native output head frozen. The resulting latent RM is denoted θ1\theta_1.

Stage II — Completed Trajectory Evaluation. Unlike direct forward noising, this stage uses actual denoising trajectories sampled from the current generator. The replayable ODE/SDE intervention game (Sec. 3.1), downstream coalition replay, and frozen clean preference judge sωs_\omega provide exact Shapley credits and terminal-gain labels for completed mixed/all-ODE trajectory pairs. These labels cost more to construct because they require additional counterfactual completions and preference evaluations, but capture generator-specific denoising dynamics and transition interactions. Starting from θ1\theta_1, Stage II further trains EGE_G, AψA_\psi, and language LoRA to obtain the trajectory RM θ2\theta_2, jointly predicting step credits and terminal gain from the completed pair's prefix/window/suffix evidence, timestep and ODE/SDE role information, and final images.

The complementarity is therefore explicit: Stage I learns noisy-latent preference perception from abundant low-cost data; Stage II learns step attribution from expensive, high-information counterfactual trajectories, improving the utilization of Stage II labels.

VariantStep MAE ↓Spearman ↑Largest-absolute-step accuracy ↑
Stage II only0.0280.5162.2%
Stage I + II0.0190.7074.6%

Under the same Stage II architecture and training configuration, Stage I reduces step MAE by 32.1%, improves Spearman from 0.51 to 0.70, and increases largest-absolute-step accuracy from 62.2% to 74.6%. The label-efficiency study (Fig. 5(b), Table 6) further shows that Stage I + II first reaches step MAE ≤ 0.025 with 5k labeled Stage II windows, whereas Stage II only requires 20k at the tested budget points. Thus, Stage I improves accuracy and reduces the amount of expensive Stage II supervision needed to reach the same error threshold; this is a label-efficiency comparison, not a claim that the two-stage pipeline always has lower total compute.

(2) Post-rollout Advantage Reweighting and Error Sensitivity

As defined in Sec. 4, the predicted terminal gain rfinalr_{\mathrm{final}} is group-standardized into the terminal group advantage; predicted step credits r1:Mr_{1:M} determine nonnegative, bounded weights allocating that advantage across SDE-window transitions. This preserves the scalar terminal advantage's sign at each step and its mean across the window, while allowing different update magnitudes. Eq. (11) bounds the allocation-weight discrepancy caused by credit-prediction error under fixed terminal advantage, temperature, and allocation parameters. The credit-rule ablations and learned-versus-exact results provide complementary evidence of the resulting generation quality.

AuthorsOpenReview ↗

Response to Reviewer kDVW (Part 3/5)

4. Baseline Construction, Fairness, and Reproducibility (W2, Q3)

We clarify the implementation of the Endpoint-feature-matched and Diffusion-native temporal controls specifically raised by the reviewer, as well as the other predictors in Table 2.

All Table 2 variants are initialized from the same image-preference-trained Qwen3.5-9B VLM, and use the same completed mixed/all-ODE final-image pair, offline exact Shapley targets, data split, and evaluation protocol. They differ primarily in the process evidence provided, how it is fused with the base VLM, and which additional modules are trained. Thus, they share an initial reward model and prediction target, not necessarily identical final parameters or architectures.

Table K2. Construction and attribution accuracy of Table 2 baselines.

VariantAdditional process evidenceArchitecture / fusionStep MAE ↓Spearman ↑
Final-image-onlyNoneNative VLM visual encoder0.0340.28
Endpoint-feature-matchedIntermediate ODE completionsVLM visual encoder + fusion0.0310.36
Diffusion-native temporalSDE-window featuresFeature projector + VLM0.0270.49
StitchVM-style + Shapley3 latent-pair + 1 terminal scoreDiT–VLM stitching + 4-to-3 MLP0.0250.55
Ours Stage II onlyCompleted trajectory pairEGE_G + AψA_\psi + VLM0.0280.51
Ours Stage I + IICompleted trajectory pairSame trajectory RM; Stage I initialization0.0190.70

Final-image-only (constructed control). Uses only the native VLM image representations of the two final images to predict the step/terminal array, without intermediate process evidence.

Endpoint-feature-matched (constructed control). Performs deterministic ODE completion from each intermediate SDE state in the attribution window. The resulting completed images are encoded by the original VLM visual encoder EVE_V and fused with the mixed/all-ODE final-image representations to predict the same step/terminal array.

Diffusion-native temporal (constructed control). Extracts diffusion-process features directly from the SDE window, maps them into the VLM feature space through a trainable projector, and fuses them with the same final-image features to predict step credits. Thus, Endpoint-feature-matched tests image-space intermediate completion evidence, whereas Diffusion-native temporal tests direct diffusion-native features.

StitchVM-style + Shapley (StitchVM-inspired constructed control). We adapt the structural DiT–VLM stitching idea of StitchVM [3] to Shapley attribution prediction; this is not a direct reproduction of StitchVM's original value-prediction task. A stitching adapter inserts DiT features into a selected intermediate VLM layer, and downstream VLM layers are fine-tuned for latent-pair preference evaluation while retaining the original final-image preference path. We then freeze the adapted RM and extract three SDE/ODE latent-pair preference scores plus one terminal image-pair score. Only a 4-to-3 MLP is trained to regress the three Shapley credits from these four scalar scores; the terminal prediction comes from the preserved final-image path. This tests scalar intermediate reward compression followed by Shapley regression against our direct trajectory-evidence modeling.

Ours Stage II only vs. Stage I + II (method ablation). Both use the identical trajectory RM architecture and Stage II Shapley supervision, taking completed mixed/all-ODE prefix/window/suffix states, pre/post-state evidence, timestep/role information, and final images as input. The sole difference is Stage I latent preference perception, which initializes EGE_G and AψA_\psi before Stage II.

All variants are evaluated on the same step/terminal prediction task. Ours Stage I + II achieves the lowest step MAE (0.019) and highest within-window Spearman (0.70). We will add detailed baseline configurations and module-level training specifications to the revised Appendix.

Reviewer_nasXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work on preference learning through RL aims to address the problem of diffusion RL sharing the terminal advantage across many denoising steps, regardless of their contribution or opposition to the main generation. The approach is to replace certain stochastic transitions by deterministic ones and replay downstream states, then average each step's marginal contribution. The authors learn to predict the offline targets to avoid repetition during each actor update. Experiments support the benefits of this multi stage credit assignment method compared to the prior work.

Critical strengths
  1. Normally the models are trained with a sparse score at the end of the denoising chain, but this method aims to have better credit assignment for the intermediary steps.

  2. The paper is well written with a clear flow, and overall has a good experimental setting with various experiments showcasing the advantage and shortcoming of the method with respect to the prior work.

  3. The proposed technique to learn an extra module to circumvent the computational complexity of credit assignment as originally proposed is interesting and shows promise in some experiments.

Critical weaknesses
  1. I understand the logic behind using an ODE denoising instead to measure the credits, yet it is not clear in the paper why this choice will be unbiased. And why would the ODE be a good basis for such comparison. Comparisons with alternative replacement rules would help assess how strongly the resulting credits and performance gains depend on this design choice.

  2. In the second and third stage, the authors compute expensive Shapley credits for training examples, then train a separate model to predict those credits from much less information. The learned evaluator makes counterfactual attribution computationally practical however reliability depends on coverage of the credit training data. The reported learning curves support approximation within the tested setting yet robustness and refresh requirements as the actor evolves remain insufficiently characterized.

  3. Discussion of the limitations of the works is rather limited. For instance, the computational cost of having the seperate credit estimator trained, and deploying it is not clearly outlined. Similar comparisons are essential in the context of RL with repetitive reward and network evaluations in the loop.

Minor weakness

  1. Tables 1 2 3 report no confidence intervals or seed dispersion.

  2. Figures are overly complex for the methodology, consider working more on the presentation of figures to send the main message clearly, and remove unnecessary components that ad little to the main message of each figure.

Questions for authors

See weaknesses and rectify if possible.

7 comments
AuthorsOpenReview ↗

Response to Reviewer nasX (Part 1/4)

We thank the reviewer for the constructive feedback and for recognizing the motivation for improving process-level supervision in diffusion RL through counterfactual credit assignment, as well as the potential of a learned trajectory evaluator to reduce online attribution costs.

We address the choice of ODE reference, reliability of the trajectory RM under policy evolution, and offline/online computational costs through new replacement-rule controls, actor-checkpoint and reward-model-refresh evaluations, and end-to-end cost accounting. The main experiments have also been extended to three independent training seeds.

W1. ODE Reference and Alternative Replacement Rules

Why use an ODE reference? ODE is a natural deterministic counterpart in the flow-matching sampling setup, rather than an arbitrary replacement. Flow-GRPO [1] constructs an SDE from the flow-matching ODE, preserving the same marginal distributions under ideal continuous-time conditions while enabling stochastic exploration. MixGRPO [2] combines SDE steps in the optimized window with ODE steps outside it.

In our replayable ODE/SDE intervention game (Sec. 3.1), an ODE replacement defines the intervention, while the shared all-ODE outcome serves as the reference for preference evaluation. We fix the policy snapshot, initial noise, solver schedule, and indexed noise tape, and recompute every affected downstream state. This isolates each transition's contribution relative to deterministic progression under the same policy.

Role of the ODE baseline vs. statistical unbiasedness: Using ODE provides a clean, policy-consistent deterministic reference for the intervention game, while full-enumeration (or uniform permutation) Shapley ensures unbiased attribution of the defined game's preference gain. For this specified game, full-enumeration Shapley [3] exactly attributes the teacher-defined preference gain, and the uniformly sampled permutation estimator is unbiased for these Shapley credits (Appendix A.1). We thus distinguish the rationale for the reference choice from unbiased estimation of the defined Shapley target.

Alternative replacement rules. We further evaluate (i) Zero-noise SDE Drift, retaining the SDE drift correction but removing its stochastic innovation, and (ii) an Alternative ODE Solver, retaining ODE dynamics while changing numerical integration.

Table N1. Sensitivity to counterfactual replacement rules.

Replacement ruleCredit Spearman ↑Top-∣ϕ∣\lvert\phi\rvert agreement ↑Allocation discrepancy ↓GenEval ↑ImageReward ↑
Uniform (λ=0\lambda=0)———0.6981.15
Proposed ODE replacement1.00100%00.7581.34
Zero-noise SDE Drift0.8481%0.090.7441.30
Alternative ODE Solver0.9492%0.050.7521.31

Credit correlation and top-step agreement compare each rule's exact Shapley credits with those from the proposed ODE replacement. Allocation discrepancy compares the induced weights under the same terminal advantage (cf. Appendix A.2), not prediction error EwE_w. The proposed rule's agreement entries are self-comparisons. Generation metrics are three-seed means.

The alternative solver retains high credit agreement (Spearman 0.94; top-step agreement 92%) and reaches 0.752 GenEval, just 0.006 below the proposed reference. Zero-noise SDE Drift also improves upon uniform allocation (0.744 vs. 0.698 GenEval); the alternative solver achieves a +0.054 improvement over uniform. Thus, the proposed ODE replacement is a well-motivated default, and the observed benefit of counterfactual attribution persists under the tested alternative rules.

AuthorsOpenReview ↗

Response to Reviewer nasX (Part 2/4)

W2. Trajectory RM Reliability under Policy Evolution

We agree that a learned trajectory RM must generalize from offline counterfactual labels to evolving actor policies. Our evaluator does not uniquely reconstruct unobserved coalition outcomes from one observed trajectory pair. Instead, Stage I learns noisy-latent preference perception without Shapley labels, while Stage II learns statistical regularities from exact offline Shapley targets (Sec. 3.3 and Appendix A.3).

We therefore conducted three complementary studies: reliability across actor checkpoints, refresh-label-budget ablations, and end-to-end reward-model refresh strategies.

(1) Attribution Reliability across Actor Checkpoints

We evaluate the frozen trajectory RM on held-out counterfactual windows from actor checkpoints at 0%, 25%, 50%, 75%, and 100% of RL progress. At each noninitial checkpoint, we also independently refresh the reward model using 500 newly sampled windows from that policy, without accumulating updates from previous checkpoints.

Table N2. Attribution reliability across actor checkpoints.

Actor checkpointFrozen Step MAE ↓Refreshed Step MAE ↓Frozen Spearman ↑Refreshed Spearman ↑Frozen E~w\widetilde E_w ↓
Initial (0%)0.019—0.70—0.08
25% RL0.0200.0200.650.710.10
50% RL0.0220.0200.620.680.10
75% RL0.0250.0210.580.680.14
Final (100%)0.0260.0210.570.630.18

Three-seed means. Each refresh is checkpoint-specific. E~w=Ew/M\widetilde E_w=E_w/M is the window-length-normalized allocation discrepancy between predicted and exact Shapley credits, based on Eq. (20) and Appendix B.4.

As the actor evolves, frozen-RM Step MAE rises from 0.019 to 0.026, within-window Spearman decreases from 0.70 to 0.57, and E~w\widetilde E_w increases from 0.08 to 0.18. Thus, policy evolution produces measurable attribution degradation. At the final checkpoint, independent refresh reduces MAE from 0.026 to 0.021 and raises Spearman from 0.57 to 0.63, demonstrating that modest new-policy supervision can mitigate this degradation.

(2) Refresh Label-budget Ablation

To examine label requirements and distinguish policy coverage from simply adding training examples, we fix the final actor evaluation and vary the new-policy labeling budget. As a distribution control, we also resample 1,000 windows from the old policy and construct their exact Shapley labels, rather than reusing stored labels.

Table N3. Refresh label-budget ablation.

Adaptation dataStep MAE ↓Spearman ↑Top-∣ϕ∣\lvert\phi\rvert acc. ↑E~w\widetilde E_w ↓Extra GPU-h ↓
Frozen RM0.0260.5769%0.180
250 new-policy windows0.0240.6073%0.1313
500 new-policy windows0.0210.6374%0.0727
1,000 new-policy windows0.0220.6575%0.1053
2,000 new-policy windows0.0210.6977%0.06110
1,000 old-policy windows0.0270.6170%0.1452

Three-seed means; identical E~w\widetilde E_w definition throughout. Extra GPU-h includes the additional counterfactual labeling and RM-adaptation costs.

AuthorsOpenReview ↗

Response to Reviewer nasX (Part 3/4)

W2 (continued): Label Efficiency and End-to-end Refresh

500 new-policy windows substantially improve attribution fidelity: Step MAE falls from 0.026 to 0.021 (−19.2%), within-window Spearman rises from 0.57 to 0.63, and top-∣ϕ∣\lvert\phi\rvert accuracy rises from 69% to 74%. More importantly, normalized allocation discrepancy E~w\widetilde E_w decreases from 0.18 to 0.07 (−61.1%), indicating that the resulting allocation weights are closer to those induced by exact Shapley credits. The additional cost is 27 GPU-hours.

The benefit is not merely due to more labels. With the same 1,000-window budget and nearly identical costs, new-policy labels achieve 0.022 MAE / 0.65 Spearman / 0.10 E~w\widetilde E_w at 53 GPU-h, compared with 0.027 / 0.61 / 0.14 at 52 GPU-h for resampled old-policy labels. Larger budgets further improve some ranking metrics (2,000 windows: Spearman 0.69, top-step accuracy 77%) but cost 110 GPU-h. Hence, 500 windows provide a favorable tested accuracy–compute trade-off; improvements across metrics are not strictly monotonic in label count.

(3) End-to-end Reward Model Refresh Strategies

We next test whether improved credit prediction translates into generation gains and whether these gains justify the additional compute.

Table N4. End-to-end reward-model refresh strategies.

StrategyGenEval ↑ImageReward ↑E2E GPU-h ↓
Frozen trajectory RM0.7581.34324
Periodic refresh0.7691.37572
Drift-triggered refresh0.7661.39524
Matched extra-data (old policy)0.7621.35379

E2E GPU-h includes data acquisition, RM adaptation, and actor training up to each strategy's own endpoint; it is distinct from the incremental labeling cost in Table N3. The trigger criterion, checking frequency, and associated costs will be specified in the revision.

Periodic refresh improves GenEval from 0.758 to 0.769 (+0.011), whereas drift-triggered refresh reaches 0.766 GenEval and 1.39 ImageReward. Relative to the frozen RM, their additional end-to-end costs are 248 and 200 GPU-h, respectively. The old-policy extra-data control achieves only +0.004 GenEval at an additional 55 GPU-h. Periodic refresh yields the highest mean GenEval, whereas drift-triggered refresh yields the highest ImageReward at lower cost.

Our default remains the frozen trajectory RM: it already achieves 0.758 GenEval / 1.34 ImageReward at 324 GPU-h, without updating the evaluator during actor optimization. Refresh can provide additional gains, but incurs substantial acquisition and adaptation costs. Efficiently deciding when and how often to refresh the evaluator is therefore a promising extension.

The checkpoint-specific refreshes in Table N2 are independent prediction diagnostics, whereas Table N4 evaluates refresh strategies during complete actor RL.

Reviewer_ZKFMOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper defines counterfactual credit for diffusion transitions by replacing selected stochastic updates with deterministic ones and replaying their continuations. Shapley values allocate the resulting terminal preference gain across transitions. A two-stage evaluator learns to predict these credits, which then reweight a terminal group advantage during policy optimization.

Critical strengths
  1. The intervention is precisely defined, including downstream replay and a fixed indexed noise tape.
  2. The experiments distinguish credit prediction, terminal evaluation, and advantage allocation rather than treating them as interchangeable.
Critical weaknesses
  1. Experiments use a single random seed and provide no uncertainty estimates for the main comparisons. The 53.3% direct human preference against TreeGRPO is difficult to assess without confidence intervals and clearer annotation details.
  2. Permuting credits tests transition assignment, but does not establish that sample-specific counterfactual attribution is needed. A fixed timestep schedule or average per-step credit could also benefit from correct temporal assignment.
Questions for authors
  1. How does learned credit compare with fixed timestep weights or average per-step credits under the same terminal evaluator and allocation rule?
  2. Can you report multiple training seeds and confidence intervals for the main results, particularly the direct human comparison with TreeGRPO?
5 comments
AuthorsOpenReview ↗

Response to Reviewer ZKFM (Part 1/3): Sample-specific Attribution

We sincerely thank the reviewer for the careful assessment and constructive suggestions. We appreciate the recognition of our rigorous ODE/SDE counterfactual intervention, downstream replay, and fixed indexed noise tape, as well as the experimental separation of credit prediction, terminal evaluation, and advantage allocation.

To address the questions about the additional value of sample-specific attribution and statistical reliability, we conducted controlled ablations of fixed timestep priors, average credits, and cross-instance credit swapping, alongside three-seed analyses and a detailed human evaluation.

W2 / Q1. Necessity of Sample-specific Counterfactual Attribution

We introduce three complementary controls to assess the benefits of sample-specific attribution over fixed timestep weighting and average per-step credit: fixed temporal priors, data-driven average credits, and cross-instance credit swapping.

All variants use the same generator, terminal evaluator, SDE window, advantage-allocation framework, and RL training configuration; only the credit construction or corresponding weights differ. Generation metrics are averaged over three independent training seeds.

Table Z1. Effectiveness and efficiency of credit allocation strategies.

Credit AllocationGenEval ↑ImageReward ↑E2E GPU-h ↓
Uniform (λ=0)0.6981.15726
Fixed Early-heavy0.7041.18721
Fixed Middle-heavy0.6921.12787
Fixed Late-heavy0.6881.16744
Mean Shapley0.7241.19702
Mean Actual Weights0.7121.24712
Cross-instance Swap0.6721.08914
Ours (Learned Credit)0.7581.34324
Exact Shapley (Oracle)0.7621.321128

E2E GPU-hours measure the cost to each method's own convergence endpoint, rather than the cost to reach a common GenEval threshold.

(1) Sample-specific Attribution vs. Fixed and Average Temporal Weights

We compare three types of trajectory-independent temporal weighting:

  • Fixed Timestep Priors: Prespecified early-, middle-, and late-heavy weighting schedules.
  • Mean Shapley: Average Shapley credit at each temporal position, computed from offline training data.
  • Mean Actual Weights: Average realized advantage-allocation weight at each temporal position, also computed from offline training data.

Global temporal importance is informative, but sample-specific credits provide further gains. Among the fixed priors, Early-heavy performs best at 0.704 GenEval, versus 0.758 (+0.054) for ours.

Mean Shapley improves GenEval from 0.698 (Uniform) to 0.724, demonstrating the utility of data-driven temporal importance. Our method further achieves 0.758 (+0.034), while ImageReward increases from 1.19 to 1.34 (+0.15). Likewise, Mean Actual Weights achieves 0.712 GenEval, 0.046 below ours.

These results indicate that our gains arise not only from global timestep importance but also from trajectory-conditioned credit allocation.

(2) Importance of Trajectory–Credit Correspondence

We introduce Cross-instance Credit Swapping to test whether correctly matching credits to their trajectories matters beyond the distribution of nonuniform weights. We exchange predicted credit vectors across samples under matched sampling settings, preserving their within-vector temporal structure and overall value distribution while disrupting their correspondence with the original trajectories.

Ours achieves 0.758 GenEval / 1.34 ImageReward, compared with 0.672 / 1.08 for Cross-instance Swap. Notably, swapping also falls 0.026 GenEval below Uniform (0.698).

Thus, matching credits to the corresponding trajectory is important: nonuniform temporal weighting with incorrect sample correspondence can even impair optimization. Together with the fixed-prior and average-credit controls, this supports the added value of sample-specific counterfactual attribution.

(3) Learned Attribution vs. Exact Shapley

Exact Shapley reaches 0.762 GenEval, while our learned credit achieves 0.758, a gap of only 0.004. Our ImageReward is 1.34, compared with 1.32 for Exact Shapley.

Our end-to-end cost is 324 GPU-hours, versus 1,128 GPU-hours for Exact Shapley, a 71.3% reduction. Thus, offline counterfactual supervision and amortized credit prediction retain near-oracle generation performance at substantially lower computational cost.

Overall, the new controls support that the benefit of our method extends beyond nonuniform timestep weighting to sample-specific credits aligned with the actual denoising trajectory.

AuthorsOpenReview ↗

Response to Reviewer ZKFM (Part 2/3): Training-seed Reliability

W1 / Q2. Statistical Reliability Across Training Seeds and Human Evaluation

To evaluate training variability and the reliability of the direct human comparison, we provide results from three independent training seeds, together with detailed win/tie/loss counts and image-pair-cluster bootstrap confidence intervals for human preferences.

(1) Multiple Training Seeds and Confidence Intervals

For the principal RL experiments, we report seed-level GenEval differences, their standard deviations (SD), and 95% confidence intervals (CI). Full results are provided in Table G1 of our General Response; the comparisons most relevant to this review are summarized below.

Table Z2. Statistical reliability of the main GenEval comparisons.

ComparisonMean Δ GenEval ↑SD of Δ95% CI
Ours − Uniform+0.0600.007[+0.0426, +0.0774]
Ours − Mean Shapley+0.0340.004[+0.0241, +0.0439]
Ours − Mean Actual Weights+0.0460.005[+0.0336, +0.0584]
Ours − Cross-instance Swap+0.0860.006[+0.0711, +0.1009]
Ours − TreeGRPO+0.0220.006[+0.0071, +0.0369]
Ours − Exact Shapley−0.0040.006[−0.0189, +0.0109]

95% CIs are calculated from differences across three paired training runs using Student's t distribution (n=3, df=2).

The gains from sample-specific attribution show consistent positive differences across training seeds. Relative to Mean Shapley, the improvement is +0.034 GenEval, with a 95% CI of [+0.0241, +0.0439]; relative to Mean Actual Weights, +0.046, CI [+0.0336, +0.0584]; and relative to Cross-instance Swap, +0.086, CI [+0.0711, +0.1009].

Compared with TreeGRPO, we obtain +0.022 GenEval, with a 95% CI of [+0.0071, +0.0369]. On the additional FLUX.1-dev generator, the mean improvement over TreeGRPO is +0.060 (SD=0.005), offering further evidence across generators.

The mean difference between learned and exact Shapley is only −0.004, with a 95% CI of [−0.0189, +0.0109], indicating similar observed mean generation performance in the evaluated setting.

We will provide the complete per-seed scores and statistical protocol in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer ZKFM (Part 3/3): Human Evaluation

(2) Human Evaluation Protocol and Confidence Intervals

To clarify the reported 53.3% direct human preference over TreeGRPO, we provide the complete annotation protocol and outcome counts.

Thirty expert annotators evaluated 100 image pairs, with six independent judgments per pair and 20 pairs per annotator, yielding 600 judgments in total. All comparisons were blind, and the left/right presentation order was randomized.

Crucially, annotators recorded a Win/Loss only when they could clearly discern a meaningful difference in overall image quality and confidently determine which image was better. If the difference was insufficiently clear, or neither image could be reliably judged superior, they selected Tie. A Tie therefore indicates the absence of a sufficiently clear preference for a decisive judgment, not necessarily identical image quality.

Table Z3. Direct human preference against TreeGRPO.

MetricResult
Expert Annotators30
Unique Image Pairs100
Judgments per Pair6
Total Judgments600
Ours Wins320 (53.33%)
TreeGRPO Wins27 (4.50%)
Ties253 (42.17%)
Raw Win Rate (95% CI)53.33% [48.4%, 58.2%]
Tie-adjusted Preference (95% CI)74.42% [71.6%, 77.3%]

The 53.3% reported in the submission is the raw win rate: 320 of 600 judgments explicitly favored our method. In the original protocol, ties remain in the denominator but do not count as wins.

Under this decisive-preference-only criterion, 253 judgments (42.17%) were ties. To represent these judgments more completely, we additionally report tie-adjusted preference, assigning half a vote to each method for a tie; our score is 74.42%.

We estimate uncertainty using image-pair-cluster bootstrap: image pairs are resampled as clusters, retaining all six judgments within each pair to account for dependence among repeated annotations. The resulting 95% CIs are [48.4%, 58.2%] for the raw win rate and [71.6%, 77.3%] for tie-adjusted preference.

The symmetric reference for tie-adjusted preference is 50%, and its confidence interval lies entirely above this value, supporting an overall human preference for our generated images under the evaluated protocol.

Summary

The fixed-timestep, average-credit, and cross-instance swapping controls show that trajectory-aligned counterfactual credits provide benefits beyond fixed or average temporal weights. The three-seed analyses, confidence intervals, and full human evaluation statistics further support the reliability of the reported improvements.

We will add implementation configurations, per-seed results, and evaluation protocols to the revision. We thank the reviewer again for the valuable feedback.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (Part 2/2):

C4. Statistical Reliability Across Training Seeds

To address the single-seed limitation of the original submission, we repeated the principal actor-RL comparisons with three independent training seeds and report the standard deviations and 95% confidence intervals of the GenEval differences.

Table G1. Statistical reliability of the main GenEval improvements.

GeneratorComparisonMean DeltaSD of Delta95% CI
SD3.5-MOurs - Uniform+0.0600.007[+0.0426, +0.0774]
Ours - Mean Shapley+0.0340.004[+0.0241, +0.0439]
Ours - Mean Actual Weights+0.0460.005[+0.0336, +0.0584]
Ours - Cross-instance Swap+0.0860.006[+0.0711, +0.1009]
Ours - TreeGRPO+0.0220.006[+0.0071, +0.0369]
Ours - TurningPoint-GRPO+0.0200.006[+0.0051, +0.0349]
Ours - Exact Shapley-0.0040.006[-0.0189, +0.0109]
FLUX.1-devOurs - TreeGRPO+0.0600.005[+0.0476, +0.0724]
Ours - TurningPoint-GRPO+0.0500.006[+0.0351, +0.0649]

The 95% CIs are computed from the seed-paired differences using Student's t distribution (n = 3, df = 2). We will provide all per-seed scores and the statistical protocol in the revision.

The results show positive mean improvements over the main baselines across three training runs, with additional support from FLUX.1-dev. We also provide full win/tie/loss counts and image-pair-cluster bootstrap CIs for the direct human comparison with TreeGRPO; see our response to Reviewer ZKFM.

C5. Manuscript Revisions and Reviewer-specific Responses

In the revised manuscript, we will:

  1. Clarify the methodology: Reorganize the Introduction and refine Figures 1-2 to explain the progression from terminal preference to counterfactual attribution, trajectory reward modeling, and RL optimization.
  2. Improve baseline reproducibility: Specify input features, fusion architectures, trainable modules, hyperparameters, parameter counts, and training budgets.
  3. Expand experimental evidence: Include the new credit ablations, TurningPoint-GRPO comparisons, FLUX.1-dev and T2I-CompBench++ evaluations, and statistical results.
  4. Detail computational costs: Separate offline attribution training, online reward evaluation, and actor optimization costs.

We address each reviewer's specific concerns separately: ZKFM -- the benefits of sample-specific attribution and the reliability of training and human evaluations; nasX -- the ODE reference, alternative interventions, policy evolution, and cost; kDVW -- TurningPoint-GRPO, Table 2 baseline construction, the necessity of Shapley/two-stage modeling, and broader evaluation.

We thank the reviewers again for their valuable feedback. We hope these clarifications and additional results better establish the contributions of counterfactual trajectory attribution, preference-aware reward modeling, and efficient diffusion RL.

AuthorsOpenReview ↗

General Response (Part 1/2): Motivation and Methodological Contributions

We sincerely thank all three reviewers for their careful evaluations and constructive feedback. To address concerns regarding the motivation, methodological contributions, and empirical evidence, we clarify the central design of our approach and provide additional experiments on sample-specific credits, alternative counterfactual replacements, TurningPoint-GRPO, computational efficiency, three training seeds, FLUX.1-dev, and T2I-CompBench++.

We first summarize the motivation and key contributions; reviewer-specific concerns are addressed in our individual responses.

C1. Motivation: From Terminal Preference to Process Attribution

Our goal is to extend an existing terminal image-preference reward model into a trajectory reward model that jointly predicts terminal preference gain and the contributions of individual denoising transitions.

Existing image-preference reward models can assess final image quality, but they do not directly identify how each denoising transition contributes to that outcome. Prior step-wise reward approaches obtain process feedback using ODE completions from intermediate states, often requiring repeated image generation and reward evaluation during online RL.

We therefore seek to infer terminal preference gain and step-level credits for the stochastic window directly from completed denoising trajectories and final images in a single retrospective reward-model call, without additional step-by-step ODE completions during online RL.

C2. Methodological Contributions

We develop three complementary components:

(1) Counterfactual Shapley attribution -- defining the supervision target. We construct counterfactual credits by switching denoising transitions between SDE and ODE updates and replaying affected downstream trajectories. Unlike a reward difference measured along one temporal ordering, Shapley averaging captures each transition's marginal contribution across different intervention coalitions, accounting for interactions among transitions.

(2) Two-stage trajectory reward modeling -- amortizing credit prediction. Stage I adapts the terminal preference VLM to noisy-latent evidence. Stage II uses offline exact Shapley targets to train the model to jointly predict step credits and terminal preference gain from completed mixed/all-ODE trajectory pairs and final images. At online RL time, a single trajectory-RM call produces the credit array directly from the already-collected mixed/all-ODE pair, avoiding any additional online coalition enumeration.

(3) Credit-aware advantage allocation -- applying attribution to RL. The terminal prediction determines the trajectory-level group advantage, while step credits determine its allocation across denoising transitions. Bounded, nonnegative weights preserve the terminal advantage's sign and its scalar mean across steps while enabling differentiated policy updates.

Together, these components integrate interaction-aware counterfactual supervision, preference-aware trajectory modeling, and efficient amortized credit allocation into diffusion RL.

C3. Additional Experimental Evidence

We conducted complementary experiments to evaluate the practical value of the proposed components.

Evaluation focusKey resultMain finding
Sample-specific creditsOurs 0.758 vs. Mean Shapley 0.724+0.034 GenEval beyond average temporal credit
Coalition averagingExact Shapley 0.762 vs. Chronological 0.731 / LOO 0.724Averaging across coalitions outperforms the tested single-context credit rules
Amortized attributionLearned 0.758 vs. Exact 0.762; 324 vs. 1,128 GPU-hNear-oracle performance with lower end-to-end compute
TurningPoint-GRPO+0.020 GenEval on SD3.5-M; +0.050 on FLUX.1-devHigher average full-system performance on two generators
Two-stage trainingStep MAE 0.028 to 0.019; Spearman 0.51 to 0.70Noisy-latent preference adaptation improves credit prediction
Compositional generationHighest mean score in all eight T2I-CompBench++ categoriesGains extend to counting, spatial relations, and complex compositions

We also evaluated alternative counterfactual replacement rules, reward-model adaptation under an out-of-distribution actor policy, and the computational costs of offline attribution construction and online RL.

Collectively, these experiments examine the value of the attribution target, sample specificity, learned amortization, and cross-generator generalization. Detailed protocols and results are provided in the reviewer-specific responses and will be included in the revision.

AuthorsOpenReview ↗

Summary of Manuscript Revisions

We thank all three reviewers for their detailed and constructive feedback. We have revised the 10-page main text and expanded the appendices to clarify the motivation, strengthen the experimental controls, and document statistical reliability, computational costs, and reward-model adaptation. The principal changes and their locations are summarized below.

Review concernRevision and location
Motivation and presentationReorganized the Introduction (pp. 1–2) around the progression from terminal image preference to trajectory credit. Simplified Fig. 2 (p. 4) to place offline supervision/training and online actor optimization in vertically arranged stages; clarified the end-to-end workflow in Secs. 3–4.
Sample-specific attributionAdded fixed early/middle/late timestep priors, Mean Shapley, mean actual weights, cross-instance swapping, chronological marginals, and leave-one-out comparisons (Sec. 5.4, Table 3; Appendix I.1, Table 11).
Methodological comparisonsAdded TurningPoint-GRPO to Related Work and complete-system comparisons (Sec. 2; Table 3; Appendix I.1), and expanded the implementation descriptions of the attribution-prediction baselines (Sec. 5.2; Appendix I.8).
Statistical reliabilityRepeated principal actor-RL comparisons with three independent training seeds (Tables 3–4); reported paired GenEval differences, SDs, and nominal 95% Student-t CIs (Appendix I.2, Table 12). Tables 1–2 are explicitly labeled as single-run diagnostics.
Human evaluationAdded the full blind pairwise annotation protocol, win/tie/loss counts, and image-pair-cluster bootstrap intervals for the direct TreeGRPO comparison (Appendix I.3, Table 13).
ODE reference sensitivityClarified the shared all-ODE reference and ODE/SDE replay in Sec. 3.1, and evaluated zero-noise SDE drift and an alternative ODE solver (Appendix I.4, Table 14).
Policy evolution and refreshAdded checkpoint-wise trajectory-RM reliability, new- versus old-policy label-budget controls, and complete-RL refresh comparisons (Sec. 5; Appendix I.5, Tables 15–17).
Compute and scopeReported offline adaptation/labeling costs, complete RL iteration times, and end-to-end GPU-hours (Sec. 5; Appendix E; Appendix I.6, Table 18), with additional scope discussion in Appendix H.
Broader evaluationAdded generator-specific experiments on FLUX.1-dev (Table 4) and eight T2I-CompBench++ categories (Appendix I.7, Table 19).

The revision now distinguishes controlled credit-allocation comparisons from full-method baselines, and separates single-run attribution diagnostics from three-seed actor-RL results. We hope the revised presentation and additional evidence make the scope and practical contributions of counterfactual trajectory reward modeling easier to assess.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.