- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies the robustness of temporal credit assignment in text-to-image reinforcement learning when the learned reward model is imperfect. The authors inject calibrated perturbations into reward-model outputs before credit construction and compare Flow-GRPO, PCPO, DenseGRPO, TP-GRPO, and, as a complementary control, TreeGRPO. The results suggest that temporal specificity is not inherently fragile; rather, DenseGRPO, which relies directly on adjacent reward differences, is particularly sensitive to reward corruption. Human evaluations further show that real HPSv3–human disagreement is more concentrated and temporally persistent than the iid corruption model predicts, and that DenseGRPO can achieve the highest HPSv3 score while being strongly disfavored by human annotators.
Critical strengths
- The paper shows that finer temporal credit is not inherently more reliable, because reward-model error may be amplified by the credit construction.
- It separates the source of reward evidence, the temporal specificity of credit, and the transformation and normalization from evidence to learning signal.
Critical weaknesses
- Causal isolation remains incomplete. The methods also differ in evaluated outcomes, rollout structure, normalization, and regularization. The results are consistent with adjacent differencing as a source of sensitivity, but do not establish it as the dominant or unique cause.
- The noise model has limited external validity. The authors validate the marginal scale and add a correlated-noise control, but real RM errors may also be prompt- or content-biased, heteroscedastic, and policy-dependent; the human study already shows stronger concentration and persistence.
- Algorithmic novelty is limited. The main contribution is a diagnostic framework and mechanistic evidence rather than a new credit-assignment method that improves both robustness and human preference.
Questions for authors
- Are the conclusions stable under structured errors? Under prompt- or content-level bias, heteroscedasticity, or policy-dependent error, could the relative ordering of DenseGRPO, TP-GRPO, and Flow-GRPO change?
- What are the concrete failure modes of DenseGRPO? The appendix shows strong stylization, semantic drift, and even replacement of target objects by human faces. Could the authors classify these failures and report their frequency, severity, and relationship to the HPSv3 margin?
- Can credit-transformation sensitivity be characterized uniformly? Could the Jacobian, a Lipschitz constant, or covariance propagation through yield a method-agnostic metric, and when would it predict downstream degradation?
- When is adjacent differencing necessarily more fragile? It amplifies independent noise but may cancel shared bias. Under a shared-bias-plus-local-residual error model, when is it worse than trajectory aggregation or descendant backup, and when might it be more robust?
7 comments
W3 — Algorithmic Novelty and Scientific Contribution
Our work provides a systematic investigation of how imperfections in learned reward models propagate through temporal credit assignment in text-to-image RL. We identify and characterize a failure mechanism that is not apparent from clean reward optimization alone, and establish several findings with direct implications for the design and evaluation of RL algorithms.
1. Temporal credit granularity does not determine robustness.
Across three learned preference reward models and multiple credit-assignment methods, we observe substantially different sensitivity to reward corruption. In particular, DenseGRPO's reliance on adjacent reward differences leads to pronounced degradation, while alternative temporally structured constructions and the descendant-backed TreeGRPO demonstrate that step-specific credit need not exhibit the same fragility.
2. Reward-error propagation depends on the credit transformation and the structure of uncertainty.
Our analysis connects local reward margins, differencing, aggregation, normalization, and temporal error covariance to the reliability of the resulting learning signal. Controlled perturbation experiments and correlated-noise ablations demonstrate that identical marginal reward uncertainty can produce different robustness outcomes depending on how that uncertainty enters credit assignment.
This provides a concrete diagnostic principle: the sensitivity of a temporal credit method should be assessed at the level of the constructed learning signal, rather than inferred solely from reward-model accuracy or temporal granularity.
3. The observed failures have measurable consequences for human preference.
Our human evaluation reveals that higher optimized reward does not necessarily translate into better human-perceived image quality. Importantly, the new reward-matched comparison shows that Flow-GRPO retains a 77.4% human-preference advantage over DenseGRPO despite nearly identical attained HPSv3 scores (13.85556 vs. 13.86070).
This demonstrates the practical importance of evaluating reward optimization and human alignment jointly, rather than relying on the optimized reward alone.
Together, these findings establish an empirical and mechanistic understanding of the interaction between reward-model uncertainty, temporal credit construction, and optimization robustness. They provide actionable insights for identifying fragile credit assignments, designing more reliable learning signals, and evaluating RL methods under imperfect reward supervision.
Q2 — DenseGRPO Failure Modes: Frequency, Severity, and Reward Misalignment
To systematically investigate the failure modes identified in the review, we conducted an additional blinded human annotation study covering 600 images from our original HPSv3 evaluation (200 DrawBench prompts × three methods).
Each image was independently evaluated by two raters, with 20 raters participating overall. We examined four failure categories, recording both their occurrence and severity (0–2). Importantly, person/face substitution refers specifically to unrequested human content replacing the intended subject, while stylization is considered a failure only when it conflicts with the prompt.
1. DenseGRPO exhibits substantially more semantic failures
The resulting annotation rates reveal consistent differences across all four failure categories:
| Failure category | DenseGRPO | Flow-GRPO | TP-GRPO |
|---|---|---|---|
| Person/face substitution | 60.75% | 22.75% | 26.00% |
| Semantic mismatch | 59.00% | 33.00% | 35.25% |
| Visible defects | 56.00% | 34.00% | 39.50% |
| Prompt-conflicting stylization | 42.50% | 23.50% | 26.75% |
The most pronounced difference concerns person/face substitution. DenseGRPO exceeds Flow-GRPO by 38.0 percentage points (95% paired-prompt bootstrap CI: 31.0–44.75) and TP-GRPO by 34.75 points (95% CI: 27.5–41.75).
Crucially, these are not merely stylistic differences. The dominant failure modes involve incorrect semantic content and replacement of the objects explicitly requested by the prompt.
2. The differences extend to severe failures
The severity annotations reinforce this pattern.
Severe person/face substitution (severity 2) occurs in 37.25% of DenseGRPO annotations, compared with 9.00% for Flow-GRPO and 9.25% for TP-GRPO.
Similarly, severe semantic mismatch reaches 31.75% for DenseGRPO, versus 15.75% and 15.25% for Flow-GRPO and TP-GRPO, respectively.
Thus, DenseGRPO exhibits not only a higher frequency of annotated failures but also a substantially higher rate of severe prompt-content errors.
3. These failures are poorly reflected in HPSv3 reward margins
We additionally investigated whether HPSv3 reward margins capture the observed semantic failures.
Across prompts, Spearman correlations between DenseGRPO's annotated failure rates and its HPSv3 margins relative to Flow-GRPO and TP-GRPO range from −0.030 to +0.054.
These near-zero correlations show no meaningful monotonic association between the annotated failures and DenseGRPO's relative HPSv3 advantage in this evaluation.
This provides complementary evidence for the reward–human divergence identified in our main study: substantial semantic failures can coexist with competitive or higher reward-model scores, illustrating why optimized reward alone is insufficient to characterize generation quality.
We appreciate the reviewer’s recognition that the paper separates temporal specificity, reward evidence, and credit construction. We address the concerns through complementary empirical and analytical evidence: a new controlled credit-assignment ablation, quantitative annotation of DenseGRPO failure modes, a unified Jacobian-based sensitivity characterization, and an analysis of structured reward errors. The completed analyses are presented below, while the training results of the new ablation will be reported separately after validation.
W1 — Causal Isolation of Credit Construction
Our experiments across three learned preference reward models reveal systematic differences in the noise sensitivity of existing temporal credit-assignment methods. The clean-performance-matched comparison, TreeGRPO control, and temporally correlated-noise experiments provide complementary evidence that robustness depends on how uncertain reward evidence is transformed into credit, rather than temporal specificity alone.
To further strengthen causal isolation, we have launched an additional controlled credit-assignment ablation within a unified training pipeline, using the restored original TP-GRPO implementation.
We hold the sampling procedure, mixed SDE/ODE rollout, independent-latent ODE reference, intermediate reward evaluations, KL regularization (β = 1e−4), PPO clipping (1e−5), advantage normalization, optimizer, and training budget fixed.
The only experimental variable is the credit construction:
Both constructions are evaluated under clean and calibrated IID HPSv3 reward corruption (α = 1), using identical initialization and seed 42.
This experiment directly isolates the contribution of turning-point corrections to reward-noise robustness under otherwise identical training conditions. We will provide the completed results in a follow-up.
W2 / Q1 — External Validity and Structured Reward Errors
Our study identifies the temporal structure of reward-model errors as an important factor in credit-assignment robustness. We investigate this through both direct human-preference measurements and controlled experiments that vary error correlation.
1. Real RM–human disagreements exhibit nontrivial temporal structure.
Our HPSv3 human study shows that the calibrated perturbation accurately reproduces the aggregate disagreement rate (40.3% predicted vs. 40.0% observed). Beyond this marginal agreement, we characterize how disagreements are distributed across prompts and denoising trajectories.
We find substantially greater temporal persistence than predicted by IID corruption: adjacent disagreement-event correlation is 0.187 versus 0.048, while conditional disagreement continuation is 53.4% versus 43.2%.
These findings demonstrate that marginal calibration alone is insufficient to characterize real RM–human disagreement, motivating the explicit study of temporal error dependence.
2. We directly test how structured errors affect downstream robustness.
Our HPSv2 AR(1) experiments vary temporal correlation while holding marginal noise variance fixed. DenseGRPO's reward degradation decreases from 17.4% under IID corruption to 15.8% at ρ = 0.5 and 10.3% at ρ = 0.9.
This behavior is consistent with the variance transformation induced by adjacent differencing, where positively correlated errors partially cancel. The experiment demonstrates that changing the dependence structure of reward errors can substantially alter downstream robustness, even at the same marginal noise magnitude.
3. Robustness depends on the interaction between error structure and credit construction.
Our results establish that credit-assignment sensitivity cannot be characterized independently of the errors entering the credit transformation. In particular, adjacent differencing amplifies independent perturbations but can suppress shared components, as further derived in our response to Q4.
The resulting robustness behavior is therefore conditional on the structure of reward uncertainty, rather than an intrinsic, covariance-independent property of an algorithm.
Taken together, our human-disagreement analysis, correlated-noise experiments, and mechanistic characterization establish why temporal error structure matters for the reliability of credit assignment under imperfect reward supervision.