acceptodds
Under review as a conference paper at ICLR 2027

Temporal Credit Assignment under Imperfect Rewards in Text-to-Image RL

Abstract

Recent text-to-image reinforcement learning methods increasingly use temporally structured credit to assign different learning signals across denoising steps. Because these signals are constructed from learned reward models that are imperfect proxies for human preference, finer temporal credit can also change how reward-model error enters optimization. We study this interaction through a controlled stress test that perturbs reward-model outputs before credit construction, using reward-specific corruption scales informed by human preference calibration. Across three reward models, we compare terminal, locally differenced, and trajectory-structured credit; a descendant-backed construction serves as a complementary control. We find that robustness is not determined by temporal specificity alone. A construction that relies directly on local reward differences consistently degrades most under corruption, while alternative forms of temporally structured credit preserve substantially more of their clean performance. Controls that match clean optimization strength and vary the temporal covariance of reward errors further support this interpretation. For HPSv3, human judgments reveal a complementary picture: the calibrated perturbation captures the marginal scale and margin dependence of reward-model–human disagreement, but observed disagreements are more concentrated and temporally persistent than the iid probe predicts. Moreover, under ordinary HPSv3 training without injected corruption, the construction that is most sensitive in the controlled study also exhibits substantial divergence between optimized reward and human preference. Together, these results point away from temporal specificity alone and toward the construction of temporal credit, with adjacent reward differencing emerging as a particularly sensitive case.

Primary area:
generative models
Keywords:
reward models, text-to-image, RL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iQ47OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies the robustness of temporal credit assignment in text-to-image reinforcement learning when the learned reward model is imperfect. The authors inject calibrated perturbations into reward-model outputs before credit construction and compare Flow-GRPO, PCPO, DenseGRPO, TP-GRPO, and, as a complementary control, TreeGRPO. The results suggest that temporal specificity is not inherently fragile; rather, DenseGRPO, which relies directly on adjacent reward differences, is particularly sensitive to reward corruption. Human evaluations further show that real HPSv3–human disagreement is more concentrated and temporally persistent than the iid corruption model predicts, and that DenseGRPO can achieve the highest HPSv3 score while being strongly disfavored by human annotators.

Critical strengths
  1. The paper shows that finer temporal credit is not inherently more reliable, because reward-model error may be amplified by the credit construction.
  2. It separates the source of reward evidence, the temporal specificity of credit, and the transformation and normalization from evidence to learning signal.
Critical weaknesses
  1. Causal isolation remains incomplete. The methods also differ in evaluated outcomes, rollout structure, normalization, and regularization. The results are consistent with adjacent differencing as a source of sensitivity, but do not establish it as the dominant or unique cause.
  2. The noise model has limited external validity. The authors validate the marginal scale and add a correlated-noise control, but real RM errors may also be prompt- or content-biased, heteroscedastic, and policy-dependent; the human study already shows stronger concentration and persistence.
  3. Algorithmic novelty is limited. The main contribution is a diagnostic framework and mechanistic evidence rather than a new credit-assignment method that improves both robustness and human preference.
Questions for authors
  1. Are the conclusions stable under structured errors? Under prompt- or content-level bias, heteroscedasticity, or policy-dependent error, could the relative ordering of DenseGRPO, TP-GRPO, and Flow-GRPO change?
  2. What are the concrete failure modes of DenseGRPO? The appendix shows strong stylization, semantic drift, and even replacement of target objects by human faces. Could the authors classify these failures and report their frequency, severity, and relationship to the HPSv3 margin?
  3. Can credit-transformation sensitivity be characterized uniformly? Could the Jacobian, a Lipschitz constant, or covariance propagation through Fa=Na∘CaF_a=N_a\circ C_a yield a method-agnostic metric, and when would it predict downstream degradation?
  4. When is adjacent differencing necessarily more fragile? It amplifies independent noise but may cancel shared bias. Under a shared-bias-plus-local-residual error model, when is it worse than trajectory aggregation or descendant backup, and when might it be more robust?
7 comments

We appreciate the reviewer’s recognition that the paper separates temporal specificity, reward evidence, and credit construction. We address the concerns through complementary empirical and analytical evidence: a new controlled credit-assignment ablation, quantitative annotation of DenseGRPO failure modes, a unified Jacobian-based sensitivity characterization, and an analysis of structured reward errors. The completed analyses are presented below, while the training results of the new ablation will be reported separately after validation.

W1 — Causal Isolation of Credit Construction

Our experiments across three learned preference reward models reveal systematic differences in the noise sensitivity of existing temporal credit-assignment methods. The clean-performance-matched comparison, TreeGRPO control, and temporally correlated-noise experiments provide complementary evidence that robustness depends on how uncertain reward evidence is transformed into credit, rather than temporal specificity alone.

To further strengthen causal isolation, we have launched an additional controlled credit-assignment ablation within a unified training pipeline, using the restored original TP-GRPO implementation.

We hold the sampling procedure, mixed SDE/ODE rollout, independent-latent ODE reference, intermediate reward evaluations, KL regularization (β = 1e−4), PPO clipping (1e−5), advantage normalization, optimizer, and training budget fixed.

The only experimental variable is the credit construction:

  • Adjacent differencing: local reward increments without turning-point corrections.
  • Turning-point-aware credit: the same local increments augmented with TP-GRPO turning-point corrections.

Both constructions are evaluated under clean and calibrated IID HPSv3 reward corruption (α = 1), using identical initialization and seed 42.

This experiment directly isolates the contribution of turning-point corrections to reward-noise robustness under otherwise identical training conditions. We will provide the completed results in a follow-up.

W2 / Q1 — External Validity and Structured Reward Errors

Our study identifies the temporal structure of reward-model errors as an important factor in credit-assignment robustness. We investigate this through both direct human-preference measurements and controlled experiments that vary error correlation.

1. Real RM–human disagreements exhibit nontrivial temporal structure.

Our HPSv3 human study shows that the calibrated perturbation accurately reproduces the aggregate disagreement rate (40.3% predicted vs. 40.0% observed). Beyond this marginal agreement, we characterize how disagreements are distributed across prompts and denoising trajectories.

We find substantially greater temporal persistence than predicted by IID corruption: adjacent disagreement-event correlation is 0.187 versus 0.048, while conditional disagreement continuation is 53.4% versus 43.2%.

These findings demonstrate that marginal calibration alone is insufficient to characterize real RM–human disagreement, motivating the explicit study of temporal error dependence.

2. We directly test how structured errors affect downstream robustness.

Our HPSv2 AR(1) experiments vary temporal correlation while holding marginal noise variance fixed. DenseGRPO's reward degradation decreases from 17.4% under IID corruption to 15.8% at ρ = 0.5 and 10.3% at ρ = 0.9.

This behavior is consistent with the variance transformation induced by adjacent differencing, where positively correlated errors partially cancel. The experiment demonstrates that changing the dependence structure of reward errors can substantially alter downstream robustness, even at the same marginal noise magnitude.

3. Robustness depends on the interaction between error structure and credit construction.

Our results establish that credit-assignment sensitivity cannot be characterized independently of the errors entering the credit transformation. In particular, adjacent differencing amplifies independent perturbations but can suppress shared components, as further derived in our response to Q4.

The resulting robustness behavior is therefore conditional on the structure of reward uncertainty, rather than an intrinsic, covariance-independent property of an algorithm.

Taken together, our human-disagreement analysis, correlated-noise experiments, and mechanistic characterization establish why temporal error structure matters for the reliability of credit assignment under imperfect reward supervision.

W3 — Algorithmic Novelty and Scientific Contribution

Our work provides a systematic investigation of how imperfections in learned reward models propagate through temporal credit assignment in text-to-image RL. We identify and characterize a failure mechanism that is not apparent from clean reward optimization alone, and establish several findings with direct implications for the design and evaluation of RL algorithms.

1. Temporal credit granularity does not determine robustness.

Across three learned preference reward models and multiple credit-assignment methods, we observe substantially different sensitivity to reward corruption. In particular, DenseGRPO's reliance on adjacent reward differences leads to pronounced degradation, while alternative temporally structured constructions and the descendant-backed TreeGRPO demonstrate that step-specific credit need not exhibit the same fragility.

2. Reward-error propagation depends on the credit transformation and the structure of uncertainty.

Our analysis connects local reward margins, differencing, aggregation, normalization, and temporal error covariance to the reliability of the resulting learning signal. Controlled perturbation experiments and correlated-noise ablations demonstrate that identical marginal reward uncertainty can produce different robustness outcomes depending on how that uncertainty enters credit assignment.

This provides a concrete diagnostic principle: the sensitivity of a temporal credit method should be assessed at the level of the constructed learning signal, rather than inferred solely from reward-model accuracy or temporal granularity.

3. The observed failures have measurable consequences for human preference.

Our human evaluation reveals that higher optimized reward does not necessarily translate into better human-perceived image quality. Importantly, the new reward-matched comparison shows that Flow-GRPO retains a 77.4% human-preference advantage over DenseGRPO despite nearly identical attained HPSv3 scores (13.85556 vs. 13.86070).

This demonstrates the practical importance of evaluating reward optimization and human alignment jointly, rather than relying on the optimized reward alone.

Together, these findings establish an empirical and mechanistic understanding of the interaction between reward-model uncertainty, temporal credit construction, and optimization robustness. They provide actionable insights for identifying fragile credit assignments, designing more reliable learning signals, and evaluating RL methods under imperfect reward supervision.

Q2 — DenseGRPO Failure Modes: Frequency, Severity, and Reward Misalignment

To systematically investigate the failure modes identified in the review, we conducted an additional blinded human annotation study covering 600 images from our original HPSv3 evaluation (200 DrawBench prompts × three methods).

Each image was independently evaluated by two raters, with 20 raters participating overall. We examined four failure categories, recording both their occurrence and severity (0–2). Importantly, person/face substitution refers specifically to unrequested human content replacing the intended subject, while stylization is considered a failure only when it conflicts with the prompt.

1. DenseGRPO exhibits substantially more semantic failures

The resulting annotation rates reveal consistent differences across all four failure categories:

Failure categoryDenseGRPOFlow-GRPOTP-GRPO
Person/face substitution60.75%22.75%26.00%
Semantic mismatch59.00%33.00%35.25%
Visible defects56.00%34.00%39.50%
Prompt-conflicting stylization42.50%23.50%26.75%

The most pronounced difference concerns person/face substitution. DenseGRPO exceeds Flow-GRPO by 38.0 percentage points (95% paired-prompt bootstrap CI: 31.0–44.75) and TP-GRPO by 34.75 points (95% CI: 27.5–41.75).

Crucially, these are not merely stylistic differences. The dominant failure modes involve incorrect semantic content and replacement of the objects explicitly requested by the prompt.

2. The differences extend to severe failures

The severity annotations reinforce this pattern.

Severe person/face substitution (severity 2) occurs in 37.25% of DenseGRPO annotations, compared with 9.00% for Flow-GRPO and 9.25% for TP-GRPO.

Similarly, severe semantic mismatch reaches 31.75% for DenseGRPO, versus 15.75% and 15.25% for Flow-GRPO and TP-GRPO, respectively.

Thus, DenseGRPO exhibits not only a higher frequency of annotated failures but also a substantially higher rate of severe prompt-content errors.

3. These failures are poorly reflected in HPSv3 reward margins

We additionally investigated whether HPSv3 reward margins capture the observed semantic failures.

Across prompts, Spearman correlations between DenseGRPO's annotated failure rates and its HPSv3 margins relative to Flow-GRPO and TP-GRPO range from −0.030 to +0.054.

These near-zero correlations show no meaningful monotonic association between the annotated failures and DenseGRPO's relative HPSv3 advantage in this evaluation.

This provides complementary evidence for the reward–human divergence identified in our main study: substantial semantic failures can coexist with competitive or higher reward-model scores, illustrating why optimized reward alone is insufficient to characterize generation quality.

Reviewer_kMhzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a useful diagnostic study about temporal credit in text-to-image model RL. The reward models provide imperfect evaluations and the error may be amplified in credit assignment on timesteps. A perturbation framework is proposed with an extra weighted Gaussian noise to mimic reward model errors. The conclusions of this paper are clear and straightforward, which may shed light on future RL method design.

Critical strengths
  1. This paper provides a novel perspective on RL for text-to-image models. It has not been fully explored by previous work about how reward model error propagate through temporal credit assignment.
  2. The findings about local reward differences may provide insights to future research.
  3. The conclusions are supported by solid sensitivity experiments and human studies.
Critical weaknesses
  1. The practical contributions are limited. Although this paper identifies local reward differences as "vulnerable construction", it fails to propose any other novel alternatives, like an uncertainty-aware construction.
  2. The perturbation based on Gaussian noise is a useful probe, but the gap between this probe and real error structure is not fully explored.
  3. The reward models in experiments are all general-purpose human preference predictors. There are other rewards that are more specific towards fewer dimensions, like counting and text-image alignment. Do they also lead to the conclusions the same as preference-based reward models?
Questions for authors

Please see weaknesses above.

3 comments

We thank the reviewer for highlighting the relevance of reward-error propagation through temporal credit, the novelty of the research perspective, and the sensitivity and human-evaluation evidence. Below we clarify the scientific contribution and the empirical scope.

W1 — Practical Contribution and Novelty

Our paper provides a systematic empirical and mechanistic investigation of a largely overlooked problem in text-to-image RL: how reward-model errors propagate through temporal credit assignment and affect optimization. We establish three key findings with direct implications for the design and evaluation of RL algorithms.

1. Temporal specificity and robustness are fundamentally different properties.

Our experiments across three reward models and multiple credit-assignment methods reveal that the granularity of temporal credit alone does not determine robustness. DenseGRPO, which relies on adjacent reward differences, exhibits substantial sensitivity to reward corruption, while alternative constructions, including trajectory-structured and descendant-backed credit, demonstrate that step-specific learning signals need not exhibit the same fragility.

2. The mechanism of reward-error propagation matters more than the raw uncertainty of the reward model.

We characterize how reward uncertainty interacts with local credit margins, differencing, aggregation, and temporal error covariance. Our controlled perturbation experiments show that the same reward-error magnitude can lead to markedly different degradation depending on how the credit signal is constructed. In particular, correlated-noise experiments demonstrate that the sensitivity of adjacent differencing changes systematically with the temporal structure of reward errors.

This provides a practical diagnostic perspective: the reliability of temporal credit should be evaluated after reward-to-credit transformation, rather than inferred from reward-model accuracy or temporal granularity alone.

3. These findings are relevant to real reward–human misalignment, not merely synthetic noise sensitivity.

Our human evaluation reveals a substantial discrepancy between optimized reward and actual human preference. DenseGRPO can achieve higher HPSv3 scores while being strongly disfavored by human annotators.

Crucially, our new reward-matched evaluation shows that this discrepancy persists even when optimized reward levels are nearly identical: Flow-GRPO is preferred in 77.4% of human comparisons despite mean HPSv3 scores of 13.85556 versus 13.86070.

This demonstrates why optimizing and reporting reward alone can conceal important failures in generated-image quality.

Taken together, these results establish a new understanding of the relationship between reward uncertainty, temporal credit construction, optimization robustness, and human preference. They provide concrete principles for diagnosing unreliable temporal credit and evaluating RL methods under imperfect reward supervision.

W2 — Gaussian Perturbations and Real Reward-Model Errors

Our study goes beyond injecting arbitrary Gaussian noise. We establish a human-calibrated perturbation framework, empirically characterize the structure of real reward-model disagreements, and investigate how temporal error correlation affects optimization robustness.

1. Reward perturbations are grounded in human preference data.

Rather than choosing an arbitrary noise magnitude, we calibrate reward-specific perturbation scales using human preference judgments through a Bradley–Terry model.

Our independent HPSv3 human evaluation demonstrates that this calibration accurately reproduces the aggregate disagreement rate: 40.3% predicted versus 40.0% observed.

This provides an empirical foundation for studying reward-error sensitivity at a scale relevant to observed RM–human disagreement.

2. We directly characterize the temporal structure of real RM–human disagreement.

Beyond marginal calibration, we investigate how disagreement events are distributed across prompts and denoising trajectories.

Our analysis reveals that actual disagreements exhibit substantially greater concentration and temporal persistence than predicted by the IID perturbation model:

MetricHuman observationsIID prediction
Aggregate disagreement40.0%40.3%
Adjacent disagreement-event correlation0.1870.048
Conditional disagreement continuation53.4%43.2%

These findings reveal an important distinction: matching the marginal frequency of reward-model errors does not imply matching their temporal structure.

Identifying and quantifying this discrepancy is itself an empirical contribution of our study.

3. We experimentally establish how temporal error correlation changes robustness.

To investigate this mechanism, we additionally conduct controlled AR(1) corruption experiments, varying temporal correlation while preserving marginal noise variance.

For DenseGRPO on HPSv2, the resulting reward degradation changes substantially:

Error correlationReward degradation
IID, ρ = 017.4%
ρ = 0.515.8%
ρ = 0.910.3%

These results demonstrate that the temporal covariance of reward errors materially affects the robustness of credit assignment, even when the marginal error magnitude remains unchanged.

This behavior is consistent with our analysis of how adjacent reward differencing transforms correlated perturbations into local credit errors.

Taken together, these experiments establish more than a synthetic-noise benchmark. They provide a human-grounded calibration of reward uncertainty, a direct empirical characterization of the temporal structure of RM–human disagreement, and controlled evidence that error covariance influences downstream RL robustness.

The central insight is that reward uncertainty cannot be characterized by its marginal magnitude alone: its temporal structure and interaction with credit construction are essential to understanding optimization robustness.

W3 — Generalization Across Reward Models

Our study specifically investigates learned human-preference reward models, where imperfect predictions of human judgments provide the optimization signal in text-to-image RL. This is a well-motivated research setting: reward-model imperfections can directly influence policy optimization, making the reliability of temporal credit assignment particularly important.

We investigate this phenomenon across three established preference reward models — PickScore, HPSv2, and HPSv3. Our experiments consistently identify adjacent reward differencing as particularly sensitive to injected corruption across these models, demonstrating that the observed phenomenon is not specific to a single reward model.

Importantly, the underlying mechanism is not tied to a particular semantic reward objective. Reward-error propagation through differencing, aggregation, and normalization depends on the structure of the credit transformation, local reward margins, and error covariance. Our analysis characterizes these dependencies, while our experiments establish their downstream consequences for learned preference rewards.

Specialized counting and text-image alignment rewards may exhibit different error structures and optimization behavior, making them interesting directions for further investigation. Our contribution is a systematic empirical and mechanistic characterization of temporal credit robustness under learned human-preference supervision, supported by multiple reward models and controlled perturbation experiments.

Thank you for the detailed explanation. Although the research motivation of this paper is novel and the work on reward-error propagation has practical value, as a mechanism-diagnostic paper, it still has limitations in its research scope and insufficient methodological contributions. In its current state, I am inclined to keep my score.

Reviewer_Ncu5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

I understand this paper as an empirical study of how reward-model errors affect different temporal credit constructions in text-to-image RL. The main finding is that temporal credit is not necessarily fragile, but constructions based on adjacent reward differences appear particularly sensitive to injected noise.

Critical strengths

I find the research question useful and the paper reasonably easy to follow. The experiments cover several reward models and credit constructions, and the correlation controls help explain the observed sensitivity. I also appreciate the direct human evaluation and the authors’ clear discussion of the limitations of synthetic noise.

Critical weaknesses
  1. I think the human-preference result needs further interpretation. DenseGRPO achieves the highest optimized reward, so its lower human preference could partly reflect stronger reward over-optimization. I would like to see human comparisons at matched reward levels, or preference measurements across training checkpoints, to separate optimization strength from credit construction.
  2. The methods also differ in regularization, normalization, sampling, and optimization settings. For example, TP-GRPO uses a substantially different KL coefficient. These comparisons establish differences between the tested configurations, but provide less direct evidence that local differencing is the main cause.
Questions for authors
  1. Does DenseGRPO still perform worse in human preference at matched HPSv3 reward levels? This would help distinguish the effect of credit construction from stronger reward over-optimization.
  2. Could you compare local differencing and alternative credit constructions within the same implementation, holding regularization, sampling, and normalization fixed? I would like to understand how much of the robustness gap comes from credit construction itself.
1 comment

We thank the reviewer for recognizing the importance of the question, the cross-reward-model evidence, the correlation controls, and the direct human evaluation. We address the first question with a new blinded, reward-matched human-preference study and the second with existing controls and an ongoing, targeted credit-only ablation.

Q1 / W1 — Does the human-preference gap persist at matched HPSv3 reward?

Yes. The human-preference gap remains substantial even when Flow-GRPO and DenseGRPO achieve nearly identical HPSv3 rewards.

To directly test the reviewer's hypothesis that DenseGRPO's lower human preference merely reflects stronger reward overoptimization, we conducted an additional blinded human evaluation at matched reward levels.

We used 200 fixed DrawBench prompts, with three independent ratings per image pair. Checkpoints were selected before collecting human preferences, based solely on their mean HPSv3 scores, with a predefined matching tolerance of 0.1.

MetricFlow-GRPODenseGRPO
Mean HPSv313.8555613.86070
HPSv3 difference+0.00514
Human preference77.4%22.6%

Human preference is computed over 199 valid paired comparisons (597 judgments), with ties counted as half. The 95% prompt-bootstrap confidence interval is 74.0–80.8%.

Importantly, the original evaluation showed 81.0% human preference for Flow-GRPO. After reducing the attained mean HPSv3 gap to just 0.00514, the preference for Flow-GRPO remains 77.4%.

Therefore, the substantial human-preference divergence cannot be explained solely by DenseGRPO achieving a higher optimized reward. The discrepancy persists even at closely matched HPSv3 levels, directly addressing the reviewer's proposed alternative explanation.

Q2 / W2 — Can credit constructions be compared with fixed implementation and optimization settings?

The comparisons in the submission use the original, complete implementations. We explicitly distinguish empirical robustness differences across those tested configurations from a causal claim that local subtraction is the unique source of the gap. Existing clean-performance-matched controls, TreeGRPO, and the temporal-correlation intervention already constrain alternative explanations, but do not fully remove implementation-level confounding.

To directly address this concern, we launched an additional controlled experiment within a unified training pipeline, holding the sampling procedure, reward-evaluation protocol, KL regularization, PPO clipping, advantage normalization, optimizer, and training budget fixed.

We compare two temporal credit constructions: adjacent reward differencing and turning-point-aware credit assignment. The only experimental variable is the credit assignment rule, allowing us to directly isolate its contribution to reward-noise robustness.

Both configurations are evaluated under clean and calibrated IID reward corruption. We will report the completed results and corresponding training diagnostics in a follow-up.

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their responses. Could you please read the rebuttal and let us know whether it resolves the questions and concerns raised in your review? If any point remains unclear, feel free to follow up with the authors during the discussion period.

Please update your review and final rating at your earliest convenience.

Thank you very much for your time and contribution.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.