Temporal Credit Assignment under Imperfect Rewards in Text-to-Image RL

Recent text-to-image reinforcement learning methods increasingly use temporally structured credit to assign different learning signals across denoising steps. Because these signals are constructed from learned reward models that are imperfect proxies for human preference, finer temporal credit can also change how reward-model error enters optimization. We study this interaction through a controlled stress test that perturbs reward-model outputs before credit construction, using reward-specific corruption scales informed by human preference calibration. Across three reward models, we compare terminal, locally differenced, and trajectory-structured credit; a descendant-backed construction serves as a complementary control. We find that robustness is not determined by temporal specificity alone. A construction that relies directly on local reward differences consistently degrades most under corruption, while alternative forms of temporally structured credit preserve substantially more of their clean performance. Controls that match clean optimization strength and vary the temporal covariance of reward errors further support this interpretation. For HPSv3, human judgments reveal a complementary picture: the calibrated perturbation captures the marginal scale and margin dependence of reward-model–human disagreement, but observed disagreements are more concentrated and temporally persistent than the iid probe predicts. Moreover, under ordinary HPSv3 training without injected corruption, the construction that is most sensitive in the controlled study also exhibits substantial divergence between optimized reward and human preference. Together, these results point away from temporal specificity alone and toward the construction of temporal credit, with adjacent reward differencing emerging as a particularly sensitive case.