acceptodds
Desk rejected at ICLR 2027

Exploring the Design Space of Reward Backpropagation for Flow Matching

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

20 views

Abstract

Direct reward backpropagation aligns text-to-image flow matching models sample-efficiently, but practical methods fix different surrogate backward trajectories, leaving unclear which choices drive performance. Full-rollout differentiation requires substantial activation memory and can amplify gradients through chained Jacobians. Surrogates reduce these costs but introduce fidelity issues: detached methods query rewards on model-implied clean estimates, while connector methods such as LeapAlign use derivatives of single-velocity interval approximations. We propose FlowBP, a framework that treats the backward trajectory as the design object and exposes four axes: reward-model input, active set, integration weights, and bridge coupling. Prior direct-gradient methods occupy specific settings in this space. Transplanting endpoint reconstruction into ReFL and DRTune improves all five metrics on FLUX.1-dev with active sets and optimizer unchanged. Three instantiations evaluate the sampled image, eliminating connector residuals through Euler reconstruction (FlowBP-Sparse, FlowBP-Bridge) or targeting smaller residuals through multi-support quadrature (FlowBP-Lagrange). They retain activations for only a few velocity calls and at most one nested state Jacobian. Across SD3.5-M, FLUX.1-dev, and FLUX.2-Klein-base, they improve on prior direct-gradient baselines in preference metrics and overall compositional prompt following. Balancing alignment gains, training time, and tuning reliability, we recommend FlowBP-Sparse with K = 3 active calls as the default recipe.

Primary area:
generative models
Keywords:
Rectified Flow, Flow Matching, Reward Fine-Tuning, Direct Reward Backpropagation, Text-to-Image Generation

Related papers

Discussion (0)

Sign in to comment.