- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes State Flow Credit Assignment (SFCA), an auxiliary method added to PPO and A2C. It uses predictions of future states to construct policy-learning signals.
A deterministic encoder maps observations to latent states. A vector field is trained on consecutive latent states through flow matching: it predicts their difference along an interpolated path. Training uses Gaussian noise, a replay buffer of recent rollouts, and a periodically updated target encoder.
For each observed transition, the method predicts future latent states from the current state and its observed successor. SFCA-Adv uses predicted rewards and values to construct an auxiliary advantage. Appendix A relates this signal to an advantage decomposition under conditional distribution approximations. SFCA-Reward adds a difference of predicted future values to the reward.
Experiments compare the augmented methods with PPO and A2C on classic control, Box2D, Atari RAM, and MiniGrid tasks, averaging over five seeds. The authors report improvements on several tasks, including Kangaroo and Gopher, alongside smaller gains or regressions elsewhere. CartPole ablations examine prediction horizon, replay size, and encoder synchronization. The advantage interpretation depends on the modeling assumptions.
Critical strengths
-
Learning state evolution to construct auxiliary policy signals is an interesting empirical direction. Both variants can be incorporated into PPO and A2C, and the evaluation covers 24 tasks across several environment families with five seeds. The reported PPO gains on Kangaroo and Gopher are substantial. The appendices include per-task results and full training curves, including cases with little improvement or lower returns.
-
SFCA-Reward has a useful interpretation independent of the accuracy of the learned transition model. For frozen networks and consistent time indexing, the mean value predicted after m steps defines a state potential, and Eq. (17) has the standard potential-difference form. This interpretation requires the usual discount conditions; a bounded potential and consistent handling of terminal terms are sufficient. Poor predictions alone do not invalidate it. This gives the reward variant a clearer justification than the general advantage claim.
Critical weaknesses
-
The objective in Eq. (13), without noise augmentation, does not establish the conditional rollout laws used in Appendix A.2. With an identity encoder and independent standard-normal successive states, integrating the population-optimal field from a fixed state returns . It therefore generates rather than the true reset law , despite matching the marginals. The source-noise description in Eq. (14) and the interpolant-noise description in Algorithms 2-3 are different objectives; neither description establishes the conditional transition law required by Appendix A.2. Appendix A.2 explicitly assumes a conditional-law identity before Eq. (23). The concern is how training justifies that premise, not a contradiction of the derivation when correct rollouts are assumed. Also, a deterministic encoder alone does not ensure the latent sufficiency needed in Eq. (21). No error bound links the learned field to the claimed advantage approximation.
-
The reward model in Section 3.1 is more general than the state-pair predictor used in Eq. (15). Consider one state, two actions with identical transitions, rewards 0 and 1, and a uniform policy. The true advantages are and . With fixed predictors and action-independent noise, both actions give the same signal distribution because their state-pair inputs are identical. Its conditional mean must therefore have absolute error at least for one action. The advantage interpretation needs a restriction on the reward function, or use of the observed reward or action in the forced branch. This does not show that the augmented PPO/A2C algorithms cannot learn: they retain a standard learning signal. It also does not refute the potential-based interpretation of SFCA-Reward.
Questions for authors
-
Please state the assumptions behind Eqs. (21)-(23) and explain which are guaranteed by training. Please distinguish latent sufficiency and conditional-kernel assumptions from consequences of flow matching, and reconcile the two noise descriptions. Does the RL-time input advance during the repeated integrations in Algorithm 1?
-
Does the first term in Eq. (15) use the observed reward or action in the implementation? If not, what reward assumption excludes the one-state example? For SFCA-Reward, how are terminal potentials and truncations handled?
-
Please compare with TD-Flow (ICML 2025) and InFOM (ICLR 2026), which use flow matching to model future states for value estimation or policy learning. Their settings differ from SFCA. Also, please qualify the general criticism of HCA/CCA, which give exact identities or unbiased estimators under their stated conditions.
-
Please clarify how the reported standard deviations are aggregated and provide results by seed. Under the default CartPole cap of 500, cannot describe the mean and standard deviation of five final run means. An episode-level or other aggregation may explain this. Please also specify the ODE step, Monte Carlo sample count, auxiliary-loss weights, and relative compute cost.