- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper builds on Value Flows, which uses a flow model to learn the distribution of future returns. It uses the flow’s initial velocity to estimate expected return, then adds a regularization term inspired by an HJB equation from a separate minimum-time control problem. The regularizer is meant to shape how the estimated value changes across nearby states and actions. The paper evaluates it on offline OGBench and D4RL tasks, plus five offline-to-online tasks. Results improve on several domain-level scores, though gains vary by task.
Critical strengths
1、The paper adds a local value-shaping signal to a distributional critic without adding a separate critic or dynamics model.
2、Evaluation covers 25 OGBench tasks, 12 D4RL Adroit tasks, and five offline-to-online tasks.
3、The paper is careful to distinguish its auxiliary HJB prior from the benchmark’s actual discounted return.
Critical weaknesses
1、The HJB connection is indirect. The prior comes from a separate minimum-time control problem, not the task’s return equation. The results show that the full regularizer can help, but do not show that the HJB structure itself is what helps.
2、The paper’s mean-return identity holds for the population flow-matching solution at the initial flow time. The training objective samples flow time uniformly, so that endpoint receives no direct supervision. The paper notes this caveat, but gives no quantitative check that the trained readout matches the actual mean return.
3、The implemented loss does more than regularize the mean value. It uses one noise sample per transition, and the paper shows that this also penalizes variation across noise-conditioned derivatives. There is no ablation showing which part drives the gains.
4、Optimizing derivatives with respect to state and action requires mixed derivatives during critic training. The paper does not report the resulting runtime or memory overhead.
Questions for authors
1、Does the training procedure directly sample the initial flow time, or rely on the network’s behavior near that endpoint? Is there a quantitative comparison between the readout and the actual mean return?
2、Are there existing ablations or sensitivity results for the individual regularizer terms and the final hyperparameters?
3、Can the authors report the training-time or memory overhead from the added state and action derivatives?