- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Density-FB, which replaces the direct bilinear parameterization of the successor relative density with exponentiated scores normalized over future states. The motivation is to improve the approximation accuracy of the successor measure , especially the action-dependent differences that a direct low-rank fit may miss. The approach additionally uses a categorical Bellman objective and evaluates action values by averaging rewards under the predicted successor distribution.
Critical strengths
The paper has a clear motivation, and the central idea is simple: treating the bilinear output as a score and exponentiating and normalizing it, rather than using the bilinear output directly as a successor density.
Critical weaknesses
First, the central motivation of improving the fit to the successor measure lacks direct quantitative validation. The paper does not compare the estimation errors of vanilla FB and Density-FB for . Proposition 1 first bounds the error using total variation (TV) distance and then applies Pinsker's inequality to obtain a KL-based condition; a direct TV formulation would be more natural, and TV would also permit comparison with vanilla FB's potentially negative estimate, to which KL does not directly apply. Figure 3(c) therefore would be more informative in TV rather than KL. Moreover, a gridworld would allow to be computed exactly and the method to be tested without Monte Carlo or in-batch-softmax approximations. The current nine-state MDP is too toy-like, while the continuous-control robotic benchmarks are difficult to analyze, so neither provides a sufficiently direct and controlled test of the core idea. Second, the effect of the exponential parameterization is confounded by several other modifications: replacing MSE with categorical cross-entropy (CE), introducing hindsight-goal sampling of on some tasks but not others, and using the centered and scaled latent reward instead of vanilla FB's . In the nine-state MDP, MSE and CE appear to perform comparably, making the motivation for the loss change unclear; the parameterization should be studied with MSE (or a minimally modified version) independently of CE, potentially also with an loss motivated by the TV bound. The CE implementation additionally uses an in-batch softmax, with implications for the independence of reference-state samples and an unnecessary restriction of the candidate pool to the batch size. There is insufficient ablation of the separate roles of relativity, nonlinearity, and nonnegativity, and insufficient discussion of the loss of the linear readout and the resulting need for Monte Carlo/reference-pool estimation. The sensitivity to reference-pool size is also missing. Finally, the name "Density-FB" is somewhat misleading: the distinctive change is closer to a log-linear rather than linear FB parameterization, whereas the concept of a density was already present in vanilla FB.
Questions for authors
(1) Could Proposition 1 be stated directly in TV distance, and could Figure 3(c) and other evaluations of successor-measure approximation error report TV, so that and can be directly compared even when the former is not nonnegative? (2) Could the authors quantitatively compare how well vanilla FB and Density-FB fit , ideally on a gridworld where is exactly computable and training does not require Monte Carlo or in-batch-softmax approximations? (3) Could the effect of the exponential parameterization be isolated from the choice of loss, with MSE or minimally modified MSE as the primary comparison, and with separate CE and possibly results? In particular, why is CE necessary when MSE and CE appear on par in the nine-state MDP? (4) How much do the task-dependent hindsight-goal sampling of and the centered, scaled latent reward contribute relative to the original FB choices? (5) Why use the batch next states for the CE softmax rather than independently sampling reference states to estimate the normalizer (potentially including to retain the immediate target)? How does in-batch sampling affect the i.i.d. property and performance, and why limit the candidate pool to a batch of 1,024 when, for , even one million reference states would require only on the order of 100 million FLOPs per sample? (6) Could the authors ablate alternative transformations such as ReLU, , Softplus, and tanh to separate the effects of relativity, nonlinearity, and nonnegativity? (7) Could the authors discuss the limitation that the nonlinear parameterization prevents use of the standard readout and instead requires Monte Carlo or reference-pool estimation of ? (8) How sensitive are these estimates to reference-pool size, especially for high-dimensional states or when concentrates far from the support of ? What exactly is the "pool sweep" listed in Appendix Table 4, and what are its results?