acceptodds
Under review as a conference paper at ICLR 2027

Density Matters: Normalized Successor Measures for Zero-Shot Control

Abstract

Behavior foundation models (BFMs) promise zero-shot control: pretrain on offline data, then reuse the learned representation for downstream tasks without task-specific training. Successor-measure methods such as Forward–Backward representations (FB) realize this promise for diverse locomotion behaviors, but can struggle when success depends on precise interactions. For example, in pick-and-place tasks, FB policies can reach an object but fail to maintain a grasp or carry it to a specified goal. We attribute these failures to control bottlenecks, transitions that are necessary passages to valuable regions where the choice of action substantially affects return. We show that the low-rank squared approximation used by FB can suppress the differences in action values at these states, even at its global optimum, causing the agent to choose incorrect actions. To preserve these differences with limited capacity, we introduce Density-FB, which treats the bilinear output as a score, normalizes the scores into a distribution over future states with a softmax, and learns this distribution with a categorical Bellman objective. Across OGBench, MetaWorld, and ManiSkill, Density-FB improves manipulation success over FB by a median of , , and percentage points, respectively. Additionally, Density-FB preserves correct action rankings with smaller embeddings than vanilla FB.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Offline RL, Zero-shot RL, Manipulation, Unsupervised Learning, Object-Centric Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RKssOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Density-FB, which replaces the direct bilinear parameterization F⊤BF^\top B of the successor relative density mm with exponentiated scores exp⁡(F⊤B/τ)\exp(F^\top B/\tau) normalized over future states. The motivation is to improve the approximation accuracy of the successor measure MM, especially the action-dependent differences that a direct low-rank fit may miss. The approach additionally uses a categorical Bellman objective and evaluates action values by averaging rewards under the predicted successor distribution.

Critical strengths

The paper has a clear motivation, and the central idea is simple: treating the bilinear F⊤BF^\top B output as a score and exponentiating and normalizing it, rather than using the bilinear output directly as a successor density.

Critical weaknesses

First, the central motivation of improving the fit to the successor measure MM lacks direct quantitative validation. The paper does not compare the estimation errors of vanilla FB and Density-FB for MM. Proposition 1 first bounds the QQ error using total variation (TV) distance and then applies Pinsker's inequality to obtain a KL-based condition; a direct TV formulation would be more natural, and TV would also permit comparison with vanilla FB's potentially negative F⊤BF^\top B estimate, to which KL does not directly apply. Figure 3(c) therefore would be more informative in TV rather than KL. Moreover, a gridworld would allow MM to be computed exactly and the method to be tested without Monte Carlo or in-batch-softmax approximations. The current nine-state MDP is too toy-like, while the continuous-control robotic benchmarks are difficult to analyze, so neither provides a sufficiently direct and controlled test of the core idea. Second, the effect of the exponential parameterization is confounded by several other modifications: replacing MSE with categorical cross-entropy (CE), introducing hindsight-goal sampling of zz on some tasks but not others, and using the centered and scaled latent reward rz=(B−runningmean(B))⊤z/dr_z=(B-\mathrm{running_mean}(B))^\top z/\sqrt{d} instead of vanilla FB's rz=B⊤zr_z=B^\top z. In the nine-state MDP, MSE and CE appear to perform comparably, making the motivation for the loss change unclear; the parameterization should be studied with MSE (or a minimally modified version) independently of CE, potentially also with an L1L_1 loss motivated by the TV bound. The CE implementation additionally uses an in-batch softmax, with implications for the independence of reference-state samples and an unnecessary restriction of the candidate pool to the batch size. There is insufficient ablation of the separate roles of relativity, nonlinearity, and nonnegativity, and insufficient discussion of the loss of the linear Q=F⊤zQ=F^\top z readout and the resulting need for Monte Carlo/reference-pool estimation. The sensitivity to reference-pool size is also missing. Finally, the name "Density-FB" is somewhat misleading: the distinctive change is closer to a log-linear rather than linear FB parameterization, whereas the concept of a density was already present in vanilla FB.

Questions for authors

(1) Could Proposition 1 be stated directly in TV distance, and could Figure 3(c) and other evaluations of successor-measure approximation error report TV, so that F⊤BF^\top B and exp⁡(F⊤B/τ)\exp(F^\top B/\tau) can be directly compared even when the former is not nonnegative? (2) Could the authors quantitatively compare how well vanilla FB and Density-FB fit MM, ideally on a gridworld where MM is exactly computable and training does not require Monte Carlo or in-batch-softmax approximations? (3) Could the effect of the exponential parameterization be isolated from the choice of loss, with MSE or minimally modified MSE as the primary comparison, and with separate CE and possibly L1L_1 results? In particular, why is CE necessary when MSE and CE appear on par in the nine-state MDP? (4) How much do the task-dependent hindsight-goal sampling of zz and the centered, scaled latent reward contribute relative to the original FB choices? (5) Why use the batch next states for the CE softmax rather than independently sampling reference states u∼ρu\sim\rho to estimate the normalizer ZZ (potentially including s′s' to retain the immediate target)? How does in-batch sampling affect the i.i.d. property and performance, and why limit the candidate pool to a batch of 1,024 when, for d=50d=50, even one million reference states would require only on the order of 100 million FLOPs per sample? (6) Could the authors ablate alternative transformations such as ReLU, ReLU2\mathrm{ReLU}^2, Softplus, and tanh to separate the effects of relativity, nonlinearity, and nonnegativity? (7) Could the authors discuss the limitation that the nonlinear parameterization prevents use of the standard Q=F⊤zQ=F^\top z readout and instead requires Monte Carlo or reference-pool estimation of QQ? (8) How sensitive are these estimates to reference-pool size, especially for high-dimensional states or when MM concentrates far from the support of ρ\rho? What exactly is the "pool sweep" listed in Appendix Table 4, and what are its results?

Reviewer_gKGVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Density-FB, a method that extends Forward-Backward (FB) representations to preserve action-value differences at control bottlenecks in manipulation tasks. Building on TD InfoNCE as the basis of its estimator, Density-FB treats the bilinear output of the forward and backward networks as a score, applies a softmax over minibatch next states to form a normalized distribution trained with a categorical temporal-difference loss, and evaluates actions by averaging rewards under the predicted distribution.

Critical strengths
  • The paper provides a well-defined problem formulation and a solid mathematical framework for analyzing control bottlenecks.
  • The empirical evaluation is broad, covering many environments and tasks across manipulation and locomotion benchmarks.
Critical weaknesses
  • The proof sketch of Proposition 1 is missing a verb (line 250). There is a typo "approximatio" (line 200). Overall, the presentation can be further improved.
  • Novelty is limited as it combines an existing TD InfoNCE critic with a TD-JEPA / FQL actor. Furthermore, although the abstract claims. standard FB fails "even at its global optimum", Lemma 1 only proves projection loss when the backward features are held fixed rather than jointly optimized.
  • In Table 1, the OGBench experiments condition the policy directly on the ground-truth goal state instead of using zero-shot reward inference.
  • No limitations are discussed.
Questions for authors
  • Can this framework be extended to offline-to-online RL?
  • What are the limitations of this work?
Reviewer_dtkhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses a failure of low-rank forward-backward representations as one of the bottlenecks can be viewed in the fact two actions can look the same in the learned features and therefore get the same fitted value, even when the true action values differ. The authors propose Density-FB to alleviate that in order to keep the usual bilinear score, turn it into a distribution over future states, and train the actor on the reward under that distribution instead of the linear readout. On a small didactic task this keeps the action order at a smaller feature size than casual FB, and they also report gains over baselines on larger control suites.

Critical strengths
  • the motivation is understandable in terms of soundness and important, with the small didactic example supporting it. If two actions have the same expected backward features, least squares is forced to give them the same value, even when one action is truly better. The authors show this on a simple graph: the fitted values sit on top of each other while the exact values do not. So the reason to change the representation is visible and demanding for the field. The reasonable justification for the method also occurs from its extension to the initial idea in a form of softmax transformation analogously to InfoNCE, both theoretically and empirically
Critical weaknesses
  • it is not clear if the evaluation is truthful enough to state methods were compared in the same settings: Density-FB is given the goal directly with a hand-crafted kernel in objet positions, for FB and other baselines it is hard to say in terms of soundness did they receive the same information. When the task is acquired by the model from the reward, which is one of the concerns regarded to zero-shot RL and FB methods, success rates drop
  • on the manipulation tasks the softmax is not isolated. There is no pair of runs that share the actor, the feature size, the conditioning, and the reward, and differ only in that distribution. The small-graph result can be right, and the big table can still be a different method
Questions for authors
  • on the OGBench table, did FB see the goal and the same kernel as Density-FB, or only a reward-inferred signal?
  • what does Density-FB reach if you train the reward-inference setup for the same full compute budget?
  • have you tried an ablation on the cube tasks where the only change from FB is the softmax?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.