As unsupervised pretraining becomes increasingly ubiquitous in reinforcement learning, developing a theoretical understanding of these methods is becoming as important as their empirical success. We focus on unsupervised learning via interaction, where forward-backward (FB) representation learning serves as a prototypical example. In standard FB, the policy defining the successor measure depends on the representation being learned, resulting in a coupled system. This raises the question: *Can the successor measure Mβ of a single behaviour policy suffice for zero-shot RL?* To understand the tradeoff this simplification introduces, we build on prior work that analyzes FB and the finite-dimensional factorizations of successor measures. We derive additional necessary conditions for exact FB representations, which we use to study learning behaviour in didactic settings, and quantify the consequences of low-rank approximation used in practice. In didactic settings, we observed substantially improved convergence when learning Mβ with 105× smaller representation errors than FB. However, in a didactic counterexample, we observed that even perfect knowledge of Mβ need not yield optimal downstream control. We then compare our simplified method, One-Step FB, to prior work across 10 state-based and image-based continuous control domains. One-Step FB yields representations achieving 1.4× the zero-shot performance of FB on average. We further show that the zero-shot policies inferred by our algorithm provide an efficient initialization for further fine-tuning on downstream tasks.
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.Reviewer_y1zPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
This paper proposes One-Step FB, a simplified approach to zero-shot reinforcement learning. Instead of learning successor measures for a family of latent-conditioned policies, it learns the successor measure of a fixed behavior policy and extracts reward-conditioned policies through one step of policy improvement. The paper analyzes representational constraints and coupled updates in standard FB, and reports competitive performance across state-based and visual control tasks.
Critical strengths
The simplification is useful. A single behavior-policy successor measure provides a simple, competitive baseline, questioning whether learning a whole family of policy-dependent measures is necessary in practice.
The evaluation covers both state-based and visual tasks. I also appreciate the Drunk Spider example, which explicitly shows that one-step improvement can remain suboptimal even with an accurate behavior-policy successor measure.
Critical weaknesses
I think the theoretical contribution is modest. The rank and reward-transformation analyses mainly turn established properties into useful diagnostics for FB. The restricted non-contraction example adds insight, but the case for the proposed simplification still rests mainly on the empirical results.
I am not sure the toy experiments fully support the theoretical explanation. The theory assumes hard-greedy policies, whereas the experiments use finite-temperature softmax policies (Appendix D.1). Scaling rewards can change a softmax policy, so a nonzero affine-equivariance error alone may not establish failure to converge.
Questions for authors
Would the gap in Figure 2 remain with hard-greedy policies, or would it disappear when the diagnostics are evaluated against the corresponding softmax targets? Separating successor-measure fitting error from the effect of policy softening would help me assess the proposed explanation.
We thank the reviewer for their insightful comments and suggestions. The main concerns are (1) distinguishing successor-measure fitting error from downstream control error and (2) understanding how the use of finite-temperature softmax policies affects the results. We have revised Sec. 5.1 to separate these two errors. In Appendix E.3, we have added an evaluation comparison with hard-greedy policies and examined the effect of the softmax temperature through an ablation.
Together with the clarifications below, does this address the reviewer's concern?
W2. The theory assumes hard-greedy policies, whereas the experiments use finite-temperature softmax policies (Appendix D.1). Scaling rewards can change a softmax policy, so a nonzero affine-equivariance error alone may not establish failure to converge.
We agree. At a fixed softmax temperature, scaling the rewards can change the policy. The affine-equivariance property for hard-greedy policies therefore need not hold, even if learning has converged. We have revised Sec. 5.1 to clarify the same.
Q1: Would the gap in Figure 2 remain with greedy policies, or would it disappear when the diagnostics are evaluated against the corresponding softmax targets? Separating successor-measure fitting error from the effect of using finite-temperature softmax policy would help me assess the proposed explanation.
The fitting gap remains when each method is evaluated against its own training target, with matching training and evaluation temperatures. Using greedy policies during evaluation substantially reduces the control gap. However, using near-greedy policies during training increases the fitting gap.
Separating successor-measure fitting error from downstream control error. In Appendix E.3, we distinguish the two questions: (1) Did the representation learn its training target? (2) How good is the policy obtained from that representation?
Fig. 9 (Left) measures how closely each method fits its own successor-measure target. FB retains a nonzero fitting error of 0.03 after training, whereas One-Step FB reaches numerical precision of 10−8.
Fig. 9 (Right) evaluates the trained representations on the same set of rewards. We extract policies from each method and compute each policy’s action values exactly and compare them with optimal action values. FB’s softmax policy has a control error of 0.723. One-Step FB has a control error of 0.013, matching the behaviour policy with one-step of policy improvement.
Confounding effect of using a finite-temperature softmax policy during evaluation. In Fig. 18, we keep the trained FB representation fixed and change only how actions are selected. Selecting with argmax instead of a finite-temperature softmax policy reduces control error from 0.723 to 0.041. This remains higher than 0.013 for a behaviour policy with one-step of policy improvement. Thus, using greedy policy during evaluation does not explain the entire gap in FB’s control error.
Ablation on softmax temperature. In Fig. 12, we vary the softmax temperature, τ, while ensuring the same value during training and evaluation. The softmax temperature affects how well FB fits its target. At τ=200, the policy is nearly uniform, and the fitting error is lower than at τ=1. At τ=10−6, the policy is nearly greedy, and the fitting error is higher than at τ=1.
In a first part, this paper investigates the forward backward (FB) algorithm in finite CMPs showing that its loss may be framed as LSIF, showing that under insufficient rank FB will always have a projection error on some reward functions, and showing that its TD step update followed by a coupled greedy update of the policy is not generally a contraction which is a property often used to prove convergence of TD algorithms. In doing so, the authors offer a different view on a well-established zero-shot RL algorithm and give some theoretical intuition as to why training FB might be fundamentally harder than training single reward algorithms with similar Bellmann error losses.
In a second part, this paper uses this to motivate replacing FBs self-dependence on its greedy policies with the behavior policy restoring the contractive property of its update. They show that their one-step FB can vastly out-perform FB in a small didactic example but that it is not guaranteed to be optimal in another. They show that in practice, one-step FB is competitive with FB and other zero-shot RL algorithms on common offline goal-reaching and continuous control tasks.
Critical strengths
The paper offers a novel perspective on what the FB loss optimizes and why FB methods are hard to learn in practice. It then proposes one-step FB, a method that overcomes some these issues and is competitive with FB in common benchmarks.
Critical weaknesses
This paper combines theoretical analysis of an established baseline (FB) with the introduction of a new method (one-step FB) that addresses one of the issues found in the analysis (Contraction). One-step FB builds on existing one-step RL machinery which has its own issues in that performance may not be optimal depending on the behavior policy. I find it hard to pin down a critical weakness, however, for a clear acceptance the paper would need to provide a broader theoretical analysis that goes beyond FB, eg. extending to all successor measure learning methods or also include successor feature learning methods, or provide more convincing empirical evidence that one-step FB should be preferred over FB in practice.
Questions for authors
This paper shows that FB’s Bellman update coupled with the greedy policy update, is not guaranteed to be a contraction (Prop. 3), which is interpreted as potentially explaining why FB training is hard.
Do methods that learn successor measures and/or features with coupled greedy policies (eg. HILP or TD-JEPA) suffer from the same issue? How broad is this result?
Can a similar limitation be shown when the policy is not exactly greedy but includes SAC-style maximum entropy (MaxEnt) regularization as done in the Soft FB framework [1]?
[1] Soft Forward-Backward Representations for Zero-shot Reinforcement Learning with General Utilities, Bagatella et. al. NeurIPS '26
We thank the reviewer for their helpful comments. The main concerns are (1) the scope of our analysis in Sec. 3.3 and (2) the need for stronger empirical evidence for One-Step FB. We address the theoretical scope in Sec. 3.3, which now connects Proposition 3 to distributional RL, and provide additional empirical evaluation of One-Step FB in the TD-JEPA codebase in Sec. 5.3.
Together with the clarifications below, does this address the reviewer's concern?
W1: Provide stronger empirical evidence for choosing One-Step FB over FB in practice.
We have now updated the main text to also include the direct comparison with TD-JEPA (previously Table 5, now Table 1), which also includes our rerun of FB in the TD-JEPA codebase. We have additionally included two ExORL domains (Walker and Quadruped). We have also conducted online fine-tuning on five OGBench domains in the TD-JEPA codebase, and the results are reported in Appendix E.1. One-Step FB achieves higher mean scores than our FB rerun on four out of seven domains. These results support One-Step FB as a simpler, competitive alternative rather than a uniformly superior replacement. Additionally, the fixed-point, learnability, and limitations of One-Step FB are fully understood.
Q1: Do methods that learn successor measures and/or features with coupled greedy policies (eg. HILP or TD-JEPA) suffer from the same issue? How broad is this result?
Yes, the same non-contraction issue can arise in TD-JEPA’s exact successor-feature update; we have added a note clarifying the scope in Sec. 3.3. Our counterexample does not directly apply to HILP because its representations are updated using a different loss (not Eq. 5, which TD-JEPA also uses).
Q2: Proposition 3 shows that the FB Bellman update need not be a contraction. Sec. 5.1 empirically further investigates FB’s non-zero successor measure fitting error. These results suggest that fitting across the latent-conditioned policy family contributes to this observed difficulty. Can a similar limitation be shown when the policy is not exactly greedy but includes SAC-style maximum entropy (MaxEnt) regularization as done in the Soft FB framework [1]?
To answer this question, we ran an additional experiment comparing with a maximum entropy variant of FB following [1]. Results are in Appendix E.3, showing that maximum-entropy regularization does not eliminate the observed fitting error, which plateaus around 0.09.
[1] Soft Forward-Backward Representations for Zero-shot Reinforcement Learning with General Utilities, Bagatella et. al. NeurIPS '26.
We have worked hard to incorporate the review feedback by running new experiments and revising the paper. Do the revisions and discussions above address your concerns? We would greatly appreciate your engagement.
The paper studies forward–backward (FB) representation learning for zero-shot RL and proposes One-Step FB. One-Step FB learns a low-rank factorisation Fβ(s,a)⊤Bβ(sf,af)≈Mπβ(sf,af∣s,a)/ρ(sf,af) of the behaviour policy's successor-measure ratio. The forward representation is not conditioned on a latent, and the next action is sampled from the data. Policies are then extracted by one step of improvement over Fβ(s,a)⊤z.
a reading of the FB loss as a temporal-difference version of least-squares importance fitting;
a non-contraction result for a restricted version of FB's coupled Bellman update (Prop. 3), presented as motivating the decoupling;
a connection to an SVD of the behavioural successor measure (App. C.7).
Experimentally:
in small discrete MDPs, FB fails to fit its targets while One-Step FB fits its own to about 10−7;
a gridworld shows that one-step improvement can be suboptimal;
on ten ExORL and OGBench domains, One-Step FB averages 1.4x the zero-shot performance of the authors’ reimplementation of FB.
Critical strengths
A simple change that removes the representation target's dependence on the learned policy. Removing the latent from F and sampling a′ from the data fixes the underlying successor-measure target and removes its dependence on the learned policy, while keeping zero-shot task inference. The method is competitive across ten domains, two of them pixel-based. The authors also reimplement it in the released TD-JEPA codebase (App. E.5), which is good practice.
The cost of the simplification is shown rather than hidden. The gridworld of Sec. 5.2 exhibits a case where one step of improvement over the behaviour policy picks a longer route than value iteration, and the limitations paragraph states this. The temporal-difference reading of the FB loss (Sec. 3.2) is also a clean way to present the objective. The paper notes that it appears in Touati et al. (2022, App. B).
Critical weaknesses
For the small numerical checks I claim to have performed below, happy to send the source cross across upon request if useful; please contact the area chair to coordinate.
Weakness 1. The paper attributes FB's difficulty to feedback between the policy and the representation, but this is not established. The theorem offered for it does not identify an obstacle to control, and the didactic experiments do not separate the feedback from the other differences between the two methods.
This needs a change of framing and one targeted control. Please:
position Prop. 3 relative to the corresponding results in distributional RL, and state what it does and does not imply for control;
add a control that removes the feedback while keeping the latent dependence;
compute the didactic diagnostics at the training temperature, on a fixed bank of normalised rewards shared by both methods, and report fitting error separately from control error.
First, let me remark that while Prop. 3 is correct, the coupling is harmless for control within its restricted setting. To see this, write D=diag(ρ) and qk=YkDr, and let πk be greedy for qk. The restricted update then gives exactly qk+1=(1−γ)r+γPπkqk=(1−γ)r+γPmaxa′qk, which is value iteration. So Tr is a γ-contraction in the pseudometric dr(Y,Y′)=∣(Y−Y′)Dr∣∞, although not in ∣⋅∣∞ on Y, and qk→q∗ from any starting point in X. If Q∗ has a positive action gap, the greedy policy is eventually fixed and optimal, and Yk then converges to Mπ∗D−1 by the fixed-policy contraction. The paper's own example behaves exactly this way: after one update, a1 is uniquely greedy with gap 3(1−γ), and the iteration converges. Without a gap, Yk need not converge. In a five-state example with two tied optimal actions, qk converges while Yk has two distinct limit points, along even and odd iterations.
Indeed, Prop. 3, together with the convergence behaviour in (a), mirrors known results for the distributional Bellman optimality operator, which the paper does not cite nor discuss. Concretely, Bellemare et al. (2017, Sec. 3.4) show four things for the distributional Bellman optimality operator: it is not a contraction (Prop. 1); expected values nevertheless contract (Lemma 4); the iterates converge to the set of nonstationary optimal value distributions, hence to a single distribution when the optimal policy is unique (Thm. 1); with ties, the iterates can alternate (Prop. 2). The same picture holds here. Sec. 3.3 nevertheless says it identifies "a theoretical difficulty in FB arising from the dependence of the Bellman target in Eq. 5 on the greedy policy". It then presents the restored contraction ("fixing the behavior policy restores a standard contraction guarantee for the Bellman target in the restricted setting") as the motivation for One-Step FB. But in that same setting, the coupled update also converges given an action gap. What is lost is contraction of the full successor matrix in ∣⋅∣∞, not convergence of the control problem.
So any obstacle to convergence of the control values that is absent from this exact hard-greedy iteration must come from what the restriction removes: a backward representation shared across latents, the low-rank regression step Rfit, and latent-conditioned function approximation. Remark 3 lists these as future work, and they are where the motivation for One-Step FB would have to come from. This leaves open two things: instability of the full successor representation, as in the tie example in (a), and effects of the soft-policy update used in practice.
A second problem is that the didactic diagnostics test greedy properties that the trained objective need not have. Training uses the Boltzmann policy πτ(a∣s,z)∝exp(τF(s,a,z)⊤z) (Eq. 26, with τ acting as an inverse temperature). The Q-error, the KL divergence to π∗ and the equivariance error all compare against value iteration's Q∗ and π∗, or against the affine equivariance of Q∗. An exact fixed point of the soft objective can violate all three. For example, take one state, two self-loop actions, r=(1,0), and the paper's normalised returns. The self-consistent soft policy has p=1/(1+e−τ(1−γ)) and qτ=(1−γ+γp,γp). This is represented exactly, with zero residual, while q∗=(1,γ). At γ=0.9 and the documented training value τ=0.005, ∣qτ−q∗∣∞≈0.45, and qτ(2r)=2qτ(r). Nonzero Q* and equivariance errors therefore do not, by themselves, show that FB failed to converge. The temperature is also specified inconsistently. D.1 gives 0.005 during training and 1 during evaluation. Fig. 9 labels 10 as "(Ours)", and its smallest value is 10−12 in the legend but 10−6 in the text. It is also not stated which policy defines the ground-truth ratio in the successor-measure error.
Another important issue is that several properties of the evaluation make the comparison between FB and One-Step FB hard to read. I list them below:
The latent scale is heavy-tailed. Latents are z=d,u,x/∣x∣ with u∼Cauchy(0,0.5) (D.1), and the Cauchy radius has no second moment. For One-Step FB, the Q-prediction residual at a checkpoint is a fixed linear map Az, so its population mean squared error is infinite whenever A=0. For FB, whether it is finite depends on how the residual behaves for large latents. Either way, the empirical Q-MSE can be dominated by a few extreme evaluation latents. With 1000 of them, the largest typically contributes about 63% of ∑iui2, and the ten largest about 97%.
The equivariance metric is not symmetric between the two methods. For One-Step FB the equivariance residual is ξ(FβBβρ−1), which does not depend on z. For FB it contains the additional term ν(Fz′−Fz)z, so any residual dependence of F on z can be amplified by the heavy-tailed radius. The metric can therefore be harsher on FB.
The two methods are scored on different rewards. Evaluation rewards are recovered as r=(B−1z)⊘ρ from each method's learned B (D.1, D.2), so the reward sets differ between methods and also change during training.
The abstract's "105× smaller representation errors" compares errors against different targets: a single fixed 9×9 matrix versus a latent-indexed family of them. Sec. 5.1 acknowledges this; the abstract does not.
Finally, no experiment isolates the feedback between policy and representation, and the nearest one in the paper does not support the coupling explanation the authors provide. The didactic comparison changes two things at once: first, whether there is feedback; second, the complexity of the prediction problem: a latent-conditioned family F(s,a,z) produced by a network, versus one directly parameterised matrix. The ablations in E.3 vary the learning rate, temperature, parameterisation, objective and MDP. None of them trains against the analytically computed successor measures of a frozen, latent-dependent policy family while keeping F(s,a,z), B, the architecture, the latent distribution and the temperature unchanged. The closest the paper comes is the Fig. 9 run at τ=10−12. There, the policy should be nearly uniform unless the logit ranges τ(maxaQ−minaQ) become extreme, so feedback is at most very weak. The target is then close to the single, latent-independent matrix that One-Step FB fits to below 10−7. Yet FB does worse there than at its default temperature: successor-measure error of about 0.6, against about 10−2; equivariance error of about 0.6, against about 10−3. This raises a question about optimisation or diagnostic conventions that does not depend on the coupling. The unstated convention in (c) and the scale amplification in (d) keep that reading tentative. Reporting the policy entropy and the residual at the training temperature, or running an explicitly uniform-policy control, would resolve it. The frozen, latent-dependent control remains the stronger test.
Weakness 2. The 1.4x gain is measured in a pipeline where the baselines score far below published results, including results the authors reproduce themselves in App. E.5. Where FB is at its published level, the gain is 1.17x and depends on the task.
Please substantiate the claim that prior results are inconsistent, or withdraw it. Please explain why the main pipeline scores methods far below the numbers in App. E.5, and base the headline comparison on baselines that reach their published level, as E.5 does for OGBench; for ExORL, this means the numbers of Park et al. (2024b). Please also clarify the printed HILP objective.
(a) The paper's own appendix contradicts its reason for setting published results aside. Sec. 5.3 reimplements every method because prior work (Park et al., 2024b; Bagatella et al., 2025; Agarwal et al., 2025) "reported inconsistent results", without saying which results disagree or how. In App. E.5, the authors rerun the released code of Bagatella et al. and state that "our rerun reproduces the published TD-JEPA results". On the three OGBench domains that Table 1 shares with E.5, the main pipeline scores the same methods far lower:
antmaze-large
scene
cube-single
TD-JEPA, Table 1
14
6
8
TD-JEPA, authors' rerun (E.5)
51.4
38.2
29.6
FB, Table 1
25
6
2
FB, published (E.5)
36.8
12.8
49.6
One-Step FB, Table 1
30
8
3
One-Step FB, E.5
46.4
37.2
31.8
The two settings differ in representation dimension (512 vs. 50), seeds and per-domain tuning, but the paper gives no reason to expect gaps of this size: TD-JEPA is lower by a factor of 3.7 to 6.4 than the authors' own rerun, and FB by a factor of 25 on cube-single. Most methods are near the floor in Table 1, and the comparison with FB changes direction on cube-single (3 vs. 2 in Table 1; 31.8 vs. 49.6 in E.5).
(b) On ExORL, the reimplemented FB and HILP are far below published results. With the RND dataset, Park et al. (2024b, Table 3) report the following task averages. They are shown on this paper's scale: 100× return divided by the maximum return, which is 1000, or 250 for jaco.
walker
cheetah
quadruped
jaco
FB, published
65.9
41.2
62.9
11.4
FB, this paper
40.0
27.1
24.6
4.0
HILP, published
65.5
52.9
57.5
5.0
HILP, this paper
39.3
11.6
35.2
8.0
One-Step FB
37.9
37.8
64.5
8.8
One-Step FB averages 37.3 on these four domains, against 23.9 for the reimplemented FB and 45.4 for published FB; it is lower than published FB on walker, cheetah and jaco, and level on quadruped. Most of the headline ratio comes from quadruped (2.62x) and jaco (2.20x), where the reimplementation is furthest below published FB; without these two domains, the ten-domain mean of per-domain ratios falls from 1.43 to 1.18. The published numbers come from another implementation, so they are not a controlled comparison, but the paper gives no evidence that they are wrong, and E.5 itself uses published FB numbers as its reference.
(c) The printed HILP objective has the wrong sign. The HILP loss in D.6 is printed with residual −1(s=sf)+γ∣ϕˉ(sf)−ϕˉ(s′)∣+∣ϕ(sf)−ϕ(s)∣. With V=−d, the temporal-difference residual requires −γ∣⋅∣, as in HILP's Eq. 6. Is this a typesetting error or an implementation error? Task inference, by contrast, matches the original: D.6 states that all baselines other than FB and One-Step FB infer the latent by linear regression, as HILP does.
(d) Where FB is at its published level, the improvement is smaller and depends on the task. Over five state-based OGBench domains (E.5), One-Step FB reaches 1.17x the average success rate of TD-JEPA's published FB numbers (41.4 vs. 35.4). These FB numbers are published results, not a rerun. FB is better on cube-single (49.6 vs. 31.8) and close on antmaze-medium. HILP and ICVF exceed One-Step FB on both antmaze domains and on scene (e.g. 83.6 and 79.8 vs. 71.4 on antmaze-medium). Until the baselines in Table 1 reach their published or reproduced level, this is the comparison the paper's claims should rest on.
Minor.
Remark 4 (C.7). It states that QrotFβ⋆, QrotBβ⋆ is again a solution. With F∈Rn×d and B∈Rd×n, the invariance is (F,B)↦(FQ⊤,QB), as C.1 correctly states.
Remark 5 (C.7). It claims that Bβ⋆=Vβ⊤ from an ordinary SVD minimises the orthonormality regulariser. Eq. 8 equals ∣BDB⊤−I∣F2−d, which is minimised when BDB⊤=I, not when BB⊤=I. With uniform ρ, the proposed B gives BDB⊤=I/n, a value of −17/9 instead of the minimum −9 for n=d=9. A ρ-weighted whitening, F↦FC1/2 and B↦C−1/2B with C=BDB⊤, keeps the product and attains the minimum.
Eq. 30 versus C.8. With softmax reward weights, the inferred latent corresponds to the reward r~∝r,eτrewardr. For the dense ExORL rewards, exact one-step improvement then improves the behavioural value of r~, not r; binary rewards are unaffected up to scale. The weighting ablation (Fig. 14c) tests only positive temperatures; τreward=0 would isolate the effect.
Sec. 3.1's rank results. Prop. 1 and Cor. 1–2 apply equally to the single full-rank matrix that One-Step FB fits, so they do not distinguish the two methods. The cross-policy condition is the general solution of FzB=MzD−1. I would present these results as the elementary facts they are.
The fine-tuning claim. The abstract states that One-Step FB's zero-shot policies "provide an efficient initialization for further fine-tuning", and E.1 reports "+40% sample efficiency" over FB.
The claim rests on one task per benchmark (quadruped jump, scene play task 1), and "sample efficiency" is not defined.
In Fig. 6, the difference is mostly the starting point. All four pre-trained initialisations reach the same level within 0.2–0.4M of the 1M steps, and on scene task 1 HILP starts higher than One-Step FB.
Quadruped is also where One-Step FB's zero-shot margin over FB is largest (Table 1), so a head start there is expected.
The evidence supports "pre-training helps relative to TD3 from scratch", not a fine-tuning advantage of One-Step FB over other pre-training methods. Please define the metric, report it per task, and either add tasks or scope the claim.
Aggregation of "1.4x". It is not stated whether it is a mean of per-domain ratios (1.43 from Table 1) or a ratio of means (1.36).
My overall assessment of the paper is that one-Step FB is a simple, well-posed simplification of FB. My score reflects two problems with the paper’s case for the method, not with the method itself. First, the explanation offered for FB’s difficulty, feedback between the policy and the representation, is not established (weakness 1). Prop. 3 is correct, but in its own restricted setting the values of the coupled update follow value iteration and converge. The didactic experiments change several things at once and rely on diagnostics that an exactly converged soft-policy solution need not satisfy. Second, the 1.4x gain is measured in a pipeline where the baselines score far below published results, including results the authors reproduce in App. E.5; where FB is at its published level, the gain is 1.17x and depends on the task.
I would raise my score to weak accept if the revision does two things:
(i) reframes Sec. 3.3 and adds the frozen, latent-dependent policy control, with diagnostics computed at the training temperature;
(ii) reconciles the baseline gap under matched configurations and scopes the empirical claims to the result.
Questions for authors
Which inverse temperature defines the training target, and which defines each diagnostic in Fig. 2 and Figs. 8–13: 0.005, 1 or 10? In the successor-measure error, which policy defines the ground-truth ratio?
How does FB behave when trained against the successor measures of a frozen, latent-dependent policy family, with everything else unchanged?
Is the sign in the printed HILP loss typographical, or does the implementation use it?
Which prior results are inconsistent, and in what way? What accounts for the gap between Table 1 and App. E.5 on the shared OGBench domains, and between Table 1 and Park et al. (2024b) on ExORL?
We thank the reviewer for their insightful comments and suggestions. The main concerns are (1) positioning Proposition 3 relative to the distributional RL result in Bellemare et.al., (2) distinguishing successor-measure fitting difficulty from control error and testing the role of policy–representation coupling, and (3) reconciling the baseline performance gaps in continuous control experiments. We have revised Sec. 3.3 of the main text to clarify that non-contraction of the full representation does not imply non-convergence of control values, with a citation to Bellemare et.al. We have revised Sec. 5.1 to clearly state the softmax temperature used during training and evaluation, added a frozen latent-dependent policy baseline, and updated our interpretation of the results accordingly. Lastly, we have updated Sec. 5.3 to include comparisons in the TD-JEPA codebase and qualify our empirical claims in light of these results.
Together with the clarifications below, does this address the reviewer's concern?
W(i). Positioning Proposition 3 relative to the distributional RL result in Bellemare et.al.
In the revised paper, we have repositioned Proposition 3 relative to Bellemare et al. (2017, Sec. 3.4). We clarify that the lack of a contraction guarantee for the successor representation does not imply that the corresponding action values fail to converge. In the restricted setting where B=I and we have full rank representations, updates are computed exactly, and the policy always selects an action with the highest predicted value. The resulting action values follow standard value iteration and converge to Q∗. If each state has a unique optimal action, the policy eventually stops switching, and the successor-measure target also converges.
This result does not explain the fitting errors observed when training FB with neural networks as seen in Fig.2. We investigate those difficulties separately in Sec. 5.1, including an experiment that holds the target policy family fixed throughout training.
W(i) and Q2. How does FB behave when trained against the successor measures of a frozen, latent-dependent policy family, with everything else unchanged?
As suggested by the reviewer, we have run an additional experiment in which the target policies remain fixed throughout training. We use a frozen copy of the initial forward representation network (F0) to define policies πz=argmaxF0Tz. This means that throughout training, πz is fixed and only the forward and backward representations are updated, decoupling policy-representation learning.
As shown in Fig.10, the fitting error plateaus at 0.4 as compared to 0.03 for FB. Thus, fixing the policies does not eliminate the fitting difficulty.
In a second experiment, we use the same fixed policies but train on only 512 pre-sampled latents. The model therefore has fewer targets to fit, and the training error falls to approximately 5×10−3. By comparison, One-Step FB fits a single successor measure and reached an error of 10−8.
These results suggest that fitting across the latent-conditioned policy family contributes to the observed difficulty, but does not isolate the effects of the shared backward representation, function approximation, and complexity of the policy family. Incorporating this new result, we revised our interpretation of what makes FB learning difficult in Sec. 3.3 and Sec. 5.1.
W(ii).> Reconcile the baseline gap under matched configurations
We have now updated the main text to also include the direct comparison with TD-JEPA (earlier Table 5, now Table 1), which also includes our rerun of FB. We have additionally included two ExORL domains (Walker and Quadruped). We have also conducted online fine-tuning on five OGBench domains in the TD-JEPA codebase, and the results are reported in Appendix E.1. One-Step FB achieves higher mean scores than our FB rerun on four out of seven domains.
> Scope the empirical claims to the result
In the revision, the empirical claims in the abstract, introduction, and Sec. 5.3 are based on the results obtained from implementing One-Step FB in the TD-Jepa codebase (Appendix D.6.1) while also reporting results under our common protocol.
Q1. Which inverse temperature defines the training target, and which defines each diagnostic in Fig. 2 and Figs. 8–13: 0.005, 1 or 10?
The FB training target used inverse temperature 0.005 (in the paper’s convention, τtrain=200). For evaluation, we used softmax temperature τeval=1 throughout for the three-state and five-state cmp experiments. We thank the reviewer for rightly pointing out that this mismatch confounds the interpretation of the fitting error. Additionally, we also found that for the didactic example, a softmax temperature of τtrain=200 results in a near-uniform policy, as suggested by the reviewer.
In the revision, we use τtrain=τeval=1 throughout for Fig.2 and Appendix E.3. The only exception is the policy temperature ablation, where τ varies as stated in the legend while still maintaining τtrain=τeval. We have now explicitly stated this setting in the main text and in the caption of these figures.
> In the successor-measure error, which policy defines the ground-truth ratio?
The table below summarizes each method's policy defining the analytical target.
Method
Policy defining the ground-truth ratio
FB
The latent-dependent policy induced by the current F, which changes during training.
One-Step FB
The behavior policy π_β, sampled once per seed and held fixed independently of z.
Each curve therefore measures how well a method fits its own analytical target in Fig 2. The four diagnostics are calculated on induced policies, π=softmaxa(FTz/τ), for a sampled set of test-time latents for FB and One-Step FB.
Q3. Is the sign in the printed HILP loss typographical, or does the implementation use it?
Thanks for catching this typo, which we have fixed. We have confirmed the implementation was already correct.
Q4. Which prior results are inconsistent, and in what way?
We have rephrased this claim in Sec. 5.3 in the revised paper. We meant that published baseline scores and relative rankings vary across implementations and evaluation protocols. For example, consider these results for ExORL RND:
Park et al.[2] report HILP outperforming FB on cheetah (528.8 vs 412.3), whereas Agarwal et.al.[3] report FB outperforming HILP (586.3 vs 193.5). Similarly, consider the following example for OGBench,
Method
OGBench domain
Bagatella et al.[4]
Tu et al. (OLS)[5]
HILP
Cube-double
20.0
2.8
HILP
Scene
43.8
26.3
TD-JEPA
Scene
38.4
37.9
TD-JEPA
Cube-single
34.2
20.3
FB
Cube-single
49.6
50.4
Again, to clarify, by inconsistent, we meant differences in both absolute baseline scores and relative rankings across reported evaluations.
> What accounts for the gap between Table 1 and App. E.5 on the shared OGBench domains, and between Table 1 and Park et al. (2024b) on ExORL?
We explicitly compare the implementations in [1] to ours and find several differences:
As mentioned in Appendix G.2 of [1], they preprocess (s,a) and (s,z) through separate hidden-layer trunks before feeding them into the forward representations. For the backward representation, they explicitly normalize it to have an ℓ2-norm of d.
When learning the FB representations, [1] includes an additional regularization term that trains F(s,a,z)⊤z as a Q function (Eq. 9 of [1]).
In Appendix G.2 of [1], they smooth the policy using a target network as in TD3.
For simplicity, we use standard MLPs to represent both forward and backward representations and do not include all these implementation tricks. Nevertheless, we conducted ablation studies in which we added these implementation details to our code and used the same hyperparameters as in [1] whenever possible.
Domain
Laplacian [1]
Laplacian (ours)
Laplacian (ours + impl. details)
FB [1]
FB (ours)
FB (ours + impl. details)
walker
613±23
228±2
575±25
654±7
400±40
595±53
cheetah
242±19
125±41
233±27
513±20
271±46
533±58
quadruped
505±17
462±35
532±20
606±3
246±31
585±15
The results in this table suggest that we can achieve comparable performance on ExORL by adding those implementation tricks. Since our goal is to propose a simple method with minimal implementation overhead, it is reasonable to compare all methods under the current protocol.
[2] Park, S., Kreiman, T., and Levine, S. “Foundation Policies with Hilbert Representations.” ICML, 2024.
[3] Agarwal, S., Sikchi, H., Stone, P., and Zhang, A. “Proto Successor Measure: Representing the Behavior Space of an RL Agent.” ICML, 2025.
[4] Bagatella, M., Pirotta, M., Touati, A., Lazaric, A., and Tirinzoni, A. “TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning.” arXiv preprint, 2025.
[5] Tu, K.-H., Chiang, C.-S., Chen, H.-W., Hsieh, P.-C., and Ke, T.-W. “Task Inference Beyond Least Squares in Behavioral Foundation Models.” arXiv preprint, 2026.
As the discussion period comes to a close, we would like to summarize the reviewers’ main concerns, how we addressed them, and the discussion so far.
Reviewer FnTX’s main concerns are (1) positioning Proposition 3 relative to the distributional RL result in Bellemare et. al., (2) distinguishing successor-measure fitting difficulty from control error and testing the role of policy–representation coupling, and (3) reconciling the baseline performance gaps in continuous control experiments.
We have revised Sec. 3.3 of the main text to clarify that non-contraction of the full representation does not imply non-convergence of control values, with a citation to Bellemare et.al. We have revised Sec. 5.1 to clearly state the softmax temperature used during training and evaluation, added a frozen latent-dependent policy baseline, and updated our interpretation of the results accordingly. Lastly, we have updated Sec. 5.3 to include comparisons in the TD-JEPA codebase and qualify our empirical claims in light of these results.
Reviewer 2ZAc’s main concerns are (1) the scope of our analysis in Sec. 3.3 and (2) the need for stronger empirical evidence for One-Step FB.
We address the theoretical scope in Sec. 3.3, which now connects Proposition 3 to distributional RL, and provide additional empirical evaluation of One-Step FB in the TD-JEPA codebase in Sec. 5.3.
Reviewer y1zP’s main concerns are (1) distinguishing successor-measure fitting error from downstream control error and (2) understanding how the use of finite-temperature softmax policies affects the results.
We have revised Sec. 5.1 to separate these two errors. In Appendix E.3, we have added an evaluation comparison with hard-greedy policies and examined the effect of the softmax temperature through an ablation.
In their follow-up, reviewer y1zP confirmed that the additional experiments addressed their question about Fig. 2. They then asked when a single successor measure is sufficient to match FB in practice. We responded by highlighting that the dataset-quality experiments in Appendix E.2 suggest that the data used for pre-training matters when choosing between FB and One-Step FB. Following this exchange, the reviewer increased their rating.
Kind regards.
The Authors
From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.
Response to Reviewer y1zP
We thank the reviewer for their insightful comments and suggestions. The main concerns are (1) distinguishing successor-measure fitting error from downstream control error and (2) understanding how the use of finite-temperature softmax policies affects the results. We have revised Sec. 5.1 to separate these two errors. In Appendix E.3, we have added an evaluation comparison with hard-greedy policies and examined the effect of the softmax temperature through an ablation.
Together with the clarifications below, does this address the reviewer's concern?
W2. The theory assumes hard-greedy policies, whereas the experiments use finite-temperature softmax policies (Appendix D.1). Scaling rewards can change a softmax policy, so a nonzero affine-equivariance error alone may not establish failure to converge.
We agree. At a fixed softmax temperature, scaling the rewards can change the policy. The affine-equivariance property for hard-greedy policies therefore need not hold, even if learning has converged. We have revised Sec. 5.1 to clarify the same.
Q1: Would the gap in Figure 2 remain with greedy policies, or would it disappear when the diagnostics are evaluated against the corresponding softmax targets? Separating successor-measure fitting error from the effect of using finite-temperature softmax policy would help me assess the proposed explanation.
The fitting gap remains when each method is evaluated against its own training target, with matching training and evaluation temperatures. Using greedy policies during evaluation substantially reduces the control gap. However, using near-greedy policies during training increases the fitting gap.
Separating successor-measure fitting error from downstream control error. In Appendix E.3, we distinguish the two questions: (1) Did the representation learn its training target? (2) How good is the policy obtained from that representation?
Fig. 9 (Left) measures how closely each method fits its own successor-measure target. FB retains a nonzero fitting error of 0.03 after training, whereas One-Step FB reaches numerical precision of 10−8.
Fig. 9 (Right) evaluates the trained representations on the same set of rewards. We extract policies from each method and compute each policy’s action values exactly and compare them with optimal action values. FB’s softmax policy has a control error of 0.723. One-Step FB has a control error of 0.013, matching the behaviour policy with one-step of policy improvement.
Confounding effect of using a finite-temperature softmax policy during evaluation. In Fig. 18, we keep the trained FB representation fixed and change only how actions are selected. Selecting with argmax instead of a finite-temperature softmax policy reduces control error from 0.723 to 0.041. This remains higher than 0.013 for a behaviour policy with one-step of policy improvement. Thus, using greedy policy during evaluation does not explain the entire gap in FB’s control error.
Ablation on softmax temperature. In Fig. 12, we vary the softmax temperature, τ, while ensuring the same value during training and evaluation. The softmax temperature affects how well FB fits its target. At τ=200, the policy is nearly uniform, and the fitting error is lower than at τ=1. At τ=10−6, the policy is nearly greedy, and the fitting error is higher than at τ=1.