acceptodds
Under review as a conference paper at ICLR 2027

Bridging Reward Regression and Projected Bellman Equation with Successor Features

Abstract

Successor features (SF) decompose the Q-function into a successor feature map and a task vector, enabling efficient transfer in continual and goal-conditioned reinforcement learning. The Q-function estimate depends critically on how the task vector is estimated. We formalize the two estimators used in the literature—reward regression and the projected Bellman equation—prove that they generally yield different solutions, and characterize when they coincide. We unify them through an interpolation parameter and establish a stability–approximation tradeoff: reward regression stabilizes off-policy updates, while moving toward the solution of projected Bellman equation tightens the worst-case approximation guarantee. Building on this, we develop SF-Bellman, a scalable algorithm that improves performance and transfer speed over reward regression based SF baselines and standard baseline algorithms in continual-learning and goal-conditioned control tasks.

Primary area:
reinforcement learning
Keywords:
successor feature, projected Bellman equation, reward regression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hg2qOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies successor features (SFs), which decompose the Q-function as Qπ(s,a)=ψπ(s,a)⊤wQ^\pi(s,a) = \psi^\pi(s,a)^\top w, where ψπ(s,a)=Eπ[∑t≥0γtϕ(st,at)∣s0=s,a0=a]\psi^\pi(s,a) = \mathbb{E}_\pi\big[\sum_{t\ge 0}\gamma^t \phi(s_t,a_t) \mid s_0=s, a_0=a\big] is the expected discounted sum of features and ww is a task vector. The authors consider two existing approaches to estimating ww, (i) the projected reward equation (PRE), which fits ww by regressing immediate rewards onto the features, and (ii) the projected Bellman equation (PBE), which fits ww as the fixed point of the projected Bellman operator over the span of ψπ\psi^\pi, and aim to bridge them using theoretical results. They show that both belong to a single family of estimators parameterized by an interpolation parameter η∈[0,1]\eta \in [0,1], with PRE at η=0\eta = 0 and PBE at η=1\eta = 1, and characterize a tradeoff in which PRE favors stability while PBE yields a tighter approximation guarantee. They then evaluate the resulting algorithm, SF-Bellman, empirically on continual-learning and goal-conditioned control benchmarks, comparing different values of η\eta against SF and standard RL baselines.

Critical strengths
  1. The paper gives a thorough background on successor features and a clear literature review. Table 1 summarizes prior SF methods by their task-vector objective and SF learning scheme, which makes prior contributions and this paper's position easy to follow.
  2. The Cafe–Bakery example clearly illustrates how PRE and PBE can lead to different value estimates and decisions.
  3. The experiments evaluate several values of η\eta on both continual-learning and goal-conditioned tasks, complementing the theory with empirical evidence.
Critical weaknesses
  1. There is a gap between theory and algorithm, as the theory assumes fixed features, a fixed successor-feature map, and a fixed policy.
  2. The two sides of the tradeoff are proven under different assumptions.
  3. The non-SF baselines receive no task information and reset their replay buffers at every task switch, and no baseline uses HER, the standard remedy for sparse goal-reaching rewards.
Questions for authors
  1. Regarding Corollary 3.3, the condition PπΦ=ΦNP^\pi\Phi = \Phi N requires the expected next feature to be a linear function of the current feature under π\pi. Beyond tabular or eigenvector features, when does this hold in practice? Since it depends on π\pi, it seems unlikely to hold when features are learned and the policy changes. Could the authors comment on its practical relevance?

  2. All theoretical results assume fixed features ϕ\phi, a fixed successor-feature map ψπ\psi^\pi, and a fixed policy (policy evaluation), whereas SF-Bellman learns ϕ\phi and ψ\psi online while the policy changes (control with a max⁡\max). The theory therefore does not directly explain the empirical gains. Moreover, the approximation guarantee (Prop. 4.2) assumes on-policy (stationary) data, whereas the stability benefit of reward regression (Lemma 4.4) matters only off-policy, so the tradeoff is never shown within a single setting.

  3. Regarding the Cafe–Bakery example (lines 196–199), from Appendix C (γ=0.9\gamma = 0.9, ϵ=0.1\epsilon = 0.1, stationary μ\mu), we obtain:

& \text{Cafe} & \text{Bakery} \\\\ \hline \text{True } Q^\pi & 2.95 & 1.55 \\\\ \text{Best fit in } \mathrm{col}(\Psi^\pi) & 3.20 & 1.80 \\\\ \text{PRE} & 2.09 & 3.34 \\\\ \text{PBE} & 2.62 & 2.61 \end{array}$$ PBE ranks Cafe first by only 0.01, and both estimates are far from the true values. Does PBE give a better value estimate here, or only a correct ranking? Is the ordering robust to $\epsilon$, $\gamma$, or $\mu$? 4. In the goal-conditioned experiments, the goal is observed and rewards are sparse, which is the standard setting for HER; DDPG+HER reaches 100% success on standard PandaReach (Gallouédec et al., 2021). Why is HER not used? Since it is orthogonal to the SF parameterization, could the authors add it to both SF-Bellman and the baselines, or justify its omission? Could they also report success rates? 5. Is there an approximation guarantee for off-policy data, or an experiment showing both sides of the tradeoff in the same setting? 6. Algorithms 2–5 contain no update for the parameters of $\phi$. How is $\phi$ trained?
3 comments

Q1. There is a gap between theory and algorithm, as the theory assumes fixed features, a fixed successor-feature map, and a fixed policy.

A1. We thank the reviewer for the constructive comments on the gap between the theory and the algorithm. As the reviewer mentioned, the theory provides guarantees only in settings with fixed features and a fixed behavior policy. This is intentional, to separate the complications in the analysis arising from time-varying features or policies. The analysis under these simple assumptions provides clear intuition and analytical tools for SF theory. Extending the analysis to time-varying features and policies would be an important future direction.

Q2. The two sides of the tradeoff are proven under different assumptions.

A2. We thank the reviewer for the valuable comments on the tradeoff. We chose the term “tradeoff” to clarify the roles of reward regression and projected Bellman error. Indeed, the stability problem only applies to the off-policy case, as stability is generally guaranteed in the on-policy case with linear function approximation. Therefore, we have added a new example showing the stability–approximation tradeoff in the off-policy case in Figure 4 and Appendix D of the revised manuscript. This example uses a simple cycle MDP, and the approximation error decreases monotonically as η\eta increases, while there is a threshold that characterizes stability.

Q3. The non-SF baselines receive no task information and reset their replay buffers at every task switch, and no baseline uses HER, the standard remedy for sparse goal-reaching rewards.

A3. We thank the reviewer for the comments on the experiments. We additionally included results without task-vector conditioning and with the same replay buffer reset protocol in common response G3 and G5. The following results show improved or comparable performance with higher η\eta under the same protocols.

Q4. Regarding Corollary 3.3, the condition requires the expected next feature to be a linear function of the current feature under π\pi. Beyond tabular or eigenvector features, when does this hold in practice? Since it depends on π\pi, it seems unlikely to hold when features are learned and the policy changes. Could the authors comment on its practical relevance?

A4. We thank the reviewer for the comment on Corollary 3.3 and its application in practice. As the reviewer mentioned, the condition can be difficult to meet in practice. Satisfying this condition requires additional methods [Ghosh, 2020]. Our goal is not to meet this condition. We rather aim to design an objective that considers both objectives.

Ghosh, Dibya, and Marc G. Bellemare. “Representations for stable off-policy reinforcement learning.” International Conference on Machine Learning. PMLR, 2020.

Q6. Robustness of the Cafe-Bakery example.

A6. We thank the reviewer for the clarification on the Cafe-Bakery example. We can derive a regime in (ϵ,γ)(\epsilon,\gamma) in which PBE chooses the correct answer. For example, when ϵ=0.1\epsilon=0.1, we require γ≥0.895\gamma \geq 0.895. This also highlights that a high γ\gamma is required to reflect the future dynamics. Moreover, PBE provides more accurate estiamte not only correct ranking.

Q7. In the goal-conditioned experiments, the goal is observed and rewards are sparse, which is the standard setting for HER; DDPG+HER reaches 100% success on standard PandaReach (Gallouédec et al., 2021). Why is HER not used? Since it is orthogonal to the SF parameterization, could the authors add it to both SF-Bellman and the baselines, or justify its omission? Could they also report success rates?

A7. We thank the reviewer for the suggestion to apply HER. The following results show performance with HER, and, in general, higher η\eta yields improved or comparable performance. The following reports the held-out goal return and kk denotes the relabelling ratio.

\text{Method} & \text{HER } k & \text{PointMaze} & \text{Reacher} & \text{PandaReach} \\\\ \hline \text{DDPG} & 4 & \mathbf{270.61} \pm 0.49 & 4.59 \pm 4.33 & 29.90 \pm 3.88 \\\\ \text{SF } \eta=0 & 4 & 238.31 \pm 19.08 & \mathbf{42.12} \pm 0.74 & 30.90 \pm 1.27 \\\\ \text{SF } \eta=0.99 & 4 & \mathbf{270.75} \pm 0.69 & 34.36 \pm 8.13 & \mathbf{34.70} \pm 0.12 \\\\ \text{SF } \eta=1.0 & 4 & 259.77 \pm 7.47 & 37.92 \pm 3.05 & 33.35 \pm 1.07 \\\\ \hline \text{DDPG} & 8 & 269.38 \pm 1.27 & 17.20 \pm 7.82 & 27.15 \pm 5.44 \\\\ \text{SF } \eta=0 & 8 & 255.13 \pm 9.84 & 39.74 \pm 1.79 & 31.60 \pm 1.42 \\\\ \text{SF } \eta=0.99 & 8 & \mathbf{271.40} \pm 0.82 & \mathbf{42.96} \pm 0.07 & \mathbf{35.50} \pm 1.18 \\\\ \text{SF } \eta=1.0 & 8 & 239.94 \pm 14.04 & 42.72 \pm 0.37 & 31.15 \pm 1.64 \\\\ \hline \end{array}$$ Moreover, most of the methods achieve near $100\%$ success rate while we aim to solve a more difficult problem of to reach and stabilize at the goal. **Q8.** *Is there an approximation guarantee for off-policy data, or an experiment showing both sides of the tradeoff in the same setting?* **A8.** We thank the reviewer for the constructive comment on approximation guarantees for the off-policy case. We can derive an approximation error bound following Yu and Bertsekas, but the error bound is usually loose and does not provide much information. Similar guarantees would be possible, but, as shown in the lemma, the exact characterization of the error bound is highly nonlinear. Yu, Huizhen, and Dimitri P. Bertsekas. “Error bounds for approximations from projected linear equations.” Mathematics of Operations Research 35.2 (2010): 306–329. **Q9.** *Algorithms 2–5 contain no update for the parameters of $\phi$. How is $\phi$ trained?* **A9.** We thank the reviewer for clarification on $\phi$. $\phi$ is obtained by $l_2$-normalizing the output of the encoder, which is shared with $\psi$. The stop-gradient operation is applied to $\phi$ whenever a backward operation is performed.

I thank the authors for the rebuttal, I'm keeping my positive score.

Reviewer_q9j6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

When using successor features Ψ\Psi to approximate the Q-function, there can be different ways of approximating the task vector ww such that Q≈ΨwQ \approx \Psi w. This paper studies two particular choices: 1) solving for ww using a projected reward equation (PRE), and 2) solving for ww using a projected Bellman equation (PBE). Specifically, the paper proposes a unified view of the two approximations using the parameter η\eta, where PRE corresponds to η=0\eta=0 and PBE corresponds to η=1\eta=1. The paper characterizes theoretically the properties of the unified approximator: 1) larger η\eta reduces the worst-case Q-function approximation error, and 2) smaller η\eta improves stability in the off-policy setting. Empirically, the paper demonstrates improved performance using the unified approximator in continual learning settings.

Critical strengths
  • The paper is well-structured and well-written in general.
  • The paper contains several theoretical results that characterize the properties of the task vector approximator well.
  • The insight that the PBE solution can help distinguish state-action pairs by incorporating information about future rewards is interesting.
Critical weaknesses
  1. Clarity regarding the theory.
    1. Would appreciate a full derivation for Eq. 1 and 2 in the Appendix.
    2. Lemma 3.2 makes the assumption that Ψπ⊤Dμ(I−γPπ)Ψπ\Psi_\pi^\top D_\mu (I - \gamma P^\pi) \Psi_\pi is invertible, which was also necessary for the PBE solution (Eq. 2) to exist. However, the paper doesn't explain what this assumption means, when it holds, and how often it holds.
    3. In the Proof of Corollary 3.3, don't you need some condition on NN for the Neumann series ∑k=0∞γkNk\sum_{k=0}^\infty \gamma^k N^k to converge? Maybe the condition is satisfied already from the equation PπΦ=ΦNP^\pi \Phi = \Phi N, but this should be stated clearly in the paper to avoid confusion.
    4. There is insufficient explanation for Lemma 4.4. For example, the paragraph after Lemma 4.4 states that "the reward regression term can stabilize the update". However, it's not clear to me where in Lemma 4.4 the reward regression term shows up. Likewise, there's no explanation of what c∗c^* is a ratio of.
    5. General comment: While the math appears rigorous and high-level explanations are present, I find that there is often not enough context to help readers understand the specifics, as they often have to piece together what specific terms/assumptions mean themselves.
  2. Experiments are limited
    1. I was expecting an empirical demonstration of the stability vs approximation error tradeoff for different η\eta, since it was one of the main contributions of the paper. Section 6.1 only examines the approximation error for different η\eta. Also, it would be interesting to see this tradeoff in the function approximation settings (Section 6.2), though I'm not sure how hard it is to measure approximation error/stability in the more complex experiments.
    2. I was also expecting a sensitivity analysis for η\eta. However, Section 6.2 only shows results for η=0,0.99,1\eta=0, 0.99, 1, with 0.99 being the only intermediate value between the two endpoints (0 and 1). It is surprising as Figure 2 shows that drops in the approximation error mostly happens for small η\eta, and according to Lemma 4.4 small η\eta improves stability. So I was expecting values of η\eta smaller than 0.99 to also be reported.
    3. While the paper's main contribution is the interpolation between η=0\eta = 0 and η=1\eta = 1, there does not seem to be a significant enough performance improvement of the interpolation compared to η=1\eta = 1. Quite often, η=1\eta = 1 has overlapping confidence intervals (inferred from the reported std. error) compared to η=0.99\eta = 0.99.
  3. Limitations are not sufficiently discussed.

Typo:

  1. Eq. 2: The last Ψπ\Psi^\pi has π\pi as superscript instead of subscript.
Questions for authors

None.

2 comments

Q1. Would appreciate a full derivation for Eq. 1 and 2 in the Appendix.

A1. We thank the reviewer for the comments on Eqs. 1 and 2. The full derivations have been added in Appendix A, using the first-order stationarity conditions of strongly convex functions.


Q2. Lemma 3.2 makes the assumption that ΨT(I−γP)Ψ\Psi^T(I-\gamma P)\Psi is invertible, which was also necessary for the PBE solution (Eq. 2) to exist. However, the paper doesn't explain what this assumption means, when it holds, and how often it holds.

A2. We thank the reviewer for the valuable comments on the invertibility assumption. The invertibility condition is a standard assumption in the RL literature [Tsitsiklis and Van Roy] because it ensures the uniquenss of the solution to PBE. Moreover, this assumption holds under the on-policy case. Therefore, this is a widely used condition to study the convergence of an RL algorithm with function approximation.

Tsitsiklis, John, and Benjamin Van Roy. "Analysis of temporal-diffference learning with function approximation." Advances in neural information processing systems 9 (1996).


Q3. In the Proof of Corollary 3.3, don't you need some condition on NN for the Neumann series to converge? Maybe the condition is satisfied already from the equation PΦ=ΦNP\Phi=\Phi N, but this should be stated clearly in the paper to avoid confusion.

A3. We thank the reviewer for clarification on Corollay 3.3. The required condition is ρ(γN)<1\rho(\gamma N)<1, where ρ\rho denotes the spectral radius. Under the condition PπΦ=ΦNP^\pi\Phi=\Phi N, let (λ,v)(\lambda,v) be an eigenvalue–eigenvector pair of NN. Then

PπΦv=ΦNv=λΦv.P^\pi\Phi v=\Phi Nv=\lambda\Phi v.

Since Φ\Phi has full column rank, Φv≠0\Phi v\neq 0. Hence, λ\lambda is also an eigenvalue of PπP^\pi. As PπP^\pi is a stochastic matrix, ∣λ∣≤1|\lambda|\leq 1. Therefore,

ρ(γN)≤γ<1,\rho(\gamma N)\leq\gamma<1,

which ensures convergence of the Neumann series. We have revised the proof of Corollary 3.3 to state this explicitly.


Q4. There is insufficient explanation for Lemma 4.4. For example, the paragraph after Lemma 4.4 states that "the reward regression term can stabilize the update". However, it's not clear to me where in Lemma 4.4 the reward regression term shows up. Likewise, there's no explanation of what c∗c^* is a ratio of.

A4. We thank the reviewer for the comment on Lemma 4.4. The reward regression term adds (1−η)(R−ΦTDΦθ)(1-\eta)( R-\Phi^T D\Phi\theta) to the overall update. As ΦTDΦ\Phi^T D\Phi is positive definite, when η\eta becomes lower, this positive-definite term can dominate the update making the system overall stable. This term appear as (1−η)(R−ΦTDΦθ)+ηΨTD(I−γP)Ψ(1-\eta)( R-\Phi^T D\Phi\theta)+\eta \Psi^T D (I-\gamma P)\Psi in the oveall update and c∗c^* characterizes the ratio between the magnitude of reward-regression and the magnitude of the Bellman update.


Q5. General comment: While the math appears rigorous and high-level explanations are present, I find that there is often not enough context to help readers understand the specifics, as they often have to piece together what specific terms/assumptions mean themselves.

A5. We thank the reviewer for the general comments on the context of the theoretical results.. We have added a geometric illustration of difference between PRE and PBE solutions in Figure 1 of the revised manuscript. Suppose the best approximation Ψπw∗\Psi_\pi w^* to QπQ^\pi satisfies

\qquad \Psi_\pi^\top DP^\pi(Q^\pi-\Psi_\pi w^*)=0.$$ That is, both the Q-function approximation error and its image under $P^\pi$ are orthogonal to the successor-feature space under the $D$-weighted inner product. If the resulting reward residual $R-\Phi w^*$ is not orthogonal to the basis-feature space, PRE introduces additional approximation error compared to PBE. This explains how the solution to PRE and PBE can differ. --- **Q6.** *Experiments are limited. I was expecting an empirical demonstration of the stability vs approximation error tradeoff for different $\eta$, since it was one of the main contributions of the paper. Section 6.1 only examines the approximation error for different $\eta$. Also, it would be interesting to see this tradeoff in the function approximation settings (Section 6.2), though I'm not sure how hard it is to measure approximation error/stability in the more complex experiments.* **A6.** We thank the reviewer for the comments on the tradeoff. We have added a controlled off-policy example illustrating how approximation error and stability change as $\eta$ varies. In this simple cycle MDP, the approximation error decreases monotonically as $\eta$ increases, while a threshold separates the stable and unstable regimes. This demonstrates the stability–approximation tradeoff within the same problem. The example is provided in Figure 4 and Appendix D of the revised manuscript and summarized in **Common Response G2**.

Q7. Sensitivity analysis of η\eta.

A7. We thank the reviewer for the comments on the senstivity of η\eta. We have added a sensitivity analysis over η∈{0,0.01,0.1,0.5,0.9,0.99,1.0}\eta\in\{0,0.01,0.1,0.5,0.9,0.99,1.0\} on DoorKey6×6 and KeyCorridorS2R1, using feature dimensions 32 and 48 and five seeds per environment. The results are provided in Common Response G3. Larger values of η\eta generally perform better in these experiments, although performance is not monotonic and the best value depends on the setting.


Q8. η=1\eta=1 has overlapping intervals with η=0.99\eta=0.99.

A8. We thank the reviewer for the constructive comment on comparing η=1\eta=1 and η=0.99\eta=0.99. Indeed, choosing an proper η\eta can be difficult, therefore, one can use an scheduling for η\eta such as OL-AUX which balances η\eta by considering the reward regression and projected Bellman error as multi-objective problem. The following result ( which uses without task-vector conditioning and therefore, η=1\eta=1 coincides to DQN) shows that adopting such method can improve the overall performance :

Final evaluation score on last cycle. The protocol uses two tasks over four cycles.

EnvironmentOL-AUXη=0\eta=0DQN (η=1\eta=1)
DK8×8100.0 ± 0.092.1 ± 6.783.3 ± 11.2
KC S2R1100.0 ± 0.0100.0 ± 0.083.3 ± 11.2
FR S1399.2 ± 0.894.6 ± 4.297.1 ± 2.9

Further results are provided in Common Response G4, with details in Appendix I of the revised manuscript.


References

John Tsitsiklis and Benjamin Van Roy. Analysis of temporal-difference learning with function approximation. Advances in Neural Information Processing Systems, 9, 1996.

Xingyu Lin, Harjatin Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.

Reviewer_Rcr7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem that in successor feature reinforcement learning, reward regression can fit immediate rewards but may produce inaccurate long-term value estimates, while Bellman-based estimation may be unstable under off-policy learning. To this end, the authors propose SF-Bellman, which interpolates between reward regression and the projected Bellman equation to balance approximation accuracy and stability. They also derive the conditions under which the two estimators coincide and analyze the resulting trade-off. In the reported experiments, SF-Bellman improves the normalized score by about 22% over the reward regression baseline in continual learning and by about 31% in goal-conditioned control, while generally achieving better final performance and transfer speed than both endpoint variants.

Critical strengths
  1. A major strength of the paper is that it clarifies the relationship between reward regression and PBE-based task-vector estimation. The equivalence condition and counterexample show that these two commonly used approaches are not interchangeable and explain how their difference affects value estimation. This analysis provides a useful basis for understanding existing successor-feature methods and may also guide the design of future transfer and continual-learning algorithms.

  2. The empirical evaluation is relatively thorough and reproducible. The authors first verify the theoretical result in controlled synthetic MDPs and then evaluate the method across discrete and continuous control tasks. They report results over 20 seeds for the synthetic experiments and at least 15 seeds for the online experiments, together with standard errors, network architectures, pseudocode, hyperparameter search spaces, and selected settings. These details make it easier to assess whether the reported improvements are consistent and to reproduce the method.

Critical weaknesses
  1. My main concern is that the experiments do not fully demonstrate the stability–approximation trade-off proposed in the paper. The online experiments only consider η=0\eta=0, 0.990.99, and 11, without explaining how 0.990.99 was selected or reporting results for intermediate values such as 0.50.5. They also do not directly measure instability, for example through divergence frequency, task-vector norms, or TD-error variance. Therefore, the current results show that one setting close to the PBE endpoint works well, but do not establish how the proposed interpolation behaves more generally.

  2. The broader empirical comparison is difficult to interpret because several experimental factors are not controlled. In the continual-learning setting, the SF methods use task-vector conditioning and retain replay data across task switches, whereas DQN and DDPG receive no task identifier and reset their replay buffers. In addition, SF-Full produces zero or near-zero performance in several tasks without further analysis, and the continual-learning protocol contains only two alternating goals. These issues make it difficult to determine how much of the reported improvement comes from the proposed update and how well the conclusions extend to broader continual-learning settings.

Questions for authors
  1. How was η=0.99\eta=0.99 selected? Please report results for intermediate values, including η=0.5\eta=0.5, to clarify whether the improvement is specific to a setting close to the PBE endpoint.

  2. Can the authors provide direct evidence of the claimed stability effect across different values of η\eta, such as divergence frequency, task-vector norms, or TD-error variance?

  3. Can the authors compare SF-Bellman with DQN and DDPG under matched conditions, where all methods receive the same task information and use the same replay-buffer protocol?

  4. Why does SF-Full achieve zero or near-zero performance in several experiments? Please clarify whether this behavior is caused by the method itself or by its implementation and hyperparameter settings.

  5. Can the authors provide results on a longer task sequence or with more than two alternating goals to support the generality of the continual-learning conclusions?

2 comments

Q1. My main concern is that the experiments do not fully demonstrate the stability–approximation trade-off proposed in the paper. The online experiments only consider η=0,0.99,1.0\eta=0,0.99,1.0 without explaining how η\eta was selected or reporting results for intermediate values such as 0.50.5.

A1. We thank the reviewer for the comment on the choice of η\eta. Our choice of η=0.99\eta=0.99 was motivated by the theoretical results: higher η\eta can decrease the worst-case approximation-error bound, whereas lower η\eta can help stabilize the update. This motivated choosing a value close to the PBE while keeping the reward-regression part.

We additionally evaluated intermediate values of η\eta on DoorKey6×6 and KeyCorridorS2R1 using five seeds per environment. The results are provided in Common Response G3. Performance generally improves with higher η\eta.

We also evaluated an adaptation of OL-AUX [Lin et al., 2019] to adapt the relative weights of reward regression and the projected Bellman error during learning. This provides an alternative to selecting a fixed η\eta and showed improved performance. The results are provided in Common Response G4, with details in Appendix I of the revised manuscript.


Q2. They also do not directly measure instability, for example through divergence frequency, task-vector norms, or TD-error variance. Therefore, the current results show that one setting close to the PBE endpoint works well, but do not establish how the proposed interpolation behaves more generally.

A2. We thank the reviewer for the comment on directly examining instability. We have added a controlled example illustrating how approximation error and stability change as η\eta varies. In this simple cycle MDP, the approximation error decreases monotonically as η\eta increases, while a threshold separates the stable and unstable regimes. This illustrates the stability–approximation tradeoff within the same off-policy problem, complementing the benchmark performance results. The example is provided in Figure 4 and Appendix D of the revised manuscript and summarized in Common Response G2.


Q3. The broader empirical comparison is difficult to interpret because several experimental factors are not controlled. In the continual-learning setting, the SF methods use task-vector conditioning and retain replay data across task switches, whereas DQN and DDPG receive no task identifier and reset their replay buffers.

A3. We thank the reviewer for highlighting these differences in the experimental protocol. We have added comparisons under a matched protocol in which all methods reset their replay buffers at task-switch boundaries and operate without task-vector conditioning. The results in Common Response G4 use all the same prootocl and comparing the SF methods with DDPG are provided in Common Response G5.


Q4. In addition, SF-Full produces zero or near-zero performance in several tasks without further analysis.

A4. We thank the reviewer for the comments on SF-Full. SF-Full was evaluated using the same hyperparameter grid as SF-Bellman and SF-Simple. One possible explanation for its poorer performance is the difficulty of learning all dd dimensional successor-feature vector, whereas SF-Simple and SF-Bellman use scalar directions and losses. This makes the optimization more challenging, leading to poor performance of SF-Full.


Q5. Can the authors provide results on a longer task sequence or with more than two alternating goals to support the generality of the continual-learning conclusions?

A5. We thank the reviewer for the comments on diverse scenarios. We have added experiments with longer task sequences which considers increasing two cycles to four cycles, and experiments switching among three goals. The results are provided in Common Response G4 and Appendix I of the revised manuscript.


Reference

Xingyu Lin, Harjatin Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.

Thanks for the reply. I think the response kind of addess the concerns. Altouogh I still think it could be done better, I decide to raise the score accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response

We thank all the reviewers for their constructive comments which helped us improve the manuscript. We have uploaded a revised version with changes marked in blue. The major changes are summarized below.

G1. Geometric illustration of the difference between PRE and PBE solutions

We have added a geometric illustration of difference between PRE and PBE solutions in Figure 1 of the revised manuscript. Suppose the best approximation Ψπw∗\Psi_\pi w^* to QπQ^\pi satisfies

\qquad \Psi_\pi^\top DP^\pi(Q^\pi-\Psi_\pi w^*)=0.$$ That is, both the Q-function approximation error and its image under $P^\pi$ are orthogonal to the successor-feature space under the $D$-weighted inner product. If the resulting reward residual $R-\Phi w^*$ is not orthogonal to the basis-feature space, PRE introduces additional approximation error compared with PBE. ## **G2. Example illustrating the stability–approximation tradeoff** We have added an off-policy example illustrating the stability–approximation tradeoff in Figure 4 and Appendix D of the revised manuscript. In this simple cycle MDP, the approximation error decreases monotonically as $\eta$ increases, while a threshold separates the stable and unstable regimes. This example shows how the stability–approximation tradeoff can happen in the off-policy case. ## **G3. Sensitivity to $\eta$** The following results examine performance across a range of $\eta$ and feature dimension $d$ values. Higher values of $\eta$ generally perform better than lower values in these experiments. **Mean scores on DoorKey6×6 (DK) and KeyCorridorS2R1 (KC).** | $\eta$ | DK×6, $d= 32$ | DK×6, $d= 48$ | KC, $d=32$ | KCS2R1, $d=48$ | |---|---:|---:|---:|---:| | 0.0 | 0.874 ± 0.048 | 0.775 ± 0.082 | 0.430 ± 0.183 | 0.760 ± 0.125 | | 0.01 | 0.835 ± 0.097 | 0.826 ± 0.073 | 0.608 ± 0.176 | 0.711 ± 0.104 | | 0.1 | 0.862 ± 0.093 | 0.962 ± 0.011 | 0.843 ± 0.065 | 0.874 ± 0.032 | | 0.5 | 0.923 ± 0.030 | 0.929 ± 0.032 | 0.883 ± 0.058 | 0.880 ± 0.053 | | 0.9 | **0.988** ± 0.010 | 0.980 ± 0.012 | 0.767 ± 0.193 | 0.923 ± 0.038 | | 0.99 | 0.958 ± 0.029 | 0.890 ± 0.098 | **0.970** ± 0.009 | **0.993** ± 0.007 | | 1.0 | **0.986** ± 0.014 | **1.000** ± 0.000 | **0.962** ± 0.027 | 0.777 ± 0.117 | ## **G4. Adaptive method for choice of $\eta$ on longer cycle and increased number of tasks** The best choice of $\eta$ can depend on the environment. An alternative to using a fixed value of $\eta$ is to adapt it during learning. We evaluate an multi-objective tuning method called OL-AUX [Lin et al., 2019] becaue the reward regression and the projected Bellman error can be viewed as a multi-objective function. In most of the settings below, OL-AUX achieves a higher mean score than the $\eta=0$ baseline and DQN. The details are provided in Appendix I of the revised manuscript. **Scenario : Increase cycle : (two cycle-> four cycle. The goals alternate four times)**. | Environment | OL-AUX | $\eta=0$ | DQN | |---|---:|---:|---:| | DK8×8 | **100.0 ± 0.0** | 92.1 ± 6.7 | 83.3 ± 11.2 | | KC S2R1 | **100.0 ± 0.0** | **100.0 ± 0.0** | 83.3 ± 11.2 | | FR S13 | **99.2 ± 0.8** | 94.6 ± 4.2 | 97.1 ± 2.9 | **Scenario : Increased alternating goals (two goals->three goals) and last cycle final evaluation metric** | Environment | Number of tasks | OL-AUX | $\eta=0$ | DQN | |---|---:|---:|---:|---:| | DK8×8 | 2 | **75.0 ± 13.1** | **80.0 ± 11.3** | 41.7 ± 14.9 | | DK8×8 | 3 | **100.0 ± 0.0** | 91.7 ± 8.3 | 61.1 ± 14.1 | | KC S2R1 | 2 | **80.4 ± 11.2** | **70.8 ± 11.4** | 66.3 ± 14.1 | | KC S2R1 | 3 | **98.9 ± 1.1** | 77.8 ± 11.8 | 58.3 ± 14.9 | | FR S13 | 2 | **94.6 ± 4.2** | **91.7 ± 8.3** | 86.7 ± 8.9 | | FR S13 | 3 | **93.1 ± 3.8** | 89.2 ± 8.3 | 79.7 ± 11.1 | **Reference:** Xingyu Lin, Harjatin Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. *Advances in Neural Information Processing Systems*, 32, 2019. ## **G5. DDPG comparison under a matched protocol** We report results under a matched protocol in which all methods reset their replay buffers at task-switch boundaries and operate without task-vector conditioning. Under the same matched protocol, $\eta=0.99$ shows in general improved or comparable performance. | Method | PointMaze | Reacher | PandaReach | |---|---:|---:|---:| | DDPG | 36.310 ± 16.659 | 31.033 ± 3.679 | **27.167** ± 3.382 | | SF, $\eta=0$ | **101.470** ± 38.365 | 33.753 ± 4.871 | 13.483 ± 4.905 | | SF, $\eta=0.99$ | 34.447 ± 17.169 | **42.880** ± 0.814 | **28.983** ± 4.837 | | SF, $\eta=1.0$ | **68.665** ± 23.049 | 28.632 ± 6.379 | 23.917 ± 5.771 |

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.