- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies successor features (SFs), which decompose the Q-function as , where is the expected discounted sum of features and is a task vector. The authors consider two existing approaches to estimating , (i) the projected reward equation (PRE), which fits by regressing immediate rewards onto the features, and (ii) the projected Bellman equation (PBE), which fits as the fixed point of the projected Bellman operator over the span of , and aim to bridge them using theoretical results. They show that both belong to a single family of estimators parameterized by an interpolation parameter , with PRE at and PBE at , and characterize a tradeoff in which PRE favors stability while PBE yields a tighter approximation guarantee. They then evaluate the resulting algorithm, SF-Bellman, empirically on continual-learning and goal-conditioned control benchmarks, comparing different values of against SF and standard RL baselines.
Critical strengths
- The paper gives a thorough background on successor features and a clear literature review. Table 1 summarizes prior SF methods by their task-vector objective and SF learning scheme, which makes prior contributions and this paper's position easy to follow.
- The Cafe–Bakery example clearly illustrates how PRE and PBE can lead to different value estimates and decisions.
- The experiments evaluate several values of on both continual-learning and goal-conditioned tasks, complementing the theory with empirical evidence.
Critical weaknesses
- There is a gap between theory and algorithm, as the theory assumes fixed features, a fixed successor-feature map, and a fixed policy.
- The two sides of the tradeoff are proven under different assumptions.
- The non-SF baselines receive no task information and reset their replay buffers at every task switch, and no baseline uses HER, the standard remedy for sparse goal-reaching rewards.
Questions for authors
-
Regarding Corollary 3.3, the condition requires the expected next feature to be a linear function of the current feature under . Beyond tabular or eigenvector features, when does this hold in practice? Since it depends on , it seems unlikely to hold when features are learned and the policy changes. Could the authors comment on its practical relevance?
-
All theoretical results assume fixed features , a fixed successor-feature map , and a fixed policy (policy evaluation), whereas SF-Bellman learns and online while the policy changes (control with a ). The theory therefore does not directly explain the empirical gains. Moreover, the approximation guarantee (Prop. 4.2) assumes on-policy (stationary) data, whereas the stability benefit of reward regression (Lemma 4.4) matters only off-policy, so the tradeoff is never shown within a single setting.
-
Regarding the Cafe–Bakery example (lines 196–199), from Appendix C (, , stationary ), we obtain:
3 comments
Q7. In the goal-conditioned experiments, the goal is observed and rewards are sparse, which is the standard setting for HER; DDPG+HER reaches 100% success on standard PandaReach (Gallouédec et al., 2021). Why is HER not used? Since it is orthogonal to the SF parameterization, could the authors add it to both SF-Bellman and the baselines, or justify its omission? Could they also report success rates?
A7. We thank the reviewer for the suggestion to apply HER. The following results show performance with HER, and, in general, higher yields improved or comparable performance. The following reports the held-out goal return and denotes the relabelling ratio.
\text{Method} & \text{HER } k & \text{PointMaze} & \text{Reacher} & \text{PandaReach} \\\\ \hline \text{DDPG} & 4 & \mathbf{270.61} \pm 0.49 & 4.59 \pm 4.33 & 29.90 \pm 3.88 \\\\ \text{SF } \eta=0 & 4 & 238.31 \pm 19.08 & \mathbf{42.12} \pm 0.74 & 30.90 \pm 1.27 \\\\ \text{SF } \eta=0.99 & 4 & \mathbf{270.75} \pm 0.69 & 34.36 \pm 8.13 & \mathbf{34.70} \pm 0.12 \\\\ \text{SF } \eta=1.0 & 4 & 259.77 \pm 7.47 & 37.92 \pm 3.05 & 33.35 \pm 1.07 \\\\ \hline \text{DDPG} & 8 & 269.38 \pm 1.27 & 17.20 \pm 7.82 & 27.15 \pm 5.44 \\\\ \text{SF } \eta=0 & 8 & 255.13 \pm 9.84 & 39.74 \pm 1.79 & 31.60 \pm 1.42 \\\\ \text{SF } \eta=0.99 & 8 & \mathbf{271.40} \pm 0.82 & \mathbf{42.96} \pm 0.07 & \mathbf{35.50} \pm 1.18 \\\\ \text{SF } \eta=1.0 & 8 & 239.94 \pm 14.04 & 42.72 \pm 0.37 & 31.15 \pm 1.64 \\\\ \hline \end{array}$$ Moreover, most of the methods achieve near $100\%$ success rate while we aim to solve a more difficult problem of to reach and stabilize at the goal. **Q8.** *Is there an approximation guarantee for off-policy data, or an experiment showing both sides of the tradeoff in the same setting?* **A8.** We thank the reviewer for the constructive comment on approximation guarantees for the off-policy case. We can derive an approximation error bound following Yu and Bertsekas, but the error bound is usually loose and does not provide much information. Similar guarantees would be possible, but, as shown in the lemma, the exact characterization of the error bound is highly nonlinear. Yu, Huizhen, and Dimitri P. Bertsekas. “Error bounds for approximations from projected linear equations.” Mathematics of Operations Research 35.2 (2010): 306–329. **Q9.** *Algorithms 2–5 contain no update for the parameters of $\phi$. How is $\phi$ trained?* **A9.** We thank the reviewer for clarification on $\phi$. $\phi$ is obtained by $l_2$-normalizing the output of the encoder, which is shared with $\psi$. The stop-gradient operation is applied to $\phi$ whenever a backward operation is performed.I thank the authors for the rebuttal, I'm keeping my positive score.
Q1. There is a gap between theory and algorithm, as the theory assumes fixed features, a fixed successor-feature map, and a fixed policy.
A1. We thank the reviewer for the constructive comments on the gap between the theory and the algorithm. As the reviewer mentioned, the theory provides guarantees only in settings with fixed features and a fixed behavior policy. This is intentional, to separate the complications in the analysis arising from time-varying features or policies. The analysis under these simple assumptions provides clear intuition and analytical tools for SF theory. Extending the analysis to time-varying features and policies would be an important future direction.
Q2. The two sides of the tradeoff are proven under different assumptions.
A2. We thank the reviewer for the valuable comments on the tradeoff. We chose the term “tradeoff” to clarify the roles of reward regression and projected Bellman error. Indeed, the stability problem only applies to the off-policy case, as stability is generally guaranteed in the on-policy case with linear function approximation. Therefore, we have added a new example showing the stability–approximation tradeoff in the off-policy case in Figure 4 and Appendix D of the revised manuscript. This example uses a simple cycle MDP, and the approximation error decreases monotonically as η increases, while there is a threshold that characterizes stability.
Q3. The non-SF baselines receive no task information and reset their replay buffers at every task switch, and no baseline uses HER, the standard remedy for sparse goal-reaching rewards.
A3. We thank the reviewer for the comments on the experiments. We additionally included results without task-vector conditioning and with the same replay buffer reset protocol in common response G3 and G5. The following results show improved or comparable performance with higher η under the same protocols.
Q4. Regarding Corollary 3.3, the condition requires the expected next feature to be a linear function of the current feature under π. Beyond tabular or eigenvector features, when does this hold in practice? Since it depends on π, it seems unlikely to hold when features are learned and the policy changes. Could the authors comment on its practical relevance?
A4. We thank the reviewer for the comment on Corollary 3.3 and its application in practice. As the reviewer mentioned, the condition can be difficult to meet in practice. Satisfying this condition requires additional methods [Ghosh, 2020]. Our goal is not to meet this condition. We rather aim to design an objective that considers both objectives.
Ghosh, Dibya, and Marc G. Bellemare. “Representations for stable off-policy reinforcement learning.” International Conference on Machine Learning. PMLR, 2020.
Q6. Robustness of the Cafe-Bakery example.
A6. We thank the reviewer for the clarification on the Cafe-Bakery example. We can derive a regime in (ϵ,γ) in which PBE chooses the correct answer. For example, when ϵ=0.1, we require γ≥0.895. This also highlights that a high γ is required to reflect the future dynamics. Moreover, PBE provides more accurate estiamte not only correct ranking.