acceptodds
Under review as a conference paper at ICLR 2027

The Terminal Representation in Reinforcement Learning

Abstract

Representation learning is a powerful tool for spatio-temporal abstraction within reinforcement learning (RL). Two well established approaches are through the successor representation (SR) and the default representation (DR). The SR encodes states by the future trajectories they induce, capturing information flow decoupled from reward. The DR builds on this by weighting trajectories with reward, integrating credit-assignment structure into the representation. Eigenvectors of both representations have been used to support a range of downstream tasks—including option discovery, reward shaping, transfer learning, and exploration. We introduce a structurally distinct formulation: the terminal representation (TR). The TR encodes reward-weighted trajectories similarly to the DR, but can be learned as a lower-dimensionality object, and can be used directly for the mentioned applications without eigenvector computations. Eigendecomposition also imposes the assumption of symmetric transition dynamics, which the TR can bypass. In this work we develop the theoretical foundations of the TR: its derivation, convergence of two learning algorithms, its use for zero-shot compositionality, and equivalences between alternative reward formulations. We further show the TR is embedded in the top DR eigenvector, allowing it to capture the same underlying knowledge without eigendecomposition. Additionally, we provide empirical evidence of the TR as a viable alternative to existing representations in subsidiary applications, while requiring less computational overhead to learn, store, and use.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Representation, Composition, Hierarchy, Structure, LMDP

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cQdQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In linearly-solvable MDPs with terminal states, the exponentiated optimal value is linear in the exponentiated terminal rewards, z = My. The authors call the S×T matrix M the terminal representation (TR), give dynamic-programming and TD updates for M with convergence proofs, show that the top eigenvector of the default representation (DR) expands around a column of M (Theorem 5.1), and rerun the Tse et al. (2025) experiments with columns of M in place of DR/SR eigenvectors.

Critical strengths

The proofs appear correct. Theorem 5.1 is a useful observation: with absorbing terminal states the DR is block-triangular, so the leading term of the DR’s top eigenvector is a column of M. That result explains what reward-aware eigenvector methods compute in goal-based tasks and justifies replacing the eigenvector with a column of M.

Critical weaknesses

The object introduced as the TR is not new, and the efficiency advantage claimed for the TR is never measured. Section 2 already gives z = Z_S P_T y, citing Piray & Daw (2021) and Tse et al. (2025), with P_T assumed known. What Section 4 adds is to learn the product M = Z_S P_T directly by TD, so P_T is never estimated, and to use M as the representation. Both halves of that step already exist. Learning M directly: Eq. 5 is Todorov’s (2006) Z-learning update run once per terminal state with boundary value 1 at that state (M itself appears in eq. 22 of Todorov), and each column of Saxe et al.’s (2017, §2.3) desirability basis matrix is the solution of exactly that one-goal task, with their composition rule z = Zw being Theorem 6.3. Using M as the representation: default features (Tse et al., 2025) learn Z_S Φ directly for a chosen terminal feature matrix Φ, and with one-hot terminal features Φ = I that product is M, as the authors themselves say in §3 and §7. “Novel method” (§8) therefore has weak support. The abstract claims lower cost to learn, store and use, but no timing, memory or sample-efficiency numbers are reported, and in tabular gridworlds on a laptop (App. D) the avoided eigendecomposition costs almost nothing. Performance is parity at best: on SixArms the TR scores 2,443 ± 94 against 3,127 ± 214 for the DR (Table 1). The exploration experiments also use a discounted TR (γ = 0.95, Tables 8–9) that no theorem covers, in tasks with no terminal states. Measured cost and sample-efficiency advantages at scale, with the contribution reframed around Theorem 5.1, would move my rating to 2/4.

Questions for authors
  1. What does the TR add beyond default features with one-hot terminal features (Tse et al., 2025) and the desirability basis matrix of Saxe et al. (2017)? Does the default-features baseline in Figure 3 use the same one-hot features as the TR?

  2. Can you report wall-clock time, memory or sample-efficiency comparisons between the TR and the DR, in a setting where the eigendecomposition is not negligible?

  3. Define the discounted TR used in Tables 8–9, state which theorems hold for the discounted TR, and state which states were treated as terminal in RiverSwim and SixArms.

Reviewer_xBhCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes the terminal representation (TR), a representation similar to the successor representation and default representation that exploits reward and terminal-state information. Compared to the default representation, it is more compact, and can be used for different applications without eigendecomposition or assuming symmetry of transition dynamics. The paper demonstrates that the terminal representation has competitive performance compared to other representations in a broad range of applications, while having the above mentioned benefits.

Critical strengths
  • The proposed terminal representation has meaningful benefits compared to the default representation, such as not needing eigendecomposition for some applications, and being more compact.
  • There is a breadth of experiments demonstrating the applicability of the TR in a wide variety of tasks.
  • The paper is in general well-written and clear.
Critical weaknesses
  • Novelty
    • The paper acknowledges in related work that the TR can be interpreted as a special case of default features. I notice during a quick scan of the cited paper that they have TD update rules for default features. How is the one derived in this paper (Eq. 5) different from the one in the cited paper, cause they look similar to me. You might want to derive these rules in this paper specifically for the TR for completeness, but a quick reference would help readers make the connection. It would also help to clarify which of the learning results (e.g., TD convergence) are new relative to the default features formulation.
  • Experiments
    • The experiments do not seem to directly reflect the structural benefits of the TR highlighted in Section 5. For example, since the TR does not need eigendecomposition, it should be much more efficient compared to the DR in terms of run time. Couldn't you measure the run time for the SR, DR, and TR in some of the applications? You can even scale the size of the environments, then the run time of the SR and DR, which rely on eigendecomposition, would significantly increase. Having the run time vs environment size as a plot would clearly demonstrate the benefit of the TR.
    • Also, to highlight the benefit of the TR in handling asymmetric environments, couldn't you use some asymmetric environment, e.g., a four rooms with a one-way door? Since the DR is symmetrized before eigendecomposition, I would guess that it does not respect the one-way door. If you have a goal placed behind a one-way door, the DR might still encourage the agent to bump into the one-way door while the TR would encourage the agent to take another path that does not pass through the one-way door.

Happy to raise my score if these are addressed.

Questions for authors
  1. I struggle to understand the "Hierarchy" paragraph in Section 5:
    1. The use case of the TR in the hierarchical RL setting seems to require knowing in advance what the terminal states of the subproblems are. How often is this true in practice, especially when the subproblems are learned? It seems a pretty limiting assumption to me.
    2. I also do not fully understand why the DR cannot be used to zero-shot compute the subproblem value functions given a set of terminal rewards. Is it because the DR is not learned with the subproblem terminal states in mind, and has to learn PTP_T? I suspect, however, that PTP_T should be quite easy and cheap to estimate, especially when you keep a buffer of transitions.
    3. For the TR, it seems that you need to learn columns for both terminal states of the original problem and terminal states for the subproblem (assuming that you know what they are). Don't you have to learn quite some columns then? Wouldn't that impact your claim about TR's compactness?
Reviewer_YSEzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work introduces the terminal representation for linearly solvable Markov decision processes: the matrix M = (I - D_S)^-1 D_T of size S x T, for S non-terminal and T terminal states, which encodes reward-weighted reachability of terminal states from non-terminal states. Given any terminal rewards, M recovers the optimal values zero-shot through z = My. The paper proves convergence of a dynamic-programming and a sample-based learning rule, extends the terminal representation to continuous states and state-action rewards, shows how it supports compositionality, and shows that a column of the terminal representation appears inside the top eigenvector of the default representation. The experiments in small tabular environments, following the paper Tse et al. [25], compare the terminal representation with the successor representation and the default representation for option discovery, reward shaping, count-based exploration and transfer, and find it broadly competitive while avoiding eigen decomposition.

Critical strengths
  1. The terminal representation is a simple, well-defined object with the theory. Theorems 4.1 and 4.2 give dynamic-programming and sample-based learning rules with convergence guarantees, and Appendix A.4 extends the construction to continuous state spaces. The paper also covers state-action rewards (Theorem 6.2) and transition-dependent rewards (Theorem 6.1), so the terminal representation applies across the usual reward formulations.

  2. The terminal representation is compact and directly usable. It is S x T rather than S+ x S+ over all states, its columns are used without any eigen decomposition, and z = My recovers optimal values for any terminal reward configuration. The online target in Equation 5 also avoids estimating the terminal transition kernel, which is a practical advantage for hierarchical settings with shared terminal states.

  3. The experiments follow an established protocol. They reuse the setup and code of Tse et al. [25] across four applications. The terminal representation matches the default representation in reward shaping and in RiverSwim exploration and matches default features in transfer (Figure 3), which supports the claim that it is a usable substitute.

Critical weaknesses
  1. The novelty is limited. The terminal representation is M = Z_S P_T, where Z_S is the non-terminal block of the default representation and P_T holds the transitions from non-terminal to terminal states. The background section already uses z = Z_S P_T y from Piray and Daw [18] and Tse et al. [25] for zero-shot value recovery. The paper itself states that the terminal representation is a special case of default features with one-hot terminal features (Section 3). Theorem 6.3 is Todorov's compositionality with a one-hot terminal basis, and it restates z = My. Theorems 4.1 and 4.2 are a standard Neumann-series argument and a standard stochastic-approximation result. I find the novelty to be incremental to be honest.

  2. The eigenvector link and the reversibility advantage are stated more strongly than they are shown. Theorem 5.1 needs e^{R(tau*)/lambda} > rho(Z_S), where rho is the spectral radius, tau* is the highest-reward terminal state and lambda is the temperature of the KL penalty. When per-step costs are small relative to lambda, the sufficient bound gamma/(1 - gamma), with gamma = max_s e^{R(s)/lambda}, becomes large (about 9.5 for R(s)/lambda = -0.1), so the condition can fail; the top eigenvector then comes from Z_S and has no terminal component. Even when the condition holds, v*S = (I - Z_S/beta*)^-1 M{.,tau*} with beta* = e^{R(tau*)/lambda}, which matches the column of the terminal representation only at first order, and the paper does not measure how close they are. On reversibility, Z is entry wise non-negative, so the Perron-Frobenius theorem already guarantees a real top eigenvalue with a non-negative eigenvector without reversibility. Since the applications use only the top eigenvector, the advantage claimed in Section 5 is weaker than presented. Finally, the O(1) query cost ignores the cost of learning M, which is similar to learning the default representation when computed exactly.

  3. The experimental evidence is limited and partly inconsistent with the theory. All experiments are small and tabular, with no function approximation despite the claim that the terminal representation is a natural target for it. The option-discovery environment was modified to make the goals terminal with rewards from +1 to +4, explicitly to create differences between the terminal-representation variants. The terminal representation also received an 80-configuration random search, while the successor and default representations use previously reported hyperparameters. On SixArms the terminal representation is clearly worse than the default representation (2,443 versus 3,127, with non-overlapping intervals), although it is described as competitive. I feel the abstract claims lower computational overhead, but no runtime or memory measurements are reported.

  4. Several technical details need correction. Theorem 4.2 omits the requirement that every state is updated infinitely often, which the asynchronous result of Tsitsiklis [26] needs. Theorem 6.2 needs negative state-action rewards for the inverse to exist, and Theorem 5.1 needs a unique highest terminal reward for the top eigenvector to be unique. The shaping reward 1 - M(s_{t+1}, tau) depends only on the next state rather than on a difference of potentials, so it is not potential-based in the sense of Ng et al. [17]. As written, it is also larger far from the goal than near it.

Questions for authors
  1. Beyond the naming, what does the terminal representation add over default features with one-hot terminal features, and over the use of z = Z_S P_T y in prior work on the default representation?
  2. In RiverSwim and SixArms, which states are treated as terminal, and how does the terminal-representation discount of 0.95 fit the undiscounted theory?
  3. Is the shaping reward 1 - M(s_{t+1}, tau) intended to be potential-based? Please check its sign and form.
  4. In the experiments, does the condition of Theorem 5.1 hold, and how closely does the column of the terminal representation match the actual top eigenvector of the default representation?
  5. Could you report runtime and memory for learning and using each representation, and results with function approximation in at least one larger environment?
  6. How do covering eigen options and reward-aware covering eigen options perform on the modified option-discovery environment compared with the original one, to show that the modification does not favour the terminal representation? I would be willing to raise my score if my concerns are addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.