- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In linearly-solvable MDPs with terminal states, the exponentiated optimal value is linear in the exponentiated terminal rewards, z = My. The authors call the S×T matrix M the terminal representation (TR), give dynamic-programming and TD updates for M with convergence proofs, show that the top eigenvector of the default representation (DR) expands around a column of M (Theorem 5.1), and rerun the Tse et al. (2025) experiments with columns of M in place of DR/SR eigenvectors.
Critical strengths
The proofs appear correct. Theorem 5.1 is a useful observation: with absorbing terminal states the DR is block-triangular, so the leading term of the DR’s top eigenvector is a column of M. That result explains what reward-aware eigenvector methods compute in goal-based tasks and justifies replacing the eigenvector with a column of M.
Critical weaknesses
The object introduced as the TR is not new, and the efficiency advantage claimed for the TR is never measured. Section 2 already gives z = Z_S P_T y, citing Piray & Daw (2021) and Tse et al. (2025), with P_T assumed known. What Section 4 adds is to learn the product M = Z_S P_T directly by TD, so P_T is never estimated, and to use M as the representation. Both halves of that step already exist. Learning M directly: Eq. 5 is Todorov’s (2006) Z-learning update run once per terminal state with boundary value 1 at that state (M itself appears in eq. 22 of Todorov), and each column of Saxe et al.’s (2017, §2.3) desirability basis matrix is the solution of exactly that one-goal task, with their composition rule z = Zw being Theorem 6.3. Using M as the representation: default features (Tse et al., 2025) learn Z_S Φ directly for a chosen terminal feature matrix Φ, and with one-hot terminal features Φ = I that product is M, as the authors themselves say in §3 and §7. “Novel method” (§8) therefore has weak support. The abstract claims lower cost to learn, store and use, but no timing, memory or sample-efficiency numbers are reported, and in tabular gridworlds on a laptop (App. D) the avoided eigendecomposition costs almost nothing. Performance is parity at best: on SixArms the TR scores 2,443 ± 94 against 3,127 ± 214 for the DR (Table 1). The exploration experiments also use a discounted TR (γ = 0.95, Tables 8–9) that no theorem covers, in tasks with no terminal states. Measured cost and sample-efficiency advantages at scale, with the contribution reframed around Theorem 5.1, would move my rating to 2/4.
Questions for authors
-
What does the TR add beyond default features with one-hot terminal features (Tse et al., 2025) and the desirability basis matrix of Saxe et al. (2017)? Does the default-features baseline in Figure 3 use the same one-hot features as the TR?
-
Can you report wall-clock time, memory or sample-efficiency comparisons between the TR and the DR, in a setting where the eigendecomposition is not negligible?
-
Define the discounted TR used in Tables 8–9, state which theorems hold for the discounted TR, and state which states were treated as terminal in RiverSwim and SixArms.