acceptodds
Under review as a conference paper at ICLR 2027

SUAF: Spectral Universal Advantage Features for Zero-Shot Policy Improvement

Abstract

Reward-independent representations enable reinforcement learning agents to reuse knowledge across tasks, but limited representation capacity raises three fundamental questions: what statistical dependence should these representations capture, which components of that dependence should be retained for policy improvement, and how does discarding the remaining components affect decision-making? We address these questions through canonical dependence analysis. For a reference policy, we construct an action-centered, positive-lag canonical dependence matrix (CDM) that isolates the additional predictive contribution of the current action beyond the current state. Its singular decomposition yields Spectral Universal Advantage Features (SUAF), paired features that recover delayed advantage through linear reward projections. We show that the spectrum quantifies action-incremental -squared dependence and determines reward-specific approximation errors. Retaining the leading modes gives optimal rank-constrained approximations under both isotropic average-case and reward-uniform minimax criteria. Separating the immediate reward contrast also avoids the spectral floor arising from direct compression of the full advantage operator. To learn SUAF from reward-free data, we develop an action-centered Soft-HGR objective with bidirectional temporal-difference estimation. We further connect spectral approximation errors to sufficient conditions for policy improvement. Finite-state experiments validate the spectral predictions and feature-learning method, while zero-shot evaluations on discrete tasks and continuous-control benchmarks demonstrate the utility of the learned representations across rewards. abstract

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Information Theory, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_adVPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the representations learned for zero-shot reinforcement learning. Compared to most existing methods, which learn representations to represent the Q-function, the paper proposes to learn representations to represent the delayed advantage function (which, compared to a normal advantage function, omits the immediate reward) instead. The paper evaluates the resulting method, spectral universal advantage features, in a number of discrete and continuous domains.

Critical strengths

None

Critical weaknesses
  1. Concerns about AI Use
    1. While the authors claim that "Every reference surfaced with AI assistance was located and checked against the original source by the authors before inclusion", there are blatant errors in the references.
      1. One of the references appear twice: "Barreto et al. Successor features for transfer in reinforcement learning." appears twice in the references, with years 2017a and 2017b
      2. One of the references is simply wrong: The paper cites Brillinger (1964) for Pinsker's inequality. The correct citation should have been the book: "Mark S. Pinsker. Information and Information Stability of Random Variables and Processes. Holden-Day Series in Time Series Analysis. Holden-Day, San Francisco, 1964." Yet, the authors cited "David R. Brillinger. Information and information stability of random variables and processes. Journal of the Royal Statistical Society Series C: Applied Statistics , 13(2):134–135, 06 1964." which is, in fact, just a two-page review on the book which has nothing to do with Pinsker's inequality.
    2. I would have expected the authors to catch these errors if AI generated references were truly "checked against the original source by the authors". This casts significant concerns about whether the authors have truly verified AI generated content. A simple AI query revealed other errors. Among these errors, the following notably adds to the concern:
      1. In line 3349, the paper writes "The upstream source and revision used for the final runs should be archived with the experiment artifacts." This seems to me a note generated by AI that the authors forgot to delete.
  2. Clarity
    1. The paper is very dense and hard to parse. For example, the third paragraph of the introduction is just a half-page block of text that describes details of the contribution of the work that, at this point, no readers can understand since they have not even read the preliminaries section. Like, what is action-incremental χ2\chi^2-dependence? What is action-centered Soft-HGR objective? Even worse, I don't think I even see the term χ2\chi^2-dependence appear again anywhere in the rest of the paper.
    2. Section 3 and 5 immediately jumps into the content without first describing the purpose/overview of the section. For example, the first sentence of Section 3 is "We establish the SUAF with dependent CDM." Similarly, Section 5 immediate jumps into experiment setup without even first mentioning the purpose of the experiments.
    3. The paper presents a large amount of theoretical results. However, there is in general insufficient explanation for what the theoretical results mean. For example, Theorem 4.3 is simply stated without any discussion in text.
  3. Experiment results
    1. Looking at Table 2, there does not seems to be clear performance improvements with the proposed method in complex domains, casting doubt on the significance on the proposed method.
Questions for authors

None

4 comments
AuthorsOpenReview ↗

Gentle Reminder: Discussion Period Approaching Deadline

Dear Reviewer,

Hope you are doing well. As the discussion period is drawing to a close, we would really appreciate it if you could review our rebuttal and revised submission soon. We have responded point-by-point to your feedback and added new results as requested.

If you think our responses adequately resolve your concerns, we kindly ask you to consider updating your evaluation. Feel free to raise any remaining questions, and we will reply accordingly.

We sincerely appreciate your valuable time.

Best regards, Authors

AuthorsOpenReview ↗

Follow-up on Rebuttal and Discussion: Deadline Today

Dear Reviewer,

Hope you are doing well. As the discussion period closes today, we would like to kindly follow up on our rebuttal and revised submission. We have responded point by point to your comments and added the requested results.

We would be very grateful if you could take a moment to review our responses and let us know whether any concerns remain. If our replies have adequately addressed your points, we would kindly ask you to consider updating your evaluation before the deadline.

We are monitoring the discussion thread closely and will respond to any further questions or requests for clarification.

Thank you very much for your time and consideration.

Best regards,

Authors

Thank you for the response.

In my initial review, I raised concerns about clarity, stating that the paper is very dense and hard to parse. The authors have provided a revision in an attempt to address this issue. However, I don't think my concern on clarity has been addressed. Below, I use the third paragraph of the Introduction as an example to illustrate why I struggle to understand the paper.

  • Sentence 4: After introducing "action-centered, positive-lag canonical dependence matrix", shouldn't the authors explain what it is, before starting to talk about its singular value decomposition?
    • In order to understand the sentence, I had to ask an LLM: (1) What is a CDM?, (2) What does positive-lag refer to?
    • A well-written paper should be digestible for a human reader without them having to constantly google/ask an LLM what specific terms mean.
  • Sentence 5: "value" missing from singular "value" decomposition.
  • Sentence 7: You write "To determine which components to retain". Actually, up to this point, I still have no idea what a "component" means. You briefly mentioned that at the end of the second paragraph ("which components of that dependence should be retained") but with no explanation.
    • Again, I had to ask an LLM to finally get that you're referring to one of the modes of the SVD of the canonical dependence matrix.
  • Sentence 8: What is a "nonzero spectral floor"? Again I had to ask an LLM to understand this.
  • Sentence 9: How is this result relevant to the question of which components to retain, as raised in Sentence 7?
  • Sentence 10: You say that retaining the leading modes of the SVD gives some optimal approximation. Does that mean we just keep the leading "components" then? If yes, a sentence confirming that you indeed retain the leading components based on this finding is missing. Also, "What's more" is pretty informal, and I would avoid using this phrase.
  • Sentence 7 brings up the problem of determining which components to retain, and sentences 8-10 are supposed to explain what components to retain and why. But after reading these sentences I still am not fully sure which components you end up retaining and why. I can only guess based on sentence 10, but readers should not be left guessing.
  • Sentence 12: "The error depends on ..." What error? The error between what? Do you mean the error between the ground-truth CDM and the CDM approximated using a limited number of modes? Again, readers shouldn't be left guessing.
  • Sentence 13: What is "aggregate lost dependence"? What is "task-specific information loss"? What is "approximation penalty"? Is there a penalty for approximation errors?

I hope that with this more detailed breakdown, I have demonstrated why I think that this paper has significant clarity issues. I believe that (1) a clear paper should be digestible by an independent human reader without heavy assistance from search engines/LLMs, and (2) a clear paper should be explicit to minimize ambiguity. Unfortunately, this paper falls short in both regards. Since clarity is a prerequisite for evaluating other acceptance criteria, I will maintain my score of clear rejection.

Thank you for identifying the specific sentences. We have revised the paragraph in light of your suggestions. We would also like to explain the reasoning behind the wording you highlighted.

  • Sentence 4: We use canonical dependence matrix as an established term in information theory and cite the paper proposed this idea in the same sentence. Besides, we have introduced the underlying idea before giving the matrix its name. Sentence 3 explains that we subtract “the policy-averaged future visitation” to isolate how the current action changes future outcomes and sentence 4 names the matrix that captures this dependence.
  • Sentence 5: “Singular value decomposition.” We agreed. The word value has been added.
  • Sentence 7: Since sentence 5 introduces the singular decomposition and its “full expansion”, a natural problem is “which components to retain” because we are moving to a low-rank approximation when representation budget is limited. In this case, the components are the individual modes in the expansion.
  • Sentence 8: The original sentence states that the immediate term induces “a nonzero spectral floor, limiting direct low-rank compression.” The words immediately following floor identify that it is a limit on the low-rank compression of the advantage operator at low rank. Since we are using SVD, the floor is actually a spectral floor.
  • Sentence 9: The original sentence states that, under weak dependence, “the sum of squared singular values approximates the mutual information.” Its relevance to mode selection follows directly from that sum. Discarding a mode removes its squared singular value from the approximation, so the sum over the discarded modes measures how much of the approximated information is lost. At a fixed rank, retaining the leading modes minimizes that loss. This is precisely why the information-theoretic result belongs in the discussion of which components to retain.
  • Sentence 10: SUAF is the leading components. The sentence states that “retaining leading modes gives an optimal rank-constrained approximation of delayed advantage under both average and worst-case reward objectives.” This identifies both the modes and the reason for selecting them. We can make the method choice more direct by using “SUAF retains the leading modes” as the main clause.
  • Sentence 7-10: We organize the sentences under this consideration: sentence 7 asks “which components to retain.”, sentence 8 narrows the choice to the delayed component by stating that SUAF keeps the evaluable immediate contrast explicit and sentence 10 then answers the question in the same terms in which it was posed: “retaining leading modes gives an optimal rank-constrained approximation of delayed advantage under both average and worst-case reward objectives.” The words “retaining leading modes” identify the choice, and “optimal … under both average and worst-case reward objectives” gives the reason.
  • Sentence 12: Sentence 10 identifies delayed advantage as the quantity being approximated. Sentence 11 then asks what happens when modes are discarded. Thus, for a given reward, “the error” is the difference between its exact delayed advantage and the approximation obtained from the retained modes. The distinction from CDM approximation error matters here. The same truncated CDM is used across rewards, while the resulting advantage error can differ from one reward to another.
  • Sentence 13: We distinguish the “aggregate lost dependence” and “task-specific information loss” because a discarded mode can carry substantial dependence yet contribute nothing to the delayed advantage for a particular reward. That is beacuse as we stated early, the advantage is actually a projection of reward ("... recovers delayed advantage through linear reward projections"). The approximation penalty therefore refers to the resulting approximation loss. The sentence says how it enters that condition: the predicted gain must exceed the loss due to truncation.

We appreciate your point that readers should be able to follow these connections on a first reading. At the same time, we respectfully believe that a clear rejection is too strong on the grounds given. Actually, we have the response to this concerns in this paper. We hope the clarifications above and the resubmitted revision are helpful and would be grateful if you could weigh the paper’s technical contributions alongside your concerns about presentation.

Reviewer_SjNjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary
  • The authors examine a reward-independent representation for RL, based on the successor measure and canonical dependence matrix.
  • In essence, rather than predict, say, transitions, the authors predict a (policy-dependent) representation which characterizes the future state distribution after an action. This is naturally used in computing advantages when the reward function is known.
  • They have several theoretical results, covering an exact spectral representation of the advantage, optimality of rank-k truncation, a "spectral floor" argument for computing the immediate reward term exactly rather than compressing it, and sufficient conditions for policy improvement with the truncated critic.
  • Their experiments cover several finite state environments and some more complex continuous problems.
Critical strengths
  • The theoretical characterization of SUAF and its properties is extensive, and seems to reveal a novel framework for reward-free representation learning with several useful capabilities. This is important, as adapting to new tasks is an essential problem in modern AI.
  • The theoretical formulation has several elegant aspects, such as the decomposition of advantage and its relation to information-theoretic structure of the state-action space.
  • I like that the experimental sections are candid about the strengths and weaknesses of the approach.
Critical weaknesses
  • The scalability of the approach is insufficiently discussed. In other words, it was not clear to me that this framework is the right approach as the state, action, and reward spaces increase in complexity. Compared to modelling transitions, for instance, what are the intuitive advantages of SUAF? When would we expect it to be better or worse? This approach seems less versatile than "world models", as an example, as it has this policy dependence and its interaction with distribution of downstream tasks. Empirically, if I understood correctly, performance relative to the baselines does not clearly improve with more complexity.

  • The second concern I had was about the fair comparison empirically. In terms of assumptions, it was unclear to me whether SUAF had unfair access to oracle knowledge. Appendix D.1 assumptions suggest the method is allowed to compute exact expectations over actions (for reward contrast and features) and use the ref policy probs for this as well - is this fair wrt the baselines? Also, regarding capacity, did all methods have similar representational capabilities?

Questions for authors
  1. The method seems to provide an improvement that depends heavily on a behaviour policy and solving a non-trivial spectral problem. Is there reason to expect the approach to scale to larger state/action spaces and longer horizons, and to be preferred over transition-based representations (or others) in that regime? Do you have theoretical or empirical evidence for this?

  2. Regarding the experiments, can the authors clarify whether the baselines had access to the same capacity and information as SUAF? Is there an advantage from knowing the exact reward contrast compared to the baselines for instance?

Reviewer_cVCyOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SUAF, which learn reward-independent representations for policy improvement under new reward functions. Its main theoretical contribution lies in decomposing the reference policy’s advantage into an immediately computable reward difference and a future-effect component that must be learned, and in characterizing how to compress the latter. For practical learning, the paper introduces action-centered Soft-HGR and bidirectional TD. Experiments demonstrate strong performance on new reward functions in discrete environments.

Critical strengths

S1. The rationale for choosing which component to compress is clear.

S2. The paper provides a detailed error analysis.

S3. The theoretical results are supported by practical learning methods and empirical evaluations.

Critical weaknesses

W1. The proposed approach relies heavily on the reference policy. Moreover, the trust-region analysis assumes only local policy changes, so its guarantees may not extend to repeated policy improvement or substantial policy changes.

W2. The data requirements and computational costs of zero-shot adaptation are insufficiently evaluated. Performance with limited reward observations or sparse rewards, along with adaptation costs, needs further assessment.

W3. Performance is used as an indirect measure, but it reflects the combined effects of multiple factors. Although the paper includes ablation studies, the continuous-control results make it difficult to isolate the contribution of the proposed approach.

Questions for authors

Q1. Please address weaknesses above.

1 comment
AuthorsOpenReview ↗

Gentle Reminder: Discussion Period Approaching Deadline

Dear Reviewer,

Hope you are doing well. As the discussion period is drawing to a close, we would really appreciate it if you could review our rebuttal and revised submission soon. We have responded point-by-point to your feedback and added new results as requested.

If you think our responses adequately resolve your concerns, we kindly ask you to consider updating your evaluation. Feel free to raise any remaining questions, and we will reply promptly.

We sincerely appreciate your valuable time.

Best regards, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal phase

Dear Area Chair,

Thank you for considering our submission. Thank you for taking the time to consider our submission. We’d like to summarize what the reviewers concerned about and how our rebuttal addressed it.

Until we write this summary, all three reviewers responded to our rebuttal and maintained their ratings. Reviewer cVCy and SjNj recommended weak acceptance (3), and reviewer adVP maintained a clear rejection (1).

Here are the specific concerns raised by the reviewers, how we responded and the final outcomes of discussion.

Concern 1. What does the SUAF representation contribute?

Questions.Reviewer cVCy and SjHj raised this concern. Reviewer cVCy asked to isolate the contribution of the representation. Reviewer SjNj asked about comparison fairness.

Response.We conduct two controlled experiments in finite-state environments. First, we compare SUAF with direct compression of the full advantage at the same rank. Keeping the immediate reward contrast explicit and learning only the delayed effects yields lower advantage approximation error. Second, we give One-Step FB the same exact immediate reward contrast. SUAF still achieves higher policy improvement on both the GridWorld and Four-room circulation tasks. We also report each method’s feature dimension and clarify that, in continuous control, SUAF estimates action expectations from the same offline data available to all methods.

Reactions. Reviewer cVCy explicitly confirmed that it resolved their concern about the representation. Reviewer SjNj’s follow-up shifted to when SUAF works, with no further questions about fairness.

Concern 2. When does SUAF work, and how much reward data requires?

Questions.All reviewer questioned on this concern. Reviewer adVP asked whether SUAF offers a clear gain on complex tasks. Reviewer SjNj asked which task properties explain why method rankings change. Reviewer cVCy asked how many reward observations adaptation requires and how SUAF performs when rewards are sparse.

Response.We conduct a held-out-reward experiment across continuous-control domains. In each domain, SUAF ranks first on held-out tasks. To explain why rankings can change, we distinguish tasks that reward reaching a goal quickly from those that reward staying there. We also analyze a failure case in Taxi. The first 64 modes capture only 0.88% of the delayed advantage energy for the sparse delivery reward, explaining why low-rank SUAF makes little progress. On adaptation cost, we explained how the number of reward observations affects the accuracy of the reward projection and reported how Taxi performance improves with more data to estimate the shaping potential.

Reactions. Reviewer adVP’s follow-up gave no separate assessment of the new experiments. Reviewer SjNj found the method promising but still wanted a clearer account of when it works. We provided the Taxi analysis in our final reply to that comment. Reviewer cVCy found the continuous-control results competitive.

Concern 3. How far can SUAF be reused?

Questions. Reviewer cVCy asked whether features learned under one reference policy can support repeated policy improvement or large policy changes. Reviewer SjNj asked how SUAF scales to larger problems and what advantage it offers over modeling transitions directly.

Response. We explain that exact SUAF from one reference policy can represent the state–action occupancies of all stationary policies (Theorem D.5). We conduct a repeated policy-improvement experiment in which we reuse the critic within a trust region and refresh it when the policy leaves that region. Returns improve on all tasks. For scalability, we give an example in which the transition matrix is full-rank but a single SUAF mode captures all delayed action effects.

Reactions. Reviewer cVCy accepted the explaination and asked us to make clear that the guarantee for truncated SUAF covers local policy changes, concluding that the theoretical contribution merits acceptance. Reviewer SjNj recognized the potential to combine SUAF with transition models, while remaining questions about when SUAF performs well and how it scales to real-world tasks.

Concern 4. Clarity and references.

Questions. Reviewer adVP flagged two citation issues and proposed that the introduction and theorem explanations difficult to follow.

Response. We addressed both citation issues and revised the paper for better readability. After reviewer adVP identified specific sentences that remained unclear, we revised the introduction again for further explaination.

Reactions. Reviewer adVP maintained a score of 1 (clear rejection) after the first revision. We are waiting for the response to our latest revision.

In the revised paper, we have added experiments, explainations and updated related wording.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.