- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the representations learned for zero-shot reinforcement learning. Compared to most existing methods, which learn representations to represent the Q-function, the paper proposes to learn representations to represent the delayed advantage function (which, compared to a normal advantage function, omits the immediate reward) instead. The paper evaluates the resulting method, spectral universal advantage features, in a number of discrete and continuous domains.
Critical strengths
None
Critical weaknesses
- Concerns about AI Use
- While the authors claim that "Every reference surfaced with AI assistance was located and checked against the original source by the authors before inclusion", there are blatant errors in the references.
- One of the references appear twice: "Barreto et al. Successor features for transfer in reinforcement learning." appears twice in the references, with years 2017a and 2017b
- One of the references is simply wrong: The paper cites Brillinger (1964) for Pinsker's inequality. The correct citation should have been the book: "Mark S. Pinsker. Information and Information Stability of Random Variables and Processes. Holden-Day Series in Time Series Analysis. Holden-Day, San Francisco, 1964." Yet, the authors cited "David R. Brillinger. Information and information stability of random variables and processes. Journal of the Royal Statistical Society Series C: Applied Statistics , 13(2):134–135, 06 1964." which is, in fact, just a two-page review on the book which has nothing to do with Pinsker's inequality.
- I would have expected the authors to catch these errors if AI generated references were truly "checked against the original source by the authors". This casts significant concerns about whether the authors have truly verified AI generated content. A simple AI query revealed other errors. Among these errors, the following notably adds to the concern:
- In line 3349, the paper writes "The upstream source and revision used for the final runs should be archived with the experiment artifacts." This seems to me a note generated by AI that the authors forgot to delete.
- While the authors claim that "Every reference surfaced with AI assistance was located and checked against the original source by the authors before inclusion", there are blatant errors in the references.
- Clarity
- The paper is very dense and hard to parse. For example, the third paragraph of the introduction is just a half-page block of text that describes details of the contribution of the work that, at this point, no readers can understand since they have not even read the preliminaries section. Like, what is action-incremental -dependence? What is action-centered Soft-HGR objective? Even worse, I don't think I even see the term -dependence appear again anywhere in the rest of the paper.
- Section 3 and 5 immediately jumps into the content without first describing the purpose/overview of the section. For example, the first sentence of Section 3 is "We establish the SUAF with dependent CDM." Similarly, Section 5 immediate jumps into experiment setup without even first mentioning the purpose of the experiments.
- The paper presents a large amount of theoretical results. However, there is in general insufficient explanation for what the theoretical results mean. For example, Theorem 4.3 is simply stated without any discussion in text.
- Experiment results
- Looking at Table 2, there does not seems to be clear performance improvements with the proposed method in complex domains, casting doubt on the significance on the proposed method.
Questions for authors
None
4 comments
Follow-up on Rebuttal and Discussion: Deadline Today
Dear Reviewer,
Hope you are doing well. As the discussion period closes today, we would like to kindly follow up on our rebuttal and revised submission. We have responded point by point to your comments and added the requested results.
We would be very grateful if you could take a moment to review our responses and let us know whether any concerns remain. If our replies have adequately addressed your points, we would kindly ask you to consider updating your evaluation before the deadline.
We are monitoring the discussion thread closely and will respond to any further questions or requests for clarification.
Thank you very much for your time and consideration.
Best regards,
Authors
Thank you for the response.
In my initial review, I raised concerns about clarity, stating that the paper is very dense and hard to parse. The authors have provided a revision in an attempt to address this issue. However, I don't think my concern on clarity has been addressed. Below, I use the third paragraph of the Introduction as an example to illustrate why I struggle to understand the paper.
- Sentence 4: After introducing "action-centered, positive-lag canonical dependence matrix", shouldn't the authors explain what it is, before starting to talk about its singular value decomposition?
- In order to understand the sentence, I had to ask an LLM: (1) What is a CDM?, (2) What does positive-lag refer to?
- A well-written paper should be digestible for a human reader without them having to constantly google/ask an LLM what specific terms mean.
- Sentence 5: "value" missing from singular "value" decomposition.
- Sentence 7: You write "To determine which components to retain". Actually, up to this point, I still have no idea what a "component" means. You briefly mentioned that at the end of the second paragraph ("which components of that dependence should be retained") but with no explanation.
- Again, I had to ask an LLM to finally get that you're referring to one of the modes of the SVD of the canonical dependence matrix.
- Sentence 8: What is a "nonzero spectral floor"? Again I had to ask an LLM to understand this.
- Sentence 9: How is this result relevant to the question of which components to retain, as raised in Sentence 7?
- Sentence 10: You say that retaining the leading modes of the SVD gives some optimal approximation. Does that mean we just keep the leading "components" then? If yes, a sentence confirming that you indeed retain the leading components based on this finding is missing. Also, "What's more" is pretty informal, and I would avoid using this phrase.
- Sentence 7 brings up the problem of determining which components to retain, and sentences 8-10 are supposed to explain what components to retain and why. But after reading these sentences I still am not fully sure which components you end up retaining and why. I can only guess based on sentence 10, but readers should not be left guessing.
- Sentence 12: "The error depends on ..." What error? The error between what? Do you mean the error between the ground-truth CDM and the CDM approximated using a limited number of modes? Again, readers shouldn't be left guessing.
- Sentence 13: What is "aggregate lost dependence"? What is "task-specific information loss"? What is "approximation penalty"? Is there a penalty for approximation errors?
I hope that with this more detailed breakdown, I have demonstrated why I think that this paper has significant clarity issues. I believe that (1) a clear paper should be digestible by an independent human reader without heavy assistance from search engines/LLMs, and (2) a clear paper should be explicit to minimize ambiguity. Unfortunately, this paper falls short in both regards. Since clarity is a prerequisite for evaluating other acceptance criteria, I will maintain my score of clear rejection.
Thank you for identifying the specific sentences. We have revised the paragraph in light of your suggestions. We would also like to explain the reasoning behind the wording you highlighted.
- Sentence 4: We use canonical dependence matrix as an established term in information theory and cite the paper proposed this idea in the same sentence. Besides, we have introduced the underlying idea before giving the matrix its name. Sentence 3 explains that we subtract “the policy-averaged future visitation” to isolate how the current action changes future outcomes and sentence 4 names the matrix that captures this dependence.
- Sentence 5: “Singular value decomposition.” We agreed. The word value has been added.
- Sentence 7: Since sentence 5 introduces the singular decomposition and its “full expansion”, a natural problem is “which components to retain” because we are moving to a low-rank approximation when representation budget is limited. In this case, the components are the individual modes in the expansion.
- Sentence 8: The original sentence states that the immediate term induces “a nonzero spectral floor, limiting direct low-rank compression.” The words immediately following floor identify that it is a limit on the low-rank compression of the advantage operator at low rank. Since we are using SVD, the floor is actually a spectral floor.
- Sentence 9: The original sentence states that, under weak dependence, “the sum of squared singular values approximates the mutual information.” Its relevance to mode selection follows directly from that sum. Discarding a mode removes its squared singular value from the approximation, so the sum over the discarded modes measures how much of the approximated information is lost. At a fixed rank, retaining the leading modes minimizes that loss. This is precisely why the information-theoretic result belongs in the discussion of which components to retain.
- Sentence 10: SUAF is the leading components. The sentence states that “retaining leading modes gives an optimal rank-constrained approximation of delayed advantage under both average and worst-case reward objectives.” This identifies both the modes and the reason for selecting them. We can make the method choice more direct by using “SUAF retains the leading modes” as the main clause.
- Sentence 7-10: We organize the sentences under this consideration: sentence 7 asks “which components to retain.”, sentence 8 narrows the choice to the delayed component by stating that SUAF keeps the evaluable immediate contrast explicit and sentence 10 then answers the question in the same terms in which it was posed: “retaining leading modes gives an optimal rank-constrained approximation of delayed advantage under both average and worst-case reward objectives.” The words “retaining leading modes” identify the choice, and “optimal … under both average and worst-case reward objectives” gives the reason.
- Sentence 12: Sentence 10 identifies delayed advantage as the quantity being approximated. Sentence 11 then asks what happens when modes are discarded. Thus, for a given reward, “the error” is the difference between its exact delayed advantage and the approximation obtained from the retained modes. The distinction from CDM approximation error matters here. The same truncated CDM is used across rewards, while the resulting advantage error can differ from one reward to another.
- Sentence 13: We distinguish the “aggregate lost dependence” and “task-specific information loss” because a discarded mode can carry substantial dependence yet contribute nothing to the delayed advantage for a particular reward. That is beacuse as we stated early, the advantage is actually a projection of reward ("... recovers delayed advantage through linear reward projections"). The approximation penalty therefore refers to the resulting approximation loss. The sentence says how it enters that condition: the predicted gain must exceed the loss due to truncation.
We appreciate your point that readers should be able to follow these connections on a first reading. At the same time, we respectfully believe that a clear rejection is too strong on the grounds given. Actually, we have the response to this concerns in this paper. We hope the clarifications above and the resubmitted revision are helpful and would be grateful if you could weigh the paper’s technical contributions alongside your concerns about presentation.
Gentle Reminder: Discussion Period Approaching Deadline
Dear Reviewer,
Hope you are doing well. As the discussion period is drawing to a close, we would really appreciate it if you could review our rebuttal and revised submission soon. We have responded point-by-point to your feedback and added new results as requested.
If you think our responses adequately resolve your concerns, we kindly ask you to consider updating your evaluation. Feel free to raise any remaining questions, and we will reply accordingly.
We sincerely appreciate your valuable time.
Best regards, Authors