- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the learning dynamics of distribution matching against expectation matching. The goal is to gain understanding of the representation learnt in distributional reinforcement learning compared to traditional value function training. The work simplifies the policy evaluation setting by considering a set of supervised learning setups as a proxy for policy evaluation.
The paper extends prior works that studied the learning dynamics of networks with 2 linear layers for mean regression. They do so by formalizing distribution matching via expectile regression / matching.
The paper studies in detail a two-state setup that allows both obtaining closed-form solutions for the MSE objectives and performing simulations. The authors discuss the learning speed and the hidden-state geometry. Both agree and show that distributional targets allow learning embeddings that preserve distinctions between inputs with different moments beyond their means. In the case of equal means, the usual mean regression network collapses to a single embedding.
Critical strengths
The idea of borrowing from the litterature on supervised learning learning dynamics and extend it to distribution matching seems novel.
Critical weaknesses
In my opinion, the primary issue with this work is its exposition and lack of motivation. Despite spending several days reading it and familiarizing myself with the literature on learning dynamics, I still fail to grasp what the motivation for this work was.
- This work is not about distributional reinforcement learning.
- As far as I understand, it studies supervised learning as a proxy for policy evaluation. It never motivates this choice or connects it back to policy evaluation.
- Empirical distributional policy evaluation relies on temporal differences, whose dynamics might be very different from the supervised case.
- There is no empirical connection made with any policy evaluation setup. I could not find any evidence that the representational advantage exposed for the supervised learning case should bring any benefit for policy evaluation and then for control. In my opinion, this work is poorly motivated.
- There is no red line / story to follow. This work is a collection of facts tied together.
- After several days, I still cannot tell why I am reading what I am reading for almost every section of this work.
- Section 2.3, which in my opinion is the most important one for the background, compresses too much. I had to spend several hours going through each sentence step by step. The derivations take several pages. I am not sure the average reader would get any insight after reading the first two paragraphs. Yet they seem essential to the rest of the paper.
- Most experiments are scattered in the appendix, which makes it very hard to follow.
- Starting from Section 3.2, the problem gets very severe. The prose becomes very hard to follow. I would not blame any LLM usage if the text conveyed an intent that was clear to follow, but here this is not the case.
- The work does address the lack of control discussion but does not emphasize the lack of actual policy evaluation.
- The discussion section is a collection of assertions and facts, many of which are not tied to the story. As an average reader with expertise in distributional RL, I could not get any insights from “Predictions for neuroscience” or “Predictions for deep RL,” as the exposition is very cryptic.
Questions for authors
- In the discussion, you say, "We have developed an exact theory of learning for classical and distributional RL objectives in two layer linear networks." Did you? What was your contribution for the classical RL setting? Where did you go beyond simple expectation/distributional regression? You studied two losses used in RL based on MSE and expectiles. Put differently, what does your analysis establish regarding actual RL using these objectives?