- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
In this work, the authors present a model termed ATM to solve POMDPs (RL without full observability). ATM appears similar to standard linear transformers, but replaces the fast parallel linear update with a slow sequential GRU update (e.g., hidden state becomes ). ATM also bolts key/query and delta update operators on top of the GRU recurrent state (e.g., ).
The authors evaluate ATM on two synthetic RL tasks against a GRU and a "Transition-FWP" that is never defined. Then, the authors perform ablations of various components of ATM.
Critical strengths
Learning how to store and recall information via RL objectives is an interesting topic.
Critical weaknesses
- ATM is a reimplementation of linear attention, except significantly less efficient. In particular, the key-value delta correction appears to be nearly identical to the gated delta net referenced in the related work.
- The experiments are completely insufficient for ICLR, evaluating two toy environments proposed by the author. Furthermore, the GRU has much greater accuracy than the proposed method (Table 3) on their tasks, yet the authors bold their method as if it is the state of the art method. I am not sure if this is negligence or subterfuge, but both deserve an ethics review.
- The work is heavily AI-written and filled with nonsense sentences, which contain many technical-sounding but undefined or vague terms. The result is many passages that are difficult to read but simultaneously provide no information to a reader knowledgeable in RL/ML. Consider, for example, this passage from the abstract:
Explicit transition routing reaches a fixed competence criterion 1.95x sooner in PPO-update count than a parameter-matched implicit interface. At the matched-budget endpoint, ATM achieves higher five-seed rolling-20 training return under both association loads. - What is "transition routing", what is "explicit transition routing"? - What is a "fixed competence criteria"? - What is a "parameter-matched implicit interface"? - What is a "matched-budget endpoint"? - What is "five-seed rolling-20 training return"? - What are "association loads"?
- Combining memory with a PPO objective is written as if it is novel (section 3.3) when it is most certainly not.
Questions for authors
- Can you explain why you falsely highlight your proposed method in Table 3 when the GRU accuracy is significantly higher?
- Why would one choose your method (that uses a GRU internally) instead of the GRU itself?
- Please rewrite the paper, ensuring you define all terms such that a reader familiar with the Sutton and Barto textbook can follow along.
- For what reason would one choose ATM instead of a gated delta net?
- Can you compare the return? How about the wall-clock time efficiency?