- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors claim that attention models long-range temporal dependencies better than RNNs. To this end, they present AoD, which uses surprise to gate output between a recurrent model and transformer. The gate is passed the prediction error to determine whether information flows from a GRU or transformer. The model is trained end to end with PPO and evaluated across various tasks. The model is compared to an MLP, GRU, transformer, and fixed gate (fixed mixture ratio). The authors ablate various components and demonstrate that when provided the surprise, gating adds a marginal improvement (does not appear statistically-significant).
Critical strengths
Combining RNNs and transformers is likely to combine their strengths. These topics are active areas of research pursued by many and a worthwile endeavour.
Critical weaknesses
My primary concern is the basis of the proposed method, which is stated throughout the paper:
prediction error can provide a temporally grounded signal for regulating attention reliance
As I understand it, for a given target , input , and prediction , this roughly states that
I do not see any basis for this assumption and the author does not provide any evidence.
Furthermore, although the authors says the gate is determined by the surprise, this is not strictly true. Rather, the surprise is just an input to the gating function (Eq. 10). I would suspect that in practice, the surprise is likely ignored. This gating formulation makes the proposed AoD similar to the GTrXL cited in the paper. The GTrXL also combines a gate, GRU, and transformer.
Another major concern is that surprise is only available at training time, once the next state has already occurred (Eq. 8). How can one possibly use this model during rollouts? Either the text is wrong or the code/approach is wrong.
The method is relatively complex which reduces its utility. And after discussing how attention is better in long contexts, the author uses a context window size of 16, which is very small and does not match earlier text.
Questions for authors
- As far as I understand, surprise is only available during training (when you know the next state). How does your approach work once deployed if the surprise is not available?
- Can you demonstrate why "prediction error can provide a temporally grounded signal for regulating attention reliance"?
- Why is your attention window size only 16? I don't think you can claim that attention is better for long-horizon tasks, yet only use a window size of 16.
- Such technical information should not be in the abstract: "In the primary MemoryS7 evaluation, its meangate activation is E[ωt] = 0.646, corresponding to 35.4% lower representational attention reliance relative to a full-attention reference. Gate activation increases from 0.38 to 0.85 across surprise quintiles and also rises under observation corruption."
- "but do not regulate its relative contribution according to temporal predictability. " Can you explain this? What is "temporal predictability"? How would one measure this?
- "Each observation is converted to a one-hot representation and mapped to the shared latent space" Why do you convert to one-hot inputs? I do not think prior work on memory does this.