acceptodds
Withdrawn from ICLR 2027

Attention on Demand: Surprise-Driven Selective Attention for Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Attention mechanisms are widely used in reinforcement learning (RL) to capture long-range temporal dependencies, yet many architectures compute temporal attention at every timestep regardless of local predictability. In partially observable environments, recurrent representations may suffice during predictable transitions, whereas longer-range temporal context can become more useful when local dynamics fail to explain incoming observations. We introduce Attention-on-Demand (AoD), a surprise-conditioned framework that adaptively weights recurrent and attention representations. AoD combines a recurrent core with causal multi-head attention and uses an auxiliary forward model to predict the next latent observation. The resulting latent prediction error serves as a local surprise signal that conditions a differentiable gate between the two representations. Across the evaluated partially observable benchmarks, AoD achieves episodic returns comparable to full-attention baselines. In the primary MemoryS7 evaluation, its mean gate activation is , corresponding to lower representational attention reliance relative to a full-attention reference. Gate activation increases from to across surprise quintiles and also rises under observation corruption. These results show that a surprise-conditioned gate can vary the attention representation's contribution with local prediction error.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Selective Attention, Surprise-Driven Learning, Adaptive Gating, Partial Observability

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_iu4KOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors claim that attention models long-range temporal dependencies better than RNNs. To this end, they present AoD, which uses surprise to gate output between a recurrent model and transformer. The gate is passed the prediction error to determine whether information flows from a GRU or transformer. The model is trained end to end with PPO and evaluated across various tasks. The model is compared to an MLP, GRU, transformer, and fixed gate (fixed mixture ratio). The authors ablate various components and demonstrate that when provided the surprise, gating adds a marginal improvement (does not appear statistically-significant).

Critical strengths

Combining RNNs and transformers is likely to combine their strengths. These topics are active areas of research pursued by many and a worthwile endeavour.

Critical weaknesses

My primary concern is the basis of the proposed method, which is stated throughout the paper:

prediction error can provide a temporally grounded signal for regulating attention reliance

As I understand it, for a given target yy, input xx, and prediction y^\hat{y}, this roughly states that

∂y^∂x∝∥y−y^∥.\frac{\partial \hat{y}}{\partial x} \propto \lVert y - \hat{y} \rVert .

I do not see any basis for this assumption and the author does not provide any evidence.

Furthermore, although the authors says the gate is determined by the surprise, this is not strictly true. Rather, the surprise is just an input to the gating function (Eq. 10). I would suspect that in practice, the surprise is likely ignored. This gating formulation makes the proposed AoD similar to the GTrXL cited in the paper. The GTrXL also combines a gate, GRU, and transformer.

Another major concern is that surprise is only available at training time, once the next state has already occurred (Eq. 8). How can one possibly use this model during rollouts? Either the text is wrong or the code/approach is wrong.

The method is relatively complex which reduces its utility. And after discussing how attention is better in long contexts, the author uses a context window size of 16, which is very small and does not match earlier text.

Questions for authors
  1. As far as I understand, surprise is only available during training (when you know the next state). How does your approach work once deployed if the surprise is not available?
  2. Can you demonstrate why "prediction error can provide a temporally grounded signal for regulating attention reliance"?
  3. Why is your attention window size only 16? I don't think you can claim that attention is better for long-horizon tasks, yet only use a window size of 16.
  4. Such technical information should not be in the abstract: "In the primary MemoryS7 evaluation, its meangate activation is E[ωt] = 0.646, corresponding to 35.4% lower representational attention reliance relative to a full-attention reference. Gate activation increases from 0.38 to 0.85 across surprise quintiles and also rises under observation corruption."
  5. "but do not regulate its relative contribution according to temporal predictability. " Can you explain this? What is "temporal predictability"? How would one measure this?
  6. "Each observation is converted to a one-hot representation and mapped to the shared latent space" Why do you convert to one-hot inputs? I do not think prior work on memory does this.
Reviewer_PsFYOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Attention-on-Demand, a reinforcement learning model that mixes a GRU memory with an attention module and uses "surprise," meaning how badly the model predicted the next observation, to decide how much to lean on attention at each step. Tested on MiniGrid, T-Maze and POPGym tasks, it performs about as well as a standard Transformer while putting less weight on attention, and the gate behaves sensibly by rising when surprise is high or observations are noisy.

Critical strengths

None

Critical weaknesses

The idea isn’t very new. Combining a recurrent network with attention and using a learned gate to mix them has been done before. The main new part here is feeding the prediction error into the gate, which is a small change, and the paper doesn’t show that this change makes a real difference. On MiniGrid, AoD basically ties the Transformer, and on MemoryS7 every model, even a plain GRU, scores the same 0.49, so that benchmark doesn’t really tell the methods apart. On the long T-Maze it beats the GRU but still falls a bit behind the Transformer. The efficiency story isn’t there either: both branches run at every step, so the 35% less attention reliance doesn’t translate into any actual savings in compute or latency.

I’m also not convinced surprise is doing much. Taking it out of the gate costs only 0.02 to 0.05 in return, which is within seed noise, and the correlation between surprise and the gate isn’t surprising when surprise is fed straight into the gate. The gate value itself may not reflect real reliance on attention, since the attention output could simply be scaled up to make up for a lower gate.

Questions for authors

see Weaknesses

Reviewer_tf99OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Attention-on-Demand (AoD), a surprise-conditioned gating framework that adaptively combines GRU and causal attention representations in partially observable reinforcement learning. Experiments on MiniGrid, Passive T-Maze, and POPGym demonstrate performance comparable to full-attention baselines, with attention weights varying according to prediction surprise.

Critical strengths

None

Critical weaknesses

The methodological novelty and practical contribution appear limited. AoD extends existing adaptive fusion by incorporating prediction error, while uncertainty-based attention control has also been explored in prior work. Moreover, both branches are evaluated at every timestep, so the reported reduction in attention reliance does not translate into computational savings. The experiments also fail to demonstrate a clear performance advantage over simpler baselines.

Questions for authors

What fundamental advantage does AoD offer over existing adaptive gating and uncertainty-based attention methods in terms of computational efficiency or task performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.