- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a rational analysis of the problem of attention allocation, where attention is a costly action that improves the statistical fidelity of observations in a POMDP. The task is temporal signal detection, where the environment transitions from noise state to signal state at an unknown time. Optimal policies show an approximate phase transition between rhythmic attention (attending at fixed intervals) and ramping attention (observing with weak attention and then increasing attention on the next step if signal is observed). These results are taken to bear on the optimality of fixed-stride heuristics for example in video processing.
Critical strengths
Conceptually useful approach to rational analysis of attention allocation over time Interesting emergent patterns in the studied task
Critical weaknesses
Task is reasonable but quite specific (signal detection task, POMDP structure, attention cost, relation between attention and observation fidelity). It's unlear that the results generalize. In particular the rhythmic pattern seems to be an artifact of the task where the signal has a guaranteed minimum duration () and observations are binary so that a null observation resets to a fixed belief state
Questions for authors
Why a geometric distribution for noise duration but negative binomial () for signal duration?
Why discounted reward (I assume you used exponential discounting) rather than reward rate for this repeating task?
Fig 2A: I understand why the attention curve drops to zero in the outer regions, but why is it non-smooth and in some places not even monotonic in the middle region? Why is it nonzero in parts of the left region where the observation will not be used? The explanation at 214 (gathering information) seems wrong because with an observation becomes uninformative on the next time step: at the agent knows exactly the state at based on whether the game ended.
IIUC Fig 2b shows cases where the belief is supported only on the noise state and the final signal state (with zero belief for the intervening states). How realistic is this? It seems much more likely for the agent to get to a belief state supported only on the noise state and first signal state.
How can the agent get into a situation where it is so confident it’s in the signal state that it won’t use the observation? For m=1 the only possibility I see is that the transition probability is very high and . So every trial is the same: agent does nothing at (because the game is guaranteed to be in the noise state) then commits without observing at .
Can you make a case that these results are relevant for realistic tasks, where the environment transitions within a high-dimensional state space and observations are many-valued? In particular is there reason to believe rhythmic policies can still be optimal?
4 comments
Q4: IIUC Fig 2b shows cases where the belief is supported only on the noise state and the final signal state (with zero belief for the intervening states). How realistic is this?
A4: Belief supported only on the noise state and the final signal state is very unrealistic. Any distribution of signal belief only on a single state when is unrealistic outside of the step immediately following the start of the noise. Thus there is no 1d representation of higher-dimensional policies that would align with what the agent actually sees in trial. We plot Fig. 2b as a 1d approximation because it is the most readable approximation of higher-dimension signal spaces, not because it is any more or less realistic than the other options.
Q5: How can the agent get into a situation where it is so confident it’s in the signal state that it won’t use the observation?
A5: For the policies we show and discuss in the paper, the agent cannot get into a situation where it’s so confident in the signal state that it won’t observe. It will always use a final decisive observation to determine commit. The policies accurately represent that, if the agent’s signal belief ever was that high at the start of a time-step, the agent should commit without attending, but, in practice, when an agent follows the policy from the start state onward, its signal belief would never be this high at the start of a step. As you say, if the transition probabilities were high enough, we might see this behavior, or if we make attention expensive enough, the agent will merely wait a fixed duration, then commit, without ever attending. We did not include plots of these cases because they have much simpler behaviors and no attention dynamics to study, but with high transition probabilities and attention costs too high to ever pay, we would see rhythmic commits with no attention allocations.
[Comment 2/2]
Dear Reviewers,
The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!
Best, AC
Thanks for the thorough replies. They address my questions well. The new experiment in A.11.4 takes a big step toward addressing my main concern, enough to raise my score.
In the A.11.4 experiment, what is the coefficient for the cost of attention as a function of ? The expected log-likelihood ratio for a single observation in this task is so a small amount of attention yields a large amount of information (e.g., expected log-likelihood ratio of ln(100) requires only ). So unless the cost coefficient is rather high, the task essentially reduces to one with binary observations. This fits the description in the paper: “agents wait while signal belief rises, make a high-accuracy check”. I don’t think you necessarily should add runs with a greater cost coefficient, but you should report the value you used, and consider noting my point here about near-binary observations (if you agree).
Thank you for engaging with the paper through your careful review. We hope we can address your concerns.
Regarding the signal’s minimum duration, our results hold even for m=1. In that case, there is only a single signal state, so the signal’s minimum duration is the smallest unit of time the model can represent. Thus, the minimum can be made arbitrarily small; as such, we would not consider this feature a weakness of the model.
We have data from runs where we allowed observations to be gaussian rather than binary—with the tradeoff that attention choices became discrete—and found our results held. Accordingly, our results do not rely on binary observations. We also tested a slightly more complex variant, where value iteration is no longer tractable. There we trained a deep RL agent (PPO) with 25 signal states of fixed duration, Gaussian (non-binary) observations and continuous attention. Some agents discovered the same wait-check-reset rhythmic strategy (see Appendix A.11.4 of the revised PDF), showing that rhythmic allocation is not specific to binary observations or small state spaces. The strength of the rhythm varied across eight seeds, and seeds with a stronger rhythmic peak tended to earn more reward (Spearman ρ = 0.57, p = 0.14), so we present this as an illustration.
Q1: Why a geometric distribution for noise duration but negative binomial (m>1) for signal duration?
A1: One might suspect that rhythmic attention is naturally optimal for a fixed signal duration. If the signal is exactly 3 seconds long every time, then checking for it every 2.9 seconds is a naturally efficient policy. We varied m to demonstrate that the results hold similarly for multiple distributions of signal duration with different reliability, ranging from geometric (m=1, or exponential in continuous time) and negative binomial (m>1, or gamma in continuous time). We used the broadest distribution of noise duration possible (geometric/exponential) because this is the condition in which accurate observations are most necessary; increasing the number of noise states leads to a predictable onset of the signal. In this case, the agent need not attend, as the signal onset will occur at the same time each trial. The fewer noise states used, the less certain the agent can be about the time of signal onset at a trial start, so the more it will rely on its observations. We take this setting to be the most informative one for studying attention allocation strategies.
Q2: Why discounted reward (I assume you used exponential discounting) rather than reward rate for this repeating task?
A2: We chose an infinite-horizon discounted-reward objective with γ=0.995, which places substantial weight on multiple future rewards (on a timescale of ~1/(1–γ)=200 time steps), and people and animals do tend to have short-term preferences. Still, exploring alternative objective formulations like average reward rate would be an interesting direction for future work.
Q3a: Fig 2A: I understand why the attention curve drops to zero in the outer regions, but why is it non-smooth and in some places not even monotonic in the middle region?
A3a: The value iteration algorithm we applied to solve for policies required a discretization over the input space and a numerical estimation to solve for optimal attention. The function of reward of attention is not monotonic and, at the point of transition between two peaks, where their values are very similar, it may be ambiguous which peak is higher. At these points, the algorithm may return one peak for one belief, another for the belief at the next step in the discretization. Both are roughly equivalent, otherwise there would be no ambiguity, so the policy is still very near to optimal, but it can appear to oscillate in these zones.
Q3b: Why is it nonzero in parts of the left region where the observation will not be used?
A3b: An observation does not become uninformative on the next time step unless the agent commits or the trial times out in that step. Otherwise, the belief update resulting from the observation will change the agent’s belief in the next step, which will inform its behavior. If the game ends, the agent knows the game ended, but if the game does not end, the agent does not know what state it was or is in. It could be in the noise or a signal state, therefore it still needs an accurate belief to perform well, and the more it attended in the previous step, the more accurate it can expect its belief to be.
[Comment 1/2. Our response exceeded the max reply character length, so we address the remaining questions in a second comment.]