acceptodds
Under review as a conference paper at ICLR 2027

Attention when you need

Abstract

Paying attention improves performance, but attention is metabolically costly, so how should a resource-efficient agent allocate it? We study optimal allocation strategies using a normative model of a signal detection task in which attention comes at a cost. The model reveals that optimal attention is temporally structured in one of two patterns depending on task conditions: when attention costs penalize intense focus, optimal attention ramps up as evidence for the signal accumulates, but with less prohibitive attention costs, optimal attention fluctuates rhythmically. In cases with rhythmic attention, allocation frequency increases with task parameters such as reward magnitude for successful detection, signal brevity, and signal frequency. We argue that rhythmic attention emerges naturally in an agent whose belief updates are dominated by a stable temporal prior and not by the likelihood of new sensory observations. Our results characterize the conditions under which rhythmic vs. ramping attention is optimal and offer a normative account of attentional fluctuations observed in sustained attention tasks.

Primary area:
applications to neuroscience & cognitive science
Keywords:
attention, sustained attention, resource allocation, periodic, signal detection, internal action, internal control

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_uuE3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a rational analysis of the problem of attention allocation, where attention is a costly action that improves the statistical fidelity of observations in a POMDP. The task is temporal signal detection, where the environment transitions from noise state to signal state at an unknown time. Optimal policies show an approximate phase transition between rhythmic attention (attending at fixed intervals) and ramping attention (observing with weak attention and then increasing attention on the next step if signal is observed). These results are taken to bear on the optimality of fixed-stride heuristics for example in video processing.

Critical strengths

Conceptually useful approach to rational analysis of attention allocation over time Interesting emergent patterns in the studied task

Critical weaknesses

Task is reasonable but quite specific (signal detection task, POMDP structure, attention cost, relation between attention and observation fidelity). It's unlear that the results generalize. In particular the rhythmic pattern seems to be an artifact of the task where the signal has a guaranteed minimum duration (mm) and observations are binary so that a null observation resets to a fixed belief state

Questions for authors

Why a geometric distribution for noise duration but negative binomial (m>1m>1) for signal duration?

Why discounted reward (I assume you used exponential discounting) rather than reward rate for this repeating task?

Fig 2A: I understand why the attention curve drops to zero in the outer regions, but why is it non-smooth and in some places not even monotonic in the middle region? Why is it nonzero in parts of the left region where the observation will not be used? The explanation at 214 (gathering information) seems wrong because with m=1m=1 an observation becomes uninformative on the next time step: at t+1t+1 the agent knows exactly the state at tt based on whether the game ended.

IIUC Fig 2b shows cases where the belief is supported only on the noise state and the final signal state (with zero belief for the intervening states). How realistic is this? It seems much more likely for the agent to get to a belief state supported only on the noise state and first signal state.

How can the agent get into a situation where it is so confident it’s in the signal state that it won’t use the observation? For m=1 the only possibility I see is that the transition probability pp is very high and t=2t=2. So every trial is the same: agent does nothing at t=1t=1 (because the game is guaranteed to be in the noise state) then commits without observing at t=2t=2.

Can you make a case that these results are relevant for realistic tasks, where the environment transitions within a high-dimensional state space and observations are many-valued? In particular is there reason to believe rhythmic policies can still be optimal?

4 comments

Thank you for engaging with the paper through your careful review. We hope we can address your concerns.

Regarding the signal’s minimum duration, our results hold even for m=1. In that case, there is only a single signal state, so the signal’s minimum duration is the smallest unit of time the model can represent. Thus, the minimum can be made arbitrarily small; as such, we would not consider this feature a weakness of the model.

We have data from runs where we allowed observations to be gaussian rather than binary—with the tradeoff that attention choices became discrete—and found our results held. Accordingly, our results do not rely on binary observations. We also tested a slightly more complex variant, where value iteration is no longer tractable. There we trained a deep RL agent (PPO) with 25 signal states of fixed duration, Gaussian (non-binary) observations and continuous attention. Some agents discovered the same wait-check-reset rhythmic strategy (see Appendix A.11.4 of the revised PDF), showing that rhythmic allocation is not specific to binary observations or small state spaces. The strength of the rhythm varied across eight seeds, and seeds with a stronger rhythmic peak tended to earn more reward (Spearman ρ = 0.57, p = 0.14), so we present this as an illustration.

Q1: Why a geometric distribution for noise duration but negative binomial (m>1m>1) for signal duration?

A1: One might suspect that rhythmic attention is naturally optimal for a fixed signal duration. If the signal is exactly 3 seconds long every time, then checking for it every 2.9 seconds is a naturally efficient policy. We varied m to demonstrate that the results hold similarly for multiple distributions of signal duration with different reliability, ranging from geometric (m=1m=1, or exponential in continuous time) and negative binomial (m>1m>1, or gamma in continuous time). We used the broadest distribution of noise duration possible (geometric/exponential) because this is the condition in which accurate observations are most necessary; increasing the number of noise states leads to a predictable onset of the signal. In this case, the agent need not attend, as the signal onset will occur at the same time each trial. The fewer noise states used, the less certain the agent can be about the time of signal onset at a trial start, so the more it will rely on its observations. We take this setting to be the most informative one for studying attention allocation strategies.

Q2: Why discounted reward (I assume you used exponential discounting) rather than reward rate for this repeating task?

A2: We chose an infinite-horizon discounted-reward objective with γ=0.995\gamma=0.995, which places substantial weight on multiple future rewards (on a timescale of ~1/(1–γ)=2001/(1–\gamma)=200 time steps), and people and animals do tend to have short-term preferences. Still, exploring alternative objective formulations like average reward rate would be an interesting direction for future work.

Q3a: Fig 2A: I understand why the attention curve drops to zero in the outer regions, but why is it non-smooth and in some places not even monotonic in the middle region?

A3a: The value iteration algorithm we applied to solve for policies required a discretization over the input space and a numerical estimation to solve for optimal attention. The function of reward of attention is not monotonic and, at the point of transition between two peaks, where their values are very similar, it may be ambiguous which peak is higher. At these points, the algorithm may return one peak for one belief, another for the belief at the next step in the discretization. Both are roughly equivalent, otherwise there would be no ambiguity, so the policy is still very near to optimal, but it can appear to oscillate in these zones.

Q3b: Why is it nonzero in parts of the left region where the observation will not be used?

A3b: An observation does not become uninformative on the next time step unless the agent commits or the trial times out in that step. Otherwise, the belief update resulting from the observation will change the agent’s belief in the next step, which will inform its behavior. If the game ends, the agent knows the game ended, but if the game does not end, the agent does not know what state it was or is in. It could be in the noise or a signal state, therefore it still needs an accurate belief to perform well, and the more it attended in the previous step, the more accurate it can expect its belief to be.

[Comment 1/2. Our response exceeded the max reply character length, so we address the remaining questions in a second comment.]

Q4: IIUC Fig 2b shows cases where the belief is supported only on the noise state and the final signal state (with zero belief for the intervening states). How realistic is this?

A4: Belief supported only on the noise state and the final signal state is very unrealistic. Any distribution of signal belief only on a single state when m>1m>1 is unrealistic outside of the step immediately following the start of the noise. Thus there is no 1d representation of higher-dimensional policies that would align with what the agent actually sees in trial. We plot Fig. 2b as a 1d approximation because it is the most readable approximation of higher-dimension signal spaces, not because it is any more or less realistic than the other options.

Q5: How can the agent get into a situation where it is so confident it’s in the signal state that it won’t use the observation?

A5: For the policies we show and discuss in the paper, the agent cannot get into a situation where it’s so confident in the signal state that it won’t observe. It will always use a final decisive observation to determine commit. The policies accurately represent that, if the agent’s signal belief ever was that high at the start of a time-step, the agent should commit without attending, but, in practice, when an agent follows the policy from the start state onward, its signal belief would never be this high at the start of a step. As you say, if the transition probabilities were high enough, we might see this behavior, or if we make attention expensive enough, the agent will merely wait a fixed duration, then commit, without ever attending. We did not include plots of these cases because they have much simpler behaviors and no attention dynamics to study, but with high transition probabilities and attention costs too high to ever pay, we would see rhythmic commits with no attention allocations.

[Comment 2/2]

Dear Reviewers,

The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!

Best, AC

Thanks for the thorough replies. They address my questions well. The new experiment in A.11.4 takes a big step toward addressing my main concern, enough to raise my score.

In the A.11.4 experiment, what is the coefficient for the cost of attention as a function of pp? The expected log-likelihood ratio for a single observation in this task is 2p22p^2 so a small amount of attention yields a large amount of information (e.g., expected log-likelihood ratio of ln(100) requires only p=1.52p=1.52). So unless the cost coefficient is rather high, the task essentially reduces to one with binary observations. This fits the description in the paper: “agents wait while signal belief rises, make a high-accuracy check”. I don’t think you necessarily should add runs with a greater cost coefficient, but you should report the value you used, and consider noting my point here about near-binary observations (if you agree).

Reviewer_u93qOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Here the authors study a partially observable Markov decision process. The policy maker wants to successfully detect signal from noise. Given a belief btb_t at time step tt, the policy selects the attention ata_t to pay and a commit rule κ\kappa, whether to commit to the presence of signal. The authors show that when the exponent of the cost function for attention is "small", the optimal policy has rhythmic characteristics. As the exponent in the cost function for attention increases, it makes more sense to exert small amounts of attention, and the optimal attention process has less rhythmic structure.

Critical strengths

**The intuitive nature of the plots is the highlight of the paper. Particularly, the way the mathematics gives optimal policies that make intuitive sense, such as the rhythmic versus ramping plots in Figure 3 (linear versus quadratic).

**The authors have demonstrated that the results in Figure 3 are not dependent on specific choices of pp, qq, mm, and hit reward.

Critical weaknesses

** Model specification. The authors should write down the Markov transition function (at least in the appendix) instead of or along with the verbal description on page 3, and write down the mathematical equations for how the belief is updated with observation oto_t at attention ata_t.

** Dependence on modeling assumptions. This is a toy model, and I feel a lot of the results depend on the specific assumptions, such as the structure of the cost function ac2a^{c_2}, how the observations become totally uninformative at zero attention, and how the belief is updated according to observation and attention. I do not know to what extent these assumptions about the cost of attention make sense.

The authors correctly refer to the Cohen–Maunsell papers. There is subsequent work:

https://www.nature.com/articles/s41467-024-49144-w (Brain-state mediated modulation of inter-laminar dependencies in visual cortex, Nature Communications, 2024) https://www.sciencedirect.com/science/article/pii/S0896627309006953?via%3Dihub (Spatial Attention Decorrelates Intrinsic Activity Fluctuations in Macaque Area V4, Neuron, 2009)

These go into mechanisms of attention in visual tasks in the visual cortex. Perhaps they can inform what a good model of the cost of attention can be in the setting of attention in the cortex for visual tasks.

Questions for authors

In Figure 2, how do we get three different plots for m=1m=1? What is different in the underlying dynamics of the three plots in the top panel? What is the difference in dynamics between the three plots on the top panel and the leftmost one in the bottom panel?

What is the value of mm in Figure 3?

In Figure 6, are the mean signal duration and mean noise duration controlled by varying pp and qq of Section 3.2?

Do the authors assume that the policy maker knows how the state is related to the observation given an attention level, and is this used to update the belief posterior?

3 comments

We appreciate your thoughtful review and for these references. Both papers characterize how attention changes neural variability and interlaminar dependencies in the visual cortex. They are most relevant to the benefit side of our model, where attention changes observation accuracy. (For example, the attention-related decorrelation reported by Mitchell et al. 2009 is one candidate mechanism for improved accuracy, since reducing shared noise increases the information available from a population.) They do not, to our knowledge, offer a quantitative account of the cost of attention, so we cannot use them to fix a particular form for C(a)C(a), but our model does not depend on one particular C(a)C(a) structure. Our results hold without requiring C(a)=c1ac2C(a) = c_1a^{c_2} (see new section in the appendix, A.11.2, where we demonstrate rhythmic attention with a variety of other cost functions). We observe similar results even with cost functions other than the cost function formula as those in the main paper. Similarly, no-attention observations do not need to be totally uninformative for rhythmicity to occur (see Appendix A.11.5) though the more informative they are, the more regular attention allocations become.

In general, we do not claim the model’s assumptions hold in all cases. The added figures and discussion in Appendix A.11 demonstrate that the model is robust enough to variations that we can expect it to have utility even in more complex settings, that it produces clear predictions which would allow us to validate it at scale, and that it provides the benefit of interpretability.

Connecting to the experimental studies of attention that you mention, suggestive new evidence in favor of fluctuating attention in the rodent brain comes from a recent paper from Bondy et al., 2026 (https://www.biorxiv.org/content/10.1101/2024.08.21.609044v7). They show that there are large global one-dimensional neural fluctuations in sensitivity to evidence. This is consistent with a single evidence accumulator across the brain, whose sensitivity is determined by an overall fluctuating attention signal, as we predict. We now cite this work in the Discussion about biological attention.

Q1: In Figure 2, how do we get three different plots for m=1m=1? What is different in the underlying dynamics of the three plots in the top panel? What is the difference in dynamics between the three plots on the top panel and the leftmost one in the bottom panel?

A1: In Figure 2, we vary the nonlinearity c2c_2 of the relationship between attention and its cost to get three different plots for m=1m=1. We describe the difference in their dynamics in Section 4.1. “Where signal belief is low, the chance that the signal is present is too low to justify engaging, so the agent neither attends nor commits: it waits. Where signal belief is high, attention is an added cost that would likely only confirm what the agent already believes, so it commits without attending. In the regions where the agent does attend, its commit rule takes one of two forms. Either it does not commit regardless of what it observes, gathering information that moves its belief without ending the trial, or it commits only if it observes signal, using the observation to decide. Transitions between these behaviors occur at jumps in the policy. If there is a single jump, the policy switches directly from waiting to deciding. If there are several, the policy switches from waiting to information gathering at the first jump, then from information gathering to deciding at the jump to maximum attention. The number of jumps is closely tied to the two patterns of attention allocation we observe across trials: rhythmic and ramping.” The leftmost plot in the bottom panel has the same value of c2c_2 as the rightmost plot in the top panel. The rest of the bottom panel has this same value of c2c_2 for increasing mm.

Q2: What is the value of m in Figure 3?

A2: m=1m=1 in Figure 3. We have clarified this in the manuscript.

Q3: In Figure 6, are the mean signal duration and mean noise duration controlled by varying pp and qq of Section 3.2?

A3: Yes, in Figure 6, mean signal duration and mean noise duration are controlled by varying the exit probabilities for the signal and noise states, respectively.

Q4: Do the authors assume that the policy maker knows how the state is related to the observation given an attention level, and is this used to update the belief posterior?

A4: Yes, we do assume that the policy maker knows how the state is related to the observation given an attention level, and this is used to update the belief posterior. This is reasonable for an agent to have calibrated from experience. We have added details in the appendix (A.2).

Dear Reviewers,

The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!

Best, AC

The additional results in A.11.2 and Fig 13 strengthen your case. I will update my score on its basis. I like A.1 and A.2; these should have been part of the original submission. It would have made it much easier to understand. Figure 2: 'we vary the nonlinearity of the relationship between attention' -- put this in the caption; this was not evident to me upon reading your original submission. You get an extra page during reviews; I think you still have room. The rodent paper is sufficient motivation. As you said the two papers characterize how attention changes neural variability and interlaminar dependencies in the visual cortex. However, looking at what happens during attention can inform the cost which is paid to achieve it. I would suggest including the paper you have discovered and the two I originally suggested as relevant citations, as I believe this is the direction where your work can have most potential. I have no further questions

Reviewer_bjkPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how attention should be allocated under limited costs in a continuous signal detection task and formulates the problem as a POMDP. It finds that under a linear cost function, the optimal policy is rhythmic, whereas under an exponential cost function, the optimal policy shifts to a ramping strategy.

Critical strengths
  1. This paper is well-written and easy to follow.
  2. The paper conducts extensive experiments to validate these two attention patterns. This makes the agent’s attention-allocation behavior more interpretable and provides useful insights for future work.
Critical weaknesses
  1. The overall logic is clear, but the paper omits the analysis and explanation of several key issues. Please refer to the Questions section for more details.
  2. The paper provides extensive visual evidence, but lacks sufficient quantitative analysis. For example, when comparing the rhythmic and ramping strategies, it does not quantify how much the better-performing strategy outperforms the alternative, or how much performance is lost when that strategy becomes suboptimal.
  3. The paper only shows that the agent adopts different strategies under different cost functions, but it does not explain why a linear cost leads to a rhythmic strategy, whereas an exponential cost leads to a ramping strategy.
  4. The optimal policy is derived based on a manually defined cost function. However, the actual cost of attention may be more complex than the assumed cost function, which may limit the generalizability of the resulting optimal policy.
Questions for authors
  1. How exactly is the log peak SNR defined? What is the formula used to calculate it?
  2. How does the paper determine the optimal policy? Is there any scoring function used to evaluate it?
  3. “A large installment of attention costs no more per unit than a small one, so the agent saves by concentrating attention into sparse, decisive checks, which produces rhythmic policies.” Is there any quantitative evidence to support this argument? Specifically, how much does a small allocation of attention cost compared with a large allocation?
  4. As shown in Fig. 3, changing the cost constraint shifts the optimal policy from a rhythmic strategy to a ramping strategy. How much performance would be lost if the alternative strategy were used under the same cost constraint.
2 comments

Thank you for your thoughtful review. Regarding the analysis and explanation of key issues (Critical Weakness 1), we provide the following answers.

Q1: How exactly is the log peak SNR defined? What is the formula used to calculate it?

A1: Log peak SNR is the natural log of the signal-to-noise ratio at the peak of the power spectral density of a within-trial autocorrelation of attention from trials simulated under the optimal policy. We explained how it is computed in the Rhythmicity Metric section of the Appendix (A.6), where we have added further detail to ensure the process is clear.

Q2: How does the paper determine the optimal policy? Is there any scoring function used to evaluate it?

A2: The optimal policy is the policy that maximizes expected value, where expected value depends on the run’s parameters. The reward function for a single step is equal to the cost of the attention the agent paid in that step (C(a)C(a) in the paper for attention aa) plus the result of a commit, if the agent commits in that step. The result of a commit is the hit reward, if the world was in a signal state, or a false alarm penalty, if the world was not. We determine optimal policies via value iteration (detailed further in Appendix A.4), where we optimize the reward the agent collects over all steps in the future, subject to the temporal discount.

Q3. “A large installment of attention costs no more per unit than a small one, so the agent saves by concentrating attention into sparse, decisive checks, which produces rhythmic policies.” Is there any quantitative evidence to support this argument? Specifically, how much does a small allocation of attention cost compared with a large allocation?

A3: The ratio of the cost of a small installment to the cost of a large installment is C(a1)/C(a2)C(a_1)/C(a_2) for small a1a_1 and large a2a_2. For the attention cost function C(a)=c1ac2C(a)=c_1a^{c_2}, this is (a1/a2)c2(a_1/a_2)^{c_2}. Thus, the higher c2c_2 is, the cheaper small attention is relative to large attention, and the more expensive large attention is relative to small attention. Our quantitative evidence that agents pay sparse attention when c2c_2 is low is shared in the Results section of our paper (Section 4); we observe that value iteration finds policies where large installments of attention are paid periodically.

Q4: As shown in Fig. 3, changing the cost constraint shifts the optimal policy from a rhythmic strategy to a ramping strategy. How much performance would be lost if the alternative strategy were used under the same cost constraint.

A4: How much is lost depends on how badly the strategy is misaligned with the environment. To more directly show how a rhythmic policy performs when the attention cost exponent is high and how a rhythmic policy performs when the attention cost exponent is low, we added Appendix A.10. To cover our findings briefly: If the attention cost exponent is only slightly different from that for which the policy was optimized, performance is still close to optimal. The further the environment is from what the agent’s policy was trained for, the worse the agent performs. Additionally, we showed how much performance is lost when the strategy becomes suboptimal (flat) in Figures 5 and 10.

Section 4.2 explains why rhythmicity emerges from a linear or sublinear cost and why ramping emerges from a quadratic cost. For a lower attention cost exponent, it is more efficient to pay large amounts of attention rarely. For a higher attention cost exponent, it is more efficient to pay small amounts of attention frequently. Sufficiently sparse attention (see the section on Rhythmic Policies in 4.1) leads to beliefs that evolve deterministically, so the policy’s execution becomes regular, resulting in rhythmicity. Frequent attention is sensitive to many stochastic observations, so belief accumulates stochastically over consecutive observations of signal. The higher the belief, the more likely is higher attention will pay off, resulting in a pattern of attention ramps. For the parameters we ran, a linear cost function has an optimum with the former strategy, a quadratic with the latter.

Since there is no known ground truth cost of attention, we explore a family of models with convex or concave relationships between cost and information gained from attention. Even if we change the precise form, the core results about rhythmic and ramping attention hold even with other cost functions (see Appendix A.11.2).

Dear Reviewers,

The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!

Best, AC

Reviewer_sN2LOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies optimal allocation strategies for sustained attention tasks by considering attention as a costly action. The task is modelled as a partially observable Markov decision process, and the authors show that constant attention over time is not an optimal strategy. Instead, depending on the cost of attention, the optimal approach is to vary attention levels over time in either a "rhythmic" or a "ramping" pattern.

Critical strengths

The paper clearly communicates its core message regarding the optimal strategy in a specific task. I find their findings interesting and intuitive, and they may provide a theoretical explanation for fluctuations in attention levels over time during sustained attention tasks.

Critical weaknesses

Because the paper relies on a theoretical model, the absence of detailed theoretical explanations and equations in several sections makes it difficult to link certain intuitive results directly to the theory. It would be helpful if the authors connected their empirical results more explicitly to the underlying framework where possible.

And the results presented might be limited to the specific task chosen here and may not be able to explain a more general sustained attention task, as I will elaborate on in the questions.

Questions for authors

1.It seems the signal always appears at the end of a trial in this task, meaning the agent knows there will be no subsequent noise, regardless of the number of signal states, m. Is this correct? If so, does this limit the findings strictly to this specific task format? If the task involved multiple appearances of the signal interleaved with noise, the optimal strategy would likely differ?

  1. The paper states that the agent's objective depends on penalties as well as costs and rewards, yet there is no further discussion of penalties. Is the penalty a fixed value across all tasks? One might expect the optimal strategy to vary under different penalty conditions, with the relative significance of penalties compared to rewards or costs altering the policy. Adding a theoretical/empirical discussion on this would strengthen the paper.

  2. Figure 2 illustrates the number of jumps in attention level, but how does the underlying theory account for the number of these jumps?

  3. Given that the signal always appears at the end of the trial, it is unclear why the qualitative result should differ for any m > 1 (as shown in Figure 2b). Once the agent observes the first signal, it seems it should reasonably infer that committing at every remaining step until the end of the trial (or first signal state) is the optimal action.

Minor points:

  1. The text description (L156–161) does not seem align with Figure 1B; for instance, p represents the transition probability from noise to signal, rather than between signal states.

  2. In the caption for Figure 1, there appears to be an error: the belief at time t is a function of the observation at time t and the action at time t-1, rather than the action at time t.

3 comments

Thank you for your thoughtful review. We have expanded on our model with equations in the new A.1 and A.2 in the appendix to better present the theory behind the work. Below we address your concerns regarding the generalizability of the results.

Q1: It seems the signal always appears at the end of a trial in this task, meaning the agent knows there will be no subsequent noise, regardless of the number of signal states, m. Is this correct? If so, does this limit the findings strictly to this specific task format? If the task involved multiple appearances of the signal interleaved with noise, the optimal strategy would likely differ?

A1: This does not limit the findings strictly to this specific task format. To demonstrate, we added a section to the appendix (A.11.1) where we alter the transition matrix to optimize policies for a world where each trail may contain multiple signals (an average of 3) interleaved with noise. We found that policies are 99% correlated with the policies for the single-signal-per-trial world. This is because the inference is causal and can only use past evidence, so future noise does not play a role in assessing whether the signal is currently present.

Q2: The paper states that the agent's objective depends on penalties as well as costs and rewards, yet there is no further discussion of penalties. Is the penalty a fixed value across all tasks?

A2: The penalty is detailed in Appendix A.3 (formerly Appendix A.1). In brief, varying hit reward, commit cost, and false alarm penalty would be redundant because the agent only ever sees two combinations: hit reward with commit cost and false alarm penalty with commit cost. Therefore varying only two of the three can still show us the full range of net reward on a hit vs. a false alarm. We added a section (Appendix A.11.3) demonstrating a range of false alarm penalties. We found that, for default parameters, when there is no false alarm penalty and no commit cost, the agent does not attend. It commits frequently, at a set interval after each trial start. In this case, there is no cost to being wrong. Once a penalty or commit cost is introduced, rhythmic attention becomes optimal. Rhythmic attention continues to be optimal for increasing penalties.

Q3: Figure 2 illustrates the number of jumps in attention level, but how does the underlying theory account for the number of these jumps?

A3: The number of jumps relates to the attention cost exponent. We detailed this relationship in Sections 4.1 and 4.2. To provide a brief synthesis: a low attention cost exponent incentivizes sparse, large installments of attention, leading to rhythmic, regular allocations. These allocations are at one level, so the change in the belief space from no attention (wait) to higher attention (rhythmic allocation) appears as a jump in the policy where belief passes a threshold. A high attention cost exponent incentivizes frequent, small installments of attention, but a large installment of attention is needed for reliable observations. Therefore ramping, where a larger installment of attention is allocated only after observing signal with a smaller allocation, is optimal. The ramps require multiple levels of attention that grow with belief, so they appear as multiple jumps in the policy.

Q4: Given that the signal always appears at the end of the trial, it is unclear why the qualitative result should differ for any m>1m>1 (as shown in Figure 2b). Once the agent observes the first signal, it seems it should reasonably infer that committing at every remaining step until the end of the trial (or first signal state) is the optimal action.

A4: The agent can only commit once per trial. A commit immediately ends the trial. Thus a hasty commit loses the reward in many cases, so the optimal policy does not commit the first time a signal observation occurs unless observation accuracy is high enough to justify the choice.

Regarding the minor points, both are a good catch, thank you for the notes. We have corrected some of the language and variable names used for transition probabilities to address this inconsistency, and we have adjusted the figure to accurately represent the model implemented.

Dear Reviewers,

The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!

Best, AC

Thank you to the authors. I found the answers to my questions, the additional theory provided, and the new experiments very informative, especially in Sections A.11.1 and A.11.3 (Figures 12 and 15). I suggest referring to these results in the main body, I even found the result for the "no penalty" -> "no attention" -> "commit frequently" case intuitive and interesting. I will increase my rating and recommend accepting the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.