acceptodds
Under review as a conference paper at ICLR 2027

Regulating the Past: Computational and Neural Signatures of Persistence Control

Abstract

Continual learning requires preserving useful knowledge while adapting rapidly when that knowledge becomes obsolete. Biological learners solve this problem across a continuous history. Yet adaptive-learning models primarily regulate how strongly new evidence updates a state, while analyses of human learning often reset the learner at block boundaries. Such resets can conceal how accumulated confidence constrains subsequent adaptation. We study a complementary control problem: how strongly should inherited state continue to constrain learning? We call this persistence control, and instantiate it as prior-directed tempering in the Hierarchical Gaussian Filter, a widely used model of human learning under uncertainty. In simulated HGF learners with continuous histories, accumulated confidence restricted the influence of new evidence even while volatility remained represented. Tempering restored responsiveness, and mechanism recovery distinguished persistence control from surprise-modulated volatility control. In non-stationary bandits, relaxing prior influence was beneficial when contexts changed frequently, whereas preserving it was advantageous when contexts endured. Human behavior favored graded persistence control over carrying everything forward or resetting at detected changepoints. Fitted trajectories showed that persistence in reward beliefs rose after an isolated misleading outcome and was released after contradictory evidence accumulated. Successful learners also reconfigured persistence more strongly around genuine reversals rather than maintaining uniformly lower persistence. Feedback-locked frontal-midline theta covaried with this protection following misleading outcomes. Together, these results identify persistence as a distinct target of adaptive control. Continual learning, in machines and people, may depend not only on how strongly new evidence is incorporated but also on when prior beliefs should remain influential and when they should be released.

Primary area:
applications to neuroscience & cognitive science
Keywords:
continual learning, active forgetting, non-stationary environments, adaptive learning, cognitive neuroscience

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_LWfgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how learners preserve or discard past information in changing environments, using five approaches:

  1. HGF (hierarchical Gaussian Filter) simulations testing whether forgetting helps learners adapt after changes while retaining useful information;
  2. Model-recovery experiments testing whether persistence control can be distinguished from volatility adaptation;
  3. A non-stationary bandit comparing control over forgetting with control over learning rates;
  4. A reversal-learning experiment with 74 participants, comparing models and relating fitted persistence to behaviour;
  5. EEG analyses testing whether frontal-midline theta is associated with fitted persistence.

It proposes “persistence control”: before each learning update, the learner relaxes its previous beliefs toward an initial prior. The amount of forgetting is either constant or determined by the previous outcome’s surprise.

The conclusion is that persistence is a distinct target of adaptive control, with evidence from simulations, human behaviour and EEG.

My main concern is that the results are either given in advance or already known. While the human and EEG analyses do not convincingly demonstrate the claimed distinction.

Critical strengths

The authors validate their counterfactual replay, separate bandit-controller selection from evaluation, and report negative results. These include a competitive leaky-RL baseline, a null choice-stream protection effect and null results in the stricter EEG analyses.

Critical weaknesses

I found the paper to be hard to follow, poorly structured and the tasks, simulation, and fitting procedures lacked explanations. My assessment is that there was very little surprise in the findings presented in the paper, everything follows naturally from the setting.

  1. The novelty is unclear. Interpolating Gaussian natural parameters toward a reference prior implements established stabilised forgetting. Moreover, related work also infers the forgetting weight from data, rather than specifying it through a fixed surprise-dependent function, which is more robust and less prone to leaking the design knowledge in the modelling assumptions. Relevant comparisons include Kulhavý & Kraus (1996), Masegosa et al. (2017), Moens (2018), Moens & Zénon (2019), and Yu & Cohen (2009). The paper needs to clearly explain what is new beyond applying this operation within the HGF.

  2. The experiments do not establish one coherent claim. Forgetting improving responsiveness and helping more in rapidly changing environments are expected consequences of the tracking–noise trade-off. Additionally, the long-history failure that motivates the simulations isn't clearly tested by the human task which is very short. The learning-rate comparison uses a different learner. The changepoint baseline is a restrictive threshold-reset model, rather than a full Bayesian changepoint or inferred-forgetting alternative.

  3. The model may produce these effects by design. With the fitted settings, it keeps more of its past beliefs after a surprising outcome. This may explain the reported “protection” effect. The model only uses surprise from the last trial. Consequently, forgetting after a second unexpected outcome does not show that it has combined evidence across trials. Better players also experience different outcomes which could explain the link between protection and performance. The authors should check whether this link remains when they use the same model settings for every participant.

  4. The model comparison uses participants’ choices, but the protection and forgetting effects come from a separate fit to their rewards. This matters because the fit that wins the comparison does not show the protection effect. Its success therefore does not directly support the main finding. The authors also leave unclear how the model predicts choices. This makes the comparison hard to assess. They do not test whether the task provides enough information to estimate the model’s settings reliably. Along the same line, the EEG comparison contrasts an unexpected non-reward with an expected reward. This is a problem because reward versus no reward already affects frontal-midline theta. The observed difference could therefore have a familiar explanation.

  5. The presentation completely obscures the contribution: a) the abstract should clearly identify the simulations, bandit study and human EEG experiment. There should be a list of tasks that are done in the paper (eg, "We test our hypothesis in simulation, behavioural experiments and in an EEG recording setting). Methods and results need organizing around explicit experiments and questions, and it should be clear when skimming over the paper what experiment (simulation/behavioural/EEG) we are looking at, and what questions is being answered.

  6. State variables and update operators require clearer definitions, terminology should be consistent, and several settings and figure elements are insufficiently explained.

Questions for authors

How do the fitted surprise slopes vary across participants? Does the protection effect show anything beyond the direction of those slopes?

What is the full observation and response model for the human task? How do the choice-stream and reward-stream fits relate? Can the gate parameters be recovered in the actual task?

Do the behavioral correlations survive when identical model parameters are applied to every participant’s outcome stream? Does theta predict persistence beyond surprise and reward versus non-reward?

How does the method compare with inferred-forgetting models and stronger Bayesian changepoint baselines? What happens when forgetting is applied separately at each HGF level?

Please report the missing simulation settings and joint-controller results. Please also define “reversal surge” and “reconfiguration.” Which result most clearly distinguishes this account from existing explanations?

2 comments
AuthorsOpenReview ↗

Response to LWfg (1/2)

(1/2)

We really appreciate and thank the reviewer for taking out the time on our paper to produce such a thorough and constructive review. It pushed us to test directly whether our results follow from the model's design, and those tests now anchor the revised paper!

We accept the criticism of the presentation and have restructured accordingly (point 5). Some patterns do follow from the setting, and we now say so. The central result does not: participants' evidence for surprise-dependent persistence control (ΔLME = 81) exceeds that of all 20 simulated cohorts without the mechanism (31 to 57), and of all 20 cohorts of learners fitted to each participant's own choices (52 to 65).

1. Novelty relative to stabilised and inferred forgetting. We agree that the operator is stabilised forgetting in its Gaussian form, and we now state this explicitly, with a comparison to the works the reviewer lists in the Supplement (B.6). Our weight is estimated, not fixed by design: both its baseline and the sign of its surprise dependence are fitted, and both signs were available. We also fitted the dynamic belief model of Yu & Cohen (2009), with exact Bayesian inference over every possible changepoint history: it fits comparably by BIC (ΔBIC = 7 in favour of DBM) but less well by model evidence (ΔLME = 233 in favour of our model), and we report both. What is new is not the operator but what we establish with it:

  • It is distinct from volatility control in the same learner (30/30 cohorts each way), and a precision-only variant, which mimics volatility, fails (ΔLME −476).
  • When release helps or hurts an artificial learner: it helps when contexts are short and hurts when they last, also within a single learner.
  • The protective, surprise-dependent form people use beats a constant rate and discrete resets, even resets timed exactly at the true reversals (ΔBIC +390).
  • The human evidence is calibrated against learners without the mechanism (point 3).

2. "The experiments do not establish one coherent claim."

  • Expected consequences. We agree the trade-off is expected; we test whether it survives in a learner that already represents volatility. Such a learner is still slowed by a long history (9–21 more trials to recover after 1,000 stable trials than after 50), whereas a learner with persistence control does not slow at all (C.8).
  • Long history in the human task. We agree, and tested it: post-reversal recovery does not slow across the session, in participants or in the fitted standard model, so we now rest the long-history claim on the simulations and say so (E.6).
  • Different learner. Within a single hierarchical learner, with the same surprise signal acting on carryover or on learning rate, the boundary replicates: carryover control wins 22 of 27 short-context and 8 of 27 long-context environments (D.4).
  • Changepoint baseline. Graded release outperforms every threshold-reset variant, including resets at every true reversal (ΔBIC +390 to +1,182; C.11), and the dynamic belief model is reported with both criteria (point 1).

3. "The model may produce these effects by design." We thank the reviewer for this point; it led to the analyses we now consider most important. With one shared parameter set for every participant, the protect-then-release pattern remains (dz 2.05 → 3.40) and the reversal-surge link to performance survives (ρ = 0.36 with the learning index, 0.46 with reward); the protection–performance link does not (ρ = 0.18), and we have removed it (C.16). Gate-free learners also show the protect-then-release trajectory and the surge link, so we present these as descriptions of the fitted gate. What exceeds these learners is the evidence for surprise-dependent persistence control (ΔLME 81 vs 31 to 57; C.13).

4. Streams, response model, recoverability and the EEG contrast.

  • One fit for everything: all model-based results now use the choice-stream fit that wins the model comparison (rise dz = 2.05, fall dz = −1.55); the reward-stream analysis is a labelled supplementary result (B.5).
  • The model predicts, trial by trial, whether the participant chooses the best option; there is no separate choice rule. Full specification is in B.4.
  • Recovery in this exact design: the baseline gate and volatility are recovered well (r = 0.87 and 0.85), with no trade-off; the individual slope moderately (r = 0.68, sign correct for 81%), so we report the slope at the group level (C.14).
  • EEG and reward: among non-rewarded outcomes only, theta is still larger when the loss contradicts a correct choice (dz = 0.40–0.50, q < .01; F.5).

5. Definitions and settings. We added a definitions box and harmonised notation (2.1). Reversal surge: the rise in persistence immediately after a true reversal, relative to the preceding trials. Reconfiguration: the absolute trial-to-trial change in persistence. All simulation settings are in the Supplement (C.7, D.4).

AuthorsOpenReview ↗

Response to LWfg (2/2)

(2/2)

Q1. "How do the fitted surprise slopes vary across participants? Does the protection effect show anything beyond the direction of those slopes?"

We estimated each participant's slope over a grid from −6 to +6 under the same N(0, 1) prior used elsewhere. The direction is consistent across people (posterior-mean slope median −0.54, negative in 68 of 74), but 200 trials determine each person's slope only moderately, so we make no individual-slope claims. Both signs were available, and a positive slope fits worse than no slope at all. Beyond the direction, the slope captures three concrete features of behaviour:

  • People lapse even during a run of correct choices; the negative slope relaxes slightly when nothing is surprising, which keeps the model from becoming overconfident.
  • A first error predicts another: right after a surprising error, participants chose the best option only 31% of the time, and the negative slope holds the updated belief rather than resetting it.
  • People then recover: once the surprise has passed, the slope releases the belief, and choices of the best option return to 38%.

A positive slope gets all three wrong. What goes beyond the direction is the strength of the evidence for surprise dependence, calibrated against gate-free learners (point 3). The slope distribution and this decomposition are now in C.15. This probe was really helpful for us, and we thank the reviewer for it!

Q2. "What is the full observation and response model? How do the choice-stream and reward-stream fits relate? Can the gate parameters be recovered?"

We should have specified the model fully in the submission; it is now in Supplement B.4–B.5. The input on each trial is whether the participant chose the objectively best option, with all 200 trials fitted as one sequence. A three-level binary HGF predicts this outcome from its own history, with the gate βₜ relaxing the inherited beliefs before each update; the likelihood is the model's one-step prediction. Free parameters are ω2, ω3 and b0, with bᵥ shared across participants; estimation is MAP, with Laplace model evidence and BIC. The reward-stream fit applied the same model to whether each trial was rewarded; its parameters are only weakly related to the choice-stream fit (ρ = 0.21 and 0.09), so the two estimate different quantities. Recovery: see point 4.

Q3. "Do the behavioral correlations survive identical model parameters? Does theta predict persistence beyond surprise and reward?"

Identical parameters: see point 3. Within a participant, fitted persistence is a fixed function of the previous trial's surprise, so no signal can predict it beyond surprise at the trial level, and we now state this explicitly. Of the tests that can be identified: theta is larger at a contradicting loss with reward held constant (dz = 0.40–0.50, q < .01); between participants it does not track how strongly each person gates; and it does not predict the next choice once surprise and reward are accounted for (F.5).

Q4. "How does the method compare with inferred-forgetting models and stronger Bayesian changepoint baselines? What happens when forgetting is applied separately at each HGF level?"

The dynamic belief model: see point 1. A leaky delta rule fits about as well by model evidence (ΔLME −2.9) and better by BIC (ΔBIC −385), but its fitted leak did not distinguish better from worse performers, as persistence did (C.12); we present our model as one implementation of regulated carryover. The known-reversal reset bounds any changepoint model that resets the learner, including a full Bayesian one, since no detector can place its resets better than the true reversal times; graded release still fits better (ΔBIC +390; C.11). Forgetting at each level: level 2 alone retains part of the gain (ΔLME −51 relative to both levels), the mean alone also (−38), while level 3 alone (−500) and precision alone (−476) fail. The effect therefore depends on relaxing the inherited level-2 belief: persistence control acts on what the learner believes, not on how volatile it takes the world to be (C.9).

Q5. "Missing simulation settings and joint-controller results; definitions; which result most clearly distinguishes this account?"

Settings and definitions: see point 5. The joint controller beat the better single controller in only 20% of environments; its search grid was not matched in complexity, so we draw no conclusion from it, and in aggregate held-out performance learning-rate control or no control did best. This is why the paper frames the result as an environmental boundary rather than dominance. The distinguishing result is the comparison with learners that lack the mechanism: ΔLME 31 to 57 (52 to 65 when fitted to each participant's choices) against 81 in participants. The mechanism is also recovered without confusion from volatility-based accounts (30/30 each way), and reversing the fitted gate worsens every participant's fit (74/74).

Reviewer_8CW9OpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The main idea of this paper is that learning isn’t just about deciding how much to trust new information, but also how much to hold onto what we already know. The authors build a model that can adjust how strongly past beliefs influence future decisions and compare it with other learning methods. The key finding is that being flexible about what to remember or forget helps in changing environments, and that people who adjust their beliefs more selectively tend to perform better.

Critical strengths
  • paper looks at forgetting as something useful rather than always treating it as a problem
  • compares to other learning methods to explain how their approach is different
Critical weaknesses
  • experiments use fairly simple tasks; I’m not too confident whether the findings would hold in more complicated real world situations
  • brain recordings show a connection with the proposed mechanism, but they don’t prove that this brain activity actually causes forgetting (Section 4.5)
  • some results suggest that a simpler, fixed forgetting approach performs just as well, which makes the need for a more flexible method less convincing (Section 3.1)
Questions for authors
  • why does adjusting for forgetting based on new info sometimes perform no better than forgetting at a fixed rate?
  • since the brain activity measured is also involved in other kinds of learning and decision making, how can we tell whether it’s specifically related to controlling forgetting? How confident can we be that the model is actually capturing how people forget rather than just fitting their behavior?
1 comment
AuthorsOpenReview ↗

Response to 8CW9

We thank the reviewer for their time and for recognising the value of treating forgetting as useful rather than as a failure, your questions helped us sharpen what the paper shows, and we address each below.

  1. "experiments use fairly simple tasks; I'm not too confident whether the findings would hold in more complicated real world situations"

We agree that real-world generalisation remains open, and we appreciate the reviewer raising it. The task was chosen deliberately for the question we study, in line with the paper's focus on neuroscience and cognitive science. It is a three-option probabilistic reversal task in which reward stochasticity and volatility are manipulated factorially, so isolated misleading outcomes and genuine changes arrive through the same feedback stream. We fitted each participant's full 200-trial history as one continuous stream, which preserves the opportunity to detect history-dependent effects that would be obscured by independently resetting the model at block boundaries. Beyond the human task, the computational result was tested across 81 simulated environments varying context duration, drift, noise and change magnitude, and it replicated within a single learner.

  1. "brain recordings show a connection with the proposed mechanism, but they don't prove that this brain activity actually causes forgetting"

We agree, and we do not claim causality. The EEG results (Section 3.4) are presented as convergent rather than causal evidence. To make the neural result more specific, we added a reward-matched comparison: comparing only non-rewarded outcomes, frontal-midline theta is larger when the loss contradicts a correct choice than when it follows an incorrect one (dz = 0.40–0.50, q < .01).

  1. "some results suggest that a simpler, fixed forgetting approach performs just as well, which makes the need for a more flexible method less convincing (Section 3.1)"

We thank the reviewer for raising this, since it separates two different results:

  • In simulated agents (Section 3.1), the fixed and adaptive versions tie in with the task score.
  • In human data, the adaptive version is clearly preferred. A surprise-dependent gate fits participants' choices better than a fixed forgetting rate (ΔLME = 81; better in 68 of 74 participants). Removing its surprise dependence worsens the fit in 72 of 74 participants.

Q1. "why does adjusting for forgetting based on new info sometimes perform no better than forgetting at a fixed rate?"

The simulations suggest that adaptive release may not be necessary to improve reward performance in all these environments. However, human choice-correctness sequences are better described by a surprise-dependent gate than by constant release, indicating that the additional flexibility captures behavioral structure not explained by the simpler model.

Q2. "since the brain activity measured is also involved in other kinds of learning and decision making, how can we tell whether it's specifically related to controlling forgetting? How confident can we be that the model is actually capturing how people forget rather than just fitting their behavior?"

We agree that frontal-midline theta is not specific to forgetting, and we do not claim it is. It is not explained by reward (above), and it does not index how much is forgotten: theta does not predict how strongly each participant forgets, nor their next choice once surprise and reward are accounted for. We therefore present theta as marking the moment an established belief is contradicted, the event to which persistence control responds, rather than as a signal of forgetting itself.

For the second part, we thank the reviewer for this question, which goes to the heart of the paper as it prompted the analysis we now consider most important. If the model merely fitted behaviour flexibly, it should find the same evidence for its forgetting mechanism in learners that do not have one. We simulated learners with no persistence mechanism (a standard Bayesian learner and a reinforcement learner) on each participant's exact task, 20 cohorts of 74, and fitted them with the identical pipeline.

Evidence for surprise-dependent forgettingΔLMEΔBIC
Participants81114
Simulated learners without the mechanism (20 cohorts)31 to 5718 to 54
Same learners, fitted to each participant's own choices52 to 6527 to 72

This advantage is consistently smaller than that observed in participants, which suggests that the human result is not fully explained by the particular gate-free mechanisms tested, while leaving open other possible accounts. Three further tests point the same way: recovery is clean (30/30 each way); reversing the direction of the fitted forgetting signal worsens every participant's fit (74/74); and variants that act on uncertainty alone fail (ΔLME −476).

Reviewer_qoReOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This study addresses the central question of how to maintain past knowledge while adapting to new contexts. The authors proposed "persistence control", a computational mechanism which weakens the influence of prior beliefs before each model update. Through simulation of non-stationary three-armed bandit tasks and model fitting to human reversal learning data, the authors found that selectively reducing persistence through the persistence control gate can facilitate adaptaion when environments change frequently, but may interfere with adaptation when earlier context returns. Model fitting revealed that humans tend to persist with prior reward beliefs when encountering an isolated misleading outcome and release it after contradictory evidence accumulates.

Critical strengths
  1. The question probed in this study, namely how past beliefs are updated during future learning, is one of interest to both the cognitive science and machine learning community
  2. The proposed active forgetting mechanism makes intuitive sence and is at least partly supported by simulation and model fits to human data.
  3. The model comparisons and data fitting are generally well executed. The EEG data is valuable and begins to reveal how the brain regulates the updating of past beliefs.
Critical weaknesses
  1. The main contribution is more relevant to cognitive science than machine learning. The experiments are limited to simple bandit and probabilistic reversal tasks, which revealed a task-dependent advantage in adaptive forgetting and distinct biases in how humans retain prior beliefs upon misleading outcomes. These findings are interesting, but does not clearly show a path towards a general, scalable machine learning algorithm. No experiments are done on standard continual learning or non-stationary RL benchmarks to demonstrate the applicability of this mechanism to complex machine learning problems.
  2. Novelty compared to existing active forgetting methods is moderate to weak. The "persistence control" mechanism essentially mixes previous posterior with the initial prior in an adaptive way. This is conceptually related to a number of existing studies on forgetting factors (Kulhavy and Zarrop 1993), value decay (Ito and Doya 2009), posterior tempering/adaptive partial forgetting (Li et al. 2021), Bayesian forgetting (Kurle et al. 2020), context-dependent gating (Zeng et al. 2019), to name a few. The author should more firmly establish what is novel about their proposed mechanism relative to exsiting studies.
  3. The central "persistence" mechanism is not directly demonstrated at the behavioral or neural level. The authors fitted an AF-HGF model with a latent persistence variable to the human behavioral data. So persistence is part of the model design rather than directly identified through behavioral analysis. The EEG analysis is also based on correlation to a model-derived persistence value instead of a model-agnostic discovery of a persistant neural signal. With comparison to only a restricted set of alternative models, it is not entirely clear if the brain actually implements "persistence control" or has a "persistence gate".

References: Kulhavý, R., & Zarrop, M. B. (1993). International Journal of Control Ito, M., & Doya, K. (2009). Journal of Neuroscience Li, A., Boyd, A., Smyth, P., & Mandt, S. (2021). NeurIPS Zeng, G., Chen, Y., Cui, B., & Yu, S. (2019). Nature Machine Intelligence

Questions for authors
  1. How is the persistence control mechanism fundamentally new or techinically superior compared to exisiting active forgetting mechanisms?
  2. Can the human behavior or EEG data directly reveal a persistence mode/signal withouting fitting to a model with a pre-specified persistence parameter/variable?
  3. If persistence control can interfere with adaptation to current context, how can it serve as an competivie algorithm in practical continual learning problems?
  4. Can the authors evaluate the persistence mechanism on at least one continual learning benchmark to demonstrate its utility beyond simple bandit settings?
1 comment
AuthorsOpenReview ↗

Response to qoRe

We thank the reviewer for these questions and for the references. They helped us state the contribution much more precisely.

Q1. "How is the persistence control mechanism fundamentally new or technically superior compared to existing active forgetting mechanisms?"

The operator is not new. Relaxing a learned state toward a prior belongs to the stabilised-forgetting family (Kulhavý & Zarrop, 1993; Kurle et al., 2020), and we add a comparison with the works the reviewer lists to the Supplement. Zeng et al. (2019) addresses a related but different problem, protecting learned representations from interference; neither approach regulates how strongly past evidence shapes current beliefs, which is the operation we study. What is new is what we test with it:

  • It is distinct from volatility control in the same learner (30/30 cohorts each way), and a precision-only gate, which acts like added volatility, fails on human data (ΔLME −476).
  • When release helps or hurts: it helps when contexts change often and hurts when they persist, also within a single learner (22 of 27 vs 8 of 27 environments).
  • A surprise-dependent persistence model fits participants' choices better (ΔLME 81; 68/74 participants) than discrete resets, including resets timed exactly at the true reversals (ΔBIC +390).
  • The human evidence is calibrated: learners with no gate show ΔLME 31 to 57, and 52 to 65 even when fitted to each participant's own choices, against 81 in participants.

We do not claim a generally superior algorithm. We present persistence control as an account of how people regulate the influence of past evidence, with the HGF as one implementation, in line with the "applications to neuroscience & cognitive science" track.

Q2. "Can the human behavior or EEG data directly reveal a persistence mode/signal without fitting to a model with a pre-specified persistence parameter/variable?"

This is a pretty good litmus test and we appreciate the probe! We added model-free analyses, comparing participants with simulated learners that have no persistence gate but play each participant's exact task (a Standard HGF agent and a Q-learner, 10 cohorts of 74 each), with broad parameters and fitted to each participant (fitted learners match human accuracy, 0.49–0.53 vs 0.54).

P(stay) on the next trial
After a win on an established option0.96
After a first loss on it0.44
After a second consecutive loss0.32

No gate-free learner reacts to the first contradiction as sharply: their drop is −0.11 to −0.36, against −0.52 in participants. P(stay) and model persistence measure different things: P(stay) asks whether people repeat the option, persistence asks how much of the accumulated belief is carried forward. People readily leave after one contradiction, but that single departure does not overwrite what they have learned about which option is best.

EEG: frontal-midline theta rises at the first contradiction of an established belief (+1.6 dB, dz = 0.73), and it is not explained by reward (dz = 0.40–0.50, q < .01, non-rewarded outcomes only). These observations establish sensitivity to contradiction but do not uniquely identify persistence regulation; the calibrated model-based test does. We thank the reviewer for raising this, as it helped us separate what behaviour and EEG reveal directly from what requires the model.

Q3. "If persistence control can interfere with adaptation to current context, how can it serve as a competitive algorithm in practical continual learning problems?"

We agree that persistence control is not universally beneficial, and showing exactly that is one of the paper's aims. Release pays when contexts change often and costs when they persist (21 of 27 short-context vs 2 of 27 long-context environments), and repeated release has a measurable retention cost when an earlier context returns (C.6). This suggests that methods which relax learned state at a constant rate, such as shrink-and-perturb, should help under frequent change and hurt when contexts are long, so relaxation should be conditioned on how quickly the environment changes.

Q4. "Can the authors evaluate the persistence mechanism on at least one continual learning benchmark?"

We thank the reviewer for this suggestion. This paper primarily presents a computational cognitive science approach and was not intended to be a state-of-the-art continual learning algorithm, so we have not compared against a benchmark; we agree it is the natural next step. Within the rebuttal period we prioritised testing whether the result generalises beyond our learner: the boundary replicates inside a single hierarchical Bayesian learner across 81 held-out environments (22 of 27 vs 8 of 27). We have added this prediction to the Supplement as a direction for benchmark evaluation, and thank the reviewer for encouraging us to connect the work more directly to practical continual learning.

Reviewer_wFKwOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes persistence control: a pre-update gate that tempers inherited beliefs toward the initial prior before the ordinary update, instantiated in the HGF as surprise-gated mixing of the level-2/3 statistics with μ0/π0, β_t = σ[b0 + bv(S_{t−1} − log 2)]. Evidence comes from (i) HGF simulations of an A→B transition, (ii) an 81-environment three-arm bandit sweep comparing adaptive carryover with adaptive learning-rate control, and (iii) 74 participants on a three-patch reversal task with EEG, comparing nested HGF variants, a volatility-modulated HGF, and a BOCD HGF, and relating the fitted protect/release dynamics to frontal-midline theta. The paper concludes that release is computationally distinct from gain control, helps in short contexts and hurts in durable ones, and that humans selectively protect after isolated contradiction and release as contradiction accumulates.

Critical strengths

1. The counterfactual schedule test is a strong identification strategy. Supp. C.2 holds everything else fixed and replaces only β_t: flipping the gate's sign costs 9.86 NLL in 74/74 participants and removing surprise dependence costs 6.42 in 72/74, while freezing, delaying, or shuffling costs only 1.4–2.0, and the fitted schedule beats 92.97% of 1,480 shuffles. Together with the matched recovery (30/30 each direction) and the ω2 non-redundancy check in C.4, this is a better-targeted test of the timing of release than most work in this area.

2. The "where, not how much" result is a clean dissociation. Mean persistence does not differ across reward quartiles (0.0228, p = 0.704) but reversal-linked surges do (0.1667 vs 0.0653, g = 1.222). The paper also reports results that cut against its story — the unchosen-only leaky RL tie, the retention cost, and the null choice-stream contrast — which is good practice.

Critical weaknesses

No parameter table is given for the three agents, and the only values in the paper (Supp. C.6: ω2 = −4 vs ω2 = −2) are explicitly said to differ from 3.1. At β = 0.121 the tempered agent is better in both phases (0.741 vs 0.642 in A; 0.488 vs 0.398 in B) and its A→B drop is slightly larger (0.253 vs 0.244), so the advantage is a level shift rather than reduced transition degradation; only β = 0.30 has a smaller drop (0.192), with the lowest phase-A accuracy. The human model comparisons have a related reporting gap: free-parameter counts, priors, optimizer, the LME approximation, and μ0/π0 are not stated, so ΔBIC = 693 cannot be checked, and there is no code or data-availability statement. Since a simpler leaky RL agent matches AF-HGF (207.27 vs 206.5), the model comparison alone does not yet carry the mechanism claim.

With the fitted bv < 0, higher lagged surprise mechanically raises persistence, so protection after an isolated oddball largely restates the sign of bv; the informative parts are the release after the second oddball (−0.0883) and the reversal surges. The protect/release trajectory is fit to the reward stream while the model comparison and event-level contrast use choices, where the analogous contrast is null (−0.00643, q = 0.135). Finally, the BOCD comparator thresholds at 9.64 nats, giving 1.43 resets on average with 45/74 participants never resetting, and its 20-trial cooldown exceeds the 10-trial reversal period of the fast blocks, so it cannot detect those reversals; as implemented, the comparison does not yet show that graded release outperforms discrete resets.

Questions for authors
  1. Were ω2, κ, θ, and inverse temperature identical across the three agents in 3.1, with only β differing? Please give the parameter table; if not, re-run matched and report the phase × model interaction. This is the point on which my assessment most depends.

  2. Was a precision-only gate (no shrinkage of the mean toward μ0) fitted? If it captures most of the LME gain, the "distinct computational locus" claim needs to be restated.

  3. Does adding an explicit oddball/repetition term to β_t improve model evidence beyond the surprise link? And is the C.2 schedule, taken from the reward-stream fit, evaluated against the choice stream?

  4. Was the BOCD threshold per-participant or fixed at 9.64 nats? Could you report a capacity-matched comparison (matched reset rate, or a cooldown shorter than 10 trials)?

2 comments
AuthorsOpenReview ↗

Response to wFKw (1/2)

(1/2)

It was indeed a pleasure going through the questions raised and thank you for the time you have spent on our paper. Below we address all the questions you had for us and, towards the end, list all that we have amended in our submission!

Q1. "Were ω2, κ, θ, and inverse temperature identical across the three agents in 3.1, with only β differing? Please give the parameter table; if not, re-run matched and report the phase × model interaction."

We are really grateful for this question in particular! Answering it led us to correct two reporting errors and to add a direct test of the history claim.

Section 3.1 described β = 0.121 as the cohort mean. That value came from a superseded set of fits; the Static AF-HGF fits behind every other number in the paper give β = 0.287 (SD 0.214). We have corrected E.1 and 3.1. In the matched rerun, all agents share ω2 = −4, log θ = −6, κ1 = κ2 = 1, inverse temperature 10, the same initial states and the same 100 paired environments. Only β differs. (The Standard HGF cannot run at the AF agents' faster settings: its volatility estimate diverges in 76 of 100 environments at ω2 = −2, log θ = −4.)

βPhase APhase BA→B dropPhase × model (Δdrop vs Standard)
0 (Standard HGF)0.6420.3980.244—
0.1210.7370.4480.289+0.045 [0.021, 0.069]
0.287 (cohort mean)0.6930.4770.216−0.028 [−0.052, −0.005], p = .019
0.300.6860.4740.211−0.033 [−0.057, −0.010], p = .006
  • With all other parameters matched, tempering improves accuracy in both phases at every β, and at every matched setting over ω2 ∈ {−6, …, −3} × log θ ∈ {−6, −4, −2}.
  • At the participants' actual mean release (β = 0.287), the tempered agent loses less accuracy across the transition than the Standard HGF (interaction −0.028, p = .019). This is the predicted reduction in transition cost.

Phase A itself contains reversals, so we added a direct test of the history claim (new Supplement C.8): after a stable phase of 50 to 1,000 trials and one switch, Standard HGF needs 9.5 more trials to recover after 1,000 stable trials than after 50 (20.8 under high noise), whereas trial-wise AF does not slow (difference −9.2 [−11.1, −7.4] trials, p < .001). What accumulates is Standard HGF's belief about the options it has stopped sampling, which drifts downward with history (0.25 to 0.075); tempering keeps those beliefs near the prior. We have revised the explanation in 3.1 accordingly. This comment was immensely helpful for us and we thank the reviewer again!

Q2. "Was a precision-only gate (no shrinkage of the mean toward μ0) fitted? If it captures most of the LME gain, the 'distinct computational locus' claim needs to be restated."

The precision-only gate does not capture the gain. It fits worse than the full gate by 476 nats, and worse even than a constant gate! In evidence it is nearly indistinguishable from the surprise-modulated volatility model (−476 vs −498): lowering inherited precision acts much like adding volatility, and in these data neither explains behaviour.

ModelΔLME vs full gateΔLME vs constant gate
Full gate (levels 2 + 3, mean and precision)0+81.3
Mean-only−38.2+43.1
Level-2-only−51.2+30.1
Precision-only−476.2−394.9
Surprise-modulated volatility−498.4−417.1
Level-3-only−500.3−419.0

In recovery, data generated by mean-relaxing gates were never attributed to the precision-only, level-3-only or volatility accounts, and vice versa (5/5 cohorts each way); whether precision is relaxed together with the mean is not identifiable. Following the reviewer's suggestion, we have restated the locus claim in Sections 2.1 and 4: "The gain requires relaxing the inherited level-2 belief. A gate acting on precision alone is insufficient and is evidentially equivalent to surprise-modulated volatility, and the two accounts are separable in recovery." The variant comparison and recovery results are in new Supplement C.9.

AuthorsOpenReview ↗

Response to wFKw (2/2)

(2/2)

Q3. "Does adding an explicit oddball/repetition term to β_t improve model evidence beyond the surprise link? And is the C.2 schedule, taken from the reward-stream fit, evaluated against the choice stream?"

Which stream C.2 uses: the counterfactual schedule test was run entirely on the choice stream, the same data and fitted model as the model comparison (the replay reproduces the saved likelihood to about 5 × 10⁻¹³). Reversing the gate's sign worsens the fit in 74 of 74 participants, and removing its surprise dependence worsens it in 72 of 74. The text had invited confusion between the two streams, and we thank the reviewer for highlighting this. In the revision, every model-based result, including the trajectories, is computed on the choice-stream fit that wins the model comparison; new Supplement B.5 specifies both streams.

Repetition term: we tested two extensions of the gate on the same data and with the same fitting procedure.

GateΔBIC vs surprise-only (positive = surprise-only better)ΔLME
+ consecutive-contradiction term+362−212
Surprise averaged over 3 trials+164−60

Neither improves model evidence, and the fitted repetition weight is, if anything, slightly negative (median −0.09; positive in 18 of 74). The one-trial surprise link is therefore sufficient: the drop in persistence after a second contradiction arises because surprise is lower once the belief has already moved, not because evidence is integrated across trials. Your interpretation of the model was right, and our wording overreached. We have replaced "released after contradictory evidence accumulated" throughout with "persistence fell after a second contradiction, as surprise declined once the belief had moved".

We also tested whether surprise dependence predicts unseen choices, not only fits better in-sample. In leave-one-block-out cross-validation, constant forgetting predicted held-out choices far better than the Standard HGF (+621; 60 of 74 participants), and surprise dependence added a further gain: +33 at the slope used throughout (bᵥ = −3; 43 of 74; p = .067) and +48 at the best-fitting slope (bᵥ = −6; 45 of 74; p = .009) (Supplement B.4).

Q4. "Was the BOCD threshold per-participant or fixed at 9.64 nats? Could you report a capacity-matched comparison?"

The threshold was a free parameter for each participant (prior N(10, 16)). The 9.64 nats was the cohort mean in the submitted fits; in the refitted comparator, fitted thresholds have mean 8.12 nats (SD 2.55). We now state this explicitly. We refitted the comparator with a shorter or no cooldown and with its reset rate matched to each participant's true number of reversals, and added a known-reversal reset, which resets exactly at every true reversal. No discrete-reset model can time its resets better, so it is an upper bound for this whole class.

Threshold-reset comparatorΔBIC (positive = graded release better)
Cooldown 5 trials+870
No cooldown+705
Matched reset rate, timed by its own surprise detector+1,182
Known-reversal reset, timed exactly at each reversal+390

Perfect timing narrows the gap, but graded release still fits better, so its advantage does not come from detection errors (random-effects PXP 1.00).

We are very grateful for this suggestion, which considerably strengthens this part of the paper.

Changes in the revised PDF:

  • Matched rerun with phase × model interaction (3.1) and all agent parameters (C.7); β corrected to 0.287 (3.1, E.1); history-length test (C.8); revised explanation of what accumulates (3.1).
  • Gate-variant comparison and recovery (C.9); locus claim restated (2.1, 4); repetition and longer-memory gates (C.10, Table S1).
  • "Accumulated contradiction" wording replaced (abstract, 3.3, Discussion).
  • Choice stream stated for C.2; all model-based results on the choice-stream fit; both streams specified (B.5).
  • Threshold stated per participant (B.3); capacity-matched comparator family with random-effects selection (3.3, C.11).
  • Full fitting section with priors, optimiser, LME and BIC, slope selection, per-participant evidence and random-effects selection; Trial-wise over Static ΔLME 81, ΔBIC 114 (B.4).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revisions Added with the constructive feedback we got!

We thank all the reviewers! We uploaded a revision...with details are in the individual replies

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.