acceptodds
Under review as a conference paper at ICLR 2027

Neural Networks Spontaneously Dissociate Visual from Informational Persistence Using Iconic Memory

Abstract

Tied classically to decaying persistence in early visual cortex, is a high-capacity sensory trace that briefly outlasts its stimulus. Yet, fading inside the cortex is not confined to any single region, nor does any reverberant activity that counteracts such fading cease uniformly. The ventral stream sustains retrograde insignia intractably, and the recurrence that re-amplifies this decaying activity exists at nearly every level of the visual hierarchy, rendering behavior hard to isolate: Appended to neural networks where such stage-wise differentiation already exists, would a fading percept let us determine its functional use? Confer benefits beyond what a mere expansion in parameter space warrants? To find out, a proper test should heed all stages equally, letting the network determine its own specialized regions, . Consequently, we place the same lightweight following each stage of quaternate-depth convolutional and attention-based classifiers—governed by a learnable () to index temporal persistence, () to index feedback strength, succeeded by canonical divisive normalization, all initialized identically—and train on image classification tasks. Serendipitously, ResNet, DenseNet, ConvNeXt, ventral stream model CORnet-S, transformer model Swin-Tiny, despite their varied architectures, converge on the same final structure: All concentrate Iconic persistence at the sensory input and at the categorical readout—the two ends we argue double as network's artificial counterparts to dissociating early and late intra-cortex, the two telltale components of . Dependent on natural image structure, this particular emergence finds no expression when non-ecological or phase-scrambled images are used during learning. Furthermore, the putatively V1-correspondent early gain is subsumed when fixed Gabor filters are prepended and, while the late stage jump is preserved, its divergence is amplified with increasing task complexity. Across architectures, yield relative clean gains exceeding 8%, mean corruption gains 18% and greater resilience across adversarial attack budgets—a rarely seen simultaneous improvement that through its robustness profile hints at increased alignment with the visual cortex, a finding we further corroborate with representational-similarity tests.

Primary area:
applications to neuroscience & cognitive science
Keywords:
biologically inspired neural networks, recurrent neural networks, iconic memory, fading memory, visual persistence, neural dynamics, robust computer vision, computational neuroscience

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aGZeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces an ICONICBlock, a gated residual module designed to model iconic memory, at different stages of CNNs and vision transformers. It has two learned parameters: α controls the decay of a memory trace over four artificial internal steps, while γ controls how strongly the aggregated memory signal is added back to the stage activation. The authors find that α generally increases with network depth, while γ is stronger at early and late stages than at intermediate stages. They interpret this as the spontaneous emergence of visual persistence in early processing and informational persistence in later stages. Phase scrambling and a V1-like Gabor front end reduce the early-stage enhancement, while increasing classification complexity strengthens the late-stage effect. The ICONICBlocks also improve classification accuracy and robustness. The findings are interesting, but a simpler interpretation is that different stages benefit differently from additional gated nonlinear processing, depending on image statistics and classification demands. The experiments do not establish that these computations represent temporal persistence or iconic memory.

Critical strengths
  1. The depth-dependent α and γ profiles are observed across several CNN and transformer architectures, although their magnitudes vary substantially.
  2. The phase-scrambling, coarse/fine classification, and Gabor front-end experiments suggest that early and late stages respond to different computational demands.
  3. The additional ICONICBlocks improve classification accuracy, corruption robustness, and adversarial robustness.
Critical weaknesses
  1. The connection to iconic memory is not convincing. The ICONICBlock appears to be a complicated gated residual function operating within a single feedforward pass, without actual temporal memory. Although the α and γ profiles are interesting, they do not demonstrate visual or informational persistence. The biological interpretation is therefore overstated: the experiments reveal differences in how much stages benefit from additional processing, rather than differences in how long sensory or categorical information remains available.
  2. The computational benefits of the proposed mechanism are not adequately controlled. ICONICBlocks introduce additional learned projections and nonlinear operations, but the authors compare them with the original backbones without matching parameter count or computational cost. It is unclear whether the improvements come from the proposed decay mechanism or simply from increased representational capacity. A parameter-matched gated residual baseline without artificial decay would be an important control.
Questions for authors

What evidence shows that the learned α and γ reflect temporal persistence rather than simply the optimization of stage-dependent nonlinear gating?

3 comments
AuthorsOpenReview ↗

[1/2] Rebuttal by Authors

We thank the reviewer for their attentive remarks and questions. The possibility that the profiles we report might be nothing more than an artifact of generic gating is the most natural one to raise and, admittedly, one we should have already dispelled. Therefore, we now provide two new tests (n=3n=3) that inspect the iconic claims in this light. The first keeps the gate and takes away only its fading, and the U-shape no longer forms (Table C1), attesting that the gate alone cannot be what draws the U. The second removes the stimulus and sees each stage's memory last only as long as its α\alpha (Table C3). We also showcase, through the parameter-matched controls the reviewer requested, how IconicNet\textbf{IconicNet} remains unsurpassed, and how a parameter-matched ResNet-18 that has no iconic parameters performs the worst out of all controls (Table C1), reconfirming once again the abstract's claim that what drives these gains is not a mere expansion in parameters but instead how that expansion is put to work.


Temporal memory within a single forward pass

Standard deep neural networks (DNN), as we mention in the paper, altogether lack a temporal axis, so any temporal response we wished to elicit had to be configured, as best as possible, along a spatial one, which is to say within a single forward pass itself. Indeed, our motive from the start has been that, while we cannot implement a clockwork of milliseconds into the layers of a DNN, we can very well implement what happens in that space as time passes—this is the gate. Each block hence writes its stage's activation once into a trace that fades over four internal steps, and the gate reads from what remains at each of those steps. A gated structure, we argue, while inhabits no time itself, reproduces as closely as possible what the passing of time entails.

Nevertheless, the concept of fading is not exclusive to iconic memory—all mnemonic activity in cortex fades, the short-term kind extremely rapidly so. If all memory fades, then, how can we even gauge iconic memory? What makes it proper is that it fades, is short-term, but is only accessed, and immediately, as an outside passive buffer. A trace that only faded would be no different from any other memory. It becomes iconic when something reads from it without any rewrite, which renders the implementation of the gate as a sort of access a must, which is also beneficial to separate persistence from a magnitudinal correspondence lest it collides with γ\gamma. How long that buffer then remains available at each stage is what α\alpha sets, and what the delay test of our following reply measures directly (Table C3).

The input is indeed a single static image, and we will make sure to state this and what it entails explicitly in the revised abstract, also adding as future work how the network would behave if it were not a static image but instead a video feed, where a temporal axis already exists and need not be manufactured.


A parameter-matched gated baseline without decay

To isolate decay from the rest of the IconicBlock\textbf{IconicBlock}, we preserved every component as is but reduced the memory steps to one (single gate). The control retains the IconicBlock\textbf{IconicBlock}'s parameters, projections, gate and normalization, and is hence the parameter-matched gated residual baseline without any decay that the reviewer has requested.

Table C1.\hspace{18em}\textbf{Table C1.}

ModelAdded paramsΔ cleanΔ mCAγ\gamma S1 / S2 / S3 / S4γ\gamma profile
Parameter-matched ResNet-181,399,246+0.72+0.44——
Single gate (one read)1,395,528+1.33+1.930.42 / 0.41 / 0.66 / 0.98no U
IconicBlock\textbf{IconicBlock}1,395,528+2.74+4.130.64 / 0.38 / 0.36 / 0.97U-shape

With equal parameters, the single gate variant reaches neither the gain nor the telltale U-shape of the IconicNet\textbf{IconicNet} (Table C1), so the improvements do not come simply from increased representational capacity. Losing its temporal accumulation, the single gate block has the early γ\gamma tail (S1S_1) disappear, with the terminal one (S4S_4) still intact. Not a surprising result since, at S4S_4, the trace barely fades and the decayed reads are nearly identical to the first, so that keeping only the initial trace changes the end stage very little, which has so far been affected chiefly by task demand, as we demonstrated in the paper—attesting, putatively, to its informational function.

Moreover, that the parameter-matched ResNet-18 gains the least (Table C1) is expected for so meager a change in size. The paper had posited as much, that a fading percept confers benefits beyond what a mere expansion in parameter space warrants, and we can now confirm once again that it does.

AuthorsOpenReview ↗

[2/2] Rebuttal by Authors

Temporal persistence

The persistent U-shape of γ\gamma could have two origins: The stage each block occupies, or the kind of memory that it is assigned to. To tell them apart, we replaced the IconicBlock\textbf{IconicBlock}'s passively decaying buffer with a recurrent cell without any other change. The shape did not survive—γ\gamma no longer forms a U and rises monotonically with depth instead, surprisingly recovering the cortical gradient of recurrent excitation proposed by Chaudhuri et al. (2015) [1] (Table C2).

Since the same gate, in the same places, yields two different profiles for two distinct types of memories, we posit more confidently now that γ\gamma follows the memory it is assigned more than the stage that it occupies. Each memory, moreover, yields an intrinsic profile—a cell that rewrites its state at every step yields the biological gradient known for recurrence, whereas the U-shape appears only with a buffer that fades passively and is accessed, which is how the icon functions.

Table C2.\hspace{18em}\textbf{Table C2.}

ModelAdded paramsΔ cleanΔ mCAγ\gamma S1 / S2 / S3 / S4γ\gamma profile
Recurrent cell1,395,524+1.38+2.440.52 / 0.57 / 0.87 / 0.99monotonically increasing
IconicBlock\textbf{IconicBlock}1,395,528+2.74+4.130.64 / 0.38 / 0.36 / 0.97U-shape

Since, biologically, the two persistences are differentiated by delay-readouts—where the visual persistence usually spans a shorter lifetime than its informational, categorical counterpart—we wanted to construct an analogous test on α\alpha, removing the image after the first step, letting each stage rely on the fading block alone and measuring finally how much of the memory's contribution remained one or three steps later (Table C3).

Table C3.\hspace{6em}\textbf{Table C3.}

DelayS1S2S3S4
one step later0.060.090.160.94
three steps later0.000.000.000.90

From the moment the iconic image is removed, the early stages lose increasingly, and in the end almost all, demand put to fading memory, while the final stage (S4S_4) still preserves its own demand intact even three steps later. Each stage's memory thus outlasts the stimulus as its own learned retention allows, making α\alpha denote the memory lifespan following stimulus offset—the very difference in how long information remains available that the reviewer found missing. Regardless even of any labels that denote which end is which, this is an important result demonstrating that, inside the two ends of the network, opposite delay-dependent predictions do emerge.


Inspecting memory content

Visual content, being generic, should remain largely invariant to the task at hand, whereas informational content, being specific to the task itself, should vary with it. To discern more clearly which kind of content a stage holds, we took the two IconicBlock\textbf{IconicBlock} networks of Fig. 5, trained on the same images with either the fine (200) or the coarse (20) labels, and measured with a linear probe how well the fine categories can be read off from each stage.

Table C4.\hspace{6em}\textbf{Table C4.}

Training labelsS1S2S3S4
fine (200)13.423.638.156.5
coarse (20)13.622.231.540.1

Both the fine and the coarse networks score nearly the same at S1S_1 and S2S_2 and differ most at S4S_4 (Table C4). The early stages remain invariant to the task, while the last varies with it the most. We find it striking that the two ends, told apart thus far in the paper by their learned parameters alone, also differ, when measured directly, in both how long their memory remains and in what that memory retains.

We must also note that some of the ambiguity in our biological wording is owed to the subject itself. Since neuroscience itself has yet to settle what constitutes iconic memory, let alone what its counterpart in a neural network should look like, what we can speak to with more confidence is its temporality and thereby the corresponding α\alpha. Nevertheless, our claim still is not that a single evidence exists that can pinpoint an icon but, instead, that several results evince together its emergence.

The U-shape is lost when the fading is removed, the profile changes with the kind of memory behind the gate, and each stage's memory, after the iconic image is removed, lasts for as long as its α\alpha. That each new emerging piece of evidence keeps agreeing with the last is, we argue, more than one can ask of a coincidence. We owe both the gated baseline and the delay test to the reviewer's constructive objections, for it is through them that we can now more confidently say, what unfolds within a single forward pass, given an icon, behaves as a memory in time.


[1] Chaudhuri et al. (2015). A large-scale circuit mechanism for hierarchical dynamical processing in the primate cortex.

We hope you have found our rebuttal and the revised paper, where all changes are marked in red, meaningful. The gated baseline, along with the delay readout your questions led us to, now resides in the manuscript---results suggesting strongly that the iconic profile is more than a coincidence of gating. We hope you will revisit your assessment in light of these results and would be glad to answer any remaining questions before the discussion period ends. Thank you.

Reviewer_3YctOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how classification training allocates short-lived feature retention across image classifiers. The input is a single static image and the output is a category prediction. The main stage-profile experiments use the public 200-class Tiny ImageNet dataset; ImageNet-100 and its corruption benchmark provide additional performance evaluation. Brain-representation comparisons reuse existing Algonauts 2019 fMRI/MEG data for 92 image stimuli; neural signals are not classification inputs. An IconicBlock follows each backbone stage, writes a trace once, decays it over four internal steps, gates access to fixed content, and returns the normalized averaged readout through a learned gain. Retention and re-entry are separately parameterized by α and γ and optimized for classification. The contribution is a study of how uniformly initialized blocks differentiate across stages. Table 1 reports similar γ profiles across five backbones, with lower intermediate values and a terminal rebound. Input structure, label granularity, and the visual front end alter these profiles, motivating an interpretation in terms of visible and informational persistence.

Critical strengths
  1. The cross-architecture stage-allocation pattern is a potentially useful computational finding. Table 1 reports three-seed means for ResNet-18, ConvNeXt-Tiny, CORnet-S, DenseNet-121, and Swin-T, with similar mean γ ordering. The initialization rules in Appendix A do not prescribe that ordering. This supports differentiated block use across depths under the tested conditions, independently of the cortical interpretation. The stage-placement study in Appendix E's Fig.9 already covers all 15 nonempty subsets, with the four-stage configuration giving the highest reported mean clean and corruption accuracy.

  2. The interventions meaningfully constrain the explanation, and the parameterization has a concrete rationale. In Fig.5, changing 200 fine classes into 20 coarse classes reduces the terminal α increase and γ rebound while preserving the Tiny ImageNet images and architecture. This controls an important source of input variation. Phase scrambling in Fig.4 and the VOne front end in Fig.6 provide complementary interventions. Appendix B analyzes direct scaling confounds between retention and re-entry, while Appendices B/C motivate the gated readout and normalization. The reported classification and corruption improvements further support the module's practical potential.

Critical weaknesses
  1. The paper has not separately established model readouts for visible and informational persistence; the principal measurements remain α/γ and final classification performance. Coltheart (1980) includes visual attributes in informational persistence, while Loftus and Irwin (1998) compare persistence through distinct task readouts. Stage peaks and category predictions do not directly establish that correspondence. Section 4 interprets the interventions as a double dissociation, but cross-effects at both ends in Figs.4,5 require explicit functional measures and relative-selectivity comparisons. Nonzero cross-effects do not themselves invalidate a dissociation. The parameter plots and Appendix F's descriptive brain-representation comparison currently support a motivated psychological hypothesis.

  2. The comparisons do not isolate the distinctive role of learned decay from more general gated enhancement. Appendix D removes the entire block, Fig.9 tests placement, and Appendices B/C explain the parameter design; each is informative without separating decay from projection, gating, normalization, and residual enhancement. The cross-backbone and intervention findings remain novel relative to the learned layerwise decay of Nayebi et al. (2018) and depth-dependent gating of Hu et al. (2018). A close comparison or discriminating existing analysis would clarify whether the new knowledge depends specifically on this decay design. There is no evidence here that an ordinary static gate necessarily reproduces the profiles.

  3. The Introduction clearly identifies the allocation question, but the evidential hierarchy among the central pattern, practical benefits, and brain-similarity observations needs clearer organization. The Abstract links parameter differentiation, robustness, and cortical alignment, leaving readers to distinguish direct support for stage allocation from auxiliary associations. The methods distinguish retention duration α from re-entry strength γ, whereas Section 4 moves relatively quickly from γ magnitude to persistence functions and cortical interpretation. The descriptive RSA limitation and the Discussion's treatment of audio as future work are useful boundaries. Connecting them more explicitly to the main argument would make the conclusions and their support easier to assess.

  4. The interventions yield useful observations without yet establishing a general account with clear applicability conditions. Section 4 groups label granularity, class count, and images per class under task complexity, although these probe discrimination requirements and data availability differently; the natural-image interpretation also spans distinct data conditions. Even if all reported results hold, the information gained is primarily that stage allocation varies with these training conditions. It remains unclear which changes reveal general retention demands and which depend on particular tasks or data regimes. Synthesizing their shared constraints, boundaries, and testable implications would strengthen the explanatory knowledge conveyed by these results.

My current recommendation is 2/4, weak rejection. The cross-backbone pattern and same-image intervention are substantive strengths, but the functional correspondence and the additional explanatory value of the decay design remain insufficiently distinguished. Evidence or clarification resolving these bounded concerns could change my assessment.

Questions for authors
  1. Which model observables separately operationalize visible and informational persistence? Can existing models or analyses yield different delay-dependent or intervention-dependent predictions for those readouts? Distinct functional evidence would strengthen the psychological correspondence; if the present results concern stage allocation only, please clarify that contribution's independent scope.

  2. What existing evidence separates learned decay from the other block components in explaining the γ profile and fine/coarse-label response? A fixed-decay or simplified-gate comparison preserving projection, normalization, re-entry, and training settings is one potentially informative approach. The answer would clarify whether the additional knowledge concerns decay-specific allocation or more general stage-dependent gating, and how performance, brain-representation, and audio results support that central contribution.

  3. In what measurement and comparative sense do Figs.4,5,7 establish the double dissociation claimed in Section 4? Please specify the relative selectivity and its support across seeds. Evidence distinguishing the two functions' intervention responses would strengthen the interpretation; support limited to parameter ordering or effect magnitude would warrant correspondingly narrower claims.

These questions assess the submitted contribution without making new human/animal studies, precise millisecond calibration, or leaderboard-leading results acceptance requirements.

6 comments
AuthorsOpenReview ↗

[1/3] Rebuttal by Authors

We thank the reviewer for their careful reading and pertinent questions that moved the paper forward greatly each with their answers, which we dwell on below.


Persistence readouts

We must admit that demonstrating a psychophysical behavior—one that is also closely tied to the temporal economics of the task at hand, a temporality altogether absent from standard feedforward deep neural networks—without considerable approximations taken in both implementation and interpretation, is considerably difficult. Indeed, there still is not one single result that can be purported as, we have found iconic memory, but instead there exists a plethora of emerging new evidence that we posit attests to its titular incipience:

Replacing the passive decay of iconic memory with recurrent excitation, while changing nothing else, makes the hierarchy increasingly monotonic and becomes instead a direct and striking prediction of Chaudhuri et al. (2015) [1]; cutting the fading memory off after the first step (as a putative analogue to psychobehavioral tests of stimulus masking) likewise dismantles the shape. The early part of the shape is more prone to these two changes, attesting to its need for visual persistence, while the late peak is instead mainly affected by task demand, attesting more to its reliance on late or, as we denote throughout the paper, informational content—where the accumulated knowledge needs to be sustained much longer than its earlier, visual counterpart.

Nevertheless, none of these results have so far set out to delineate how the two persistences should even be told apart, aside from any stage-wise emergence given a putative correspondence, nor did they corroborate a psychophysical account in so direct a test—so, we with these rebuttals especially set out to conduct experiments, much closer now in spirit to such accounts. These new tests measure, at every stage, the two properties by which the persistences are distinguished within the iconic nomenclature, namely, how long a memory survives once the stimulus is gone and what that memory contains. For what a memory contains, we follow the dual-buffer account of Mewhort et al. (1981) [2], and for the use of a distinct readout for each, Loftus and Irwin (1998) [3].


Memory duration

Since in psychophysics, the two persistences are often differentiated by delay-readouts, where visual persistence shows a shorter lifespan than its informational and late-comer counterpart, we wanted to try a similar test, removing the iconic image after the first step, letting each stage rely on the fading block alone and measuring how much of the memory's contribution remained one versus three steps later (Table B1, n=3n = 3).

Table B1.\hspace{6em}\textbf{Table B1.}

DelayS1S2S3S4
one step later0.060.090.160.94
three steps later0.000.000.000.90

From the first moment the iconic image is removed, the early stages lose increasingly, and in the end almost all, of the fading memory's contribution, while the final stage (S4S_4) still demands its own even three steps later. Regardless of any labels that denote which end is which persistence, this is an important result demonstrating that, inside the two ends of the network, opposite delay-dependent predictions do exist.


[1] Chaudhuri et al. (2015). A large-scale circuit mechanism for hierarchical dynamical processing in the primate cortex.

[2] Mewhort et al. (1981). Identification, localization, and "iconic memory": An evaluation of the bar-probe task.

[3] Loftus and Irwin (1998). On the relations among different measures of visible and informational persistence.

[4] Hu et al. (2018). Squeeze-and-excitation networks.

[5] Nayebi et al. (2018). Task-driven convolutional recurrent models of the visual system.

AuthorsOpenReview ↗

[2/3] Rebuttal by Authors

Memory content

Visual content, being generic, should remain largely invariant to the task at hand, whereas informational content, being specific to the task itself, should be varying with it. Therefore, to discern more clearly which kind of content a stage holds, we took the two IconicNet\textbf{IconicNet} models of Fig. 5, already trained on the same images with either the fine (200) or the coarse (20) labels, and now measured, using a linear probe, how well the fine categories can be read off from each stage. Under the assumption that, if either the fine or coarse network sustains more informational content, the differences between their respective stages would be the telltale sign.

Table B2.\hspace{6em}\textbf{Table B2.}

Training labelsS1S2S3S4
fine (200)13.423.638.156.5
coarse (20)13.622.231.540.1

Both the fine and the coarse networks score close at S1S_1 and S2S_2 but differ most at S4S_4 (Table B2). The early stages remain invariant to task, as generic visual content should, while the final stage varies with it the most, as informational and late-arriving content should. As Fig. 5 of the paper already shows, and Table B3 gives in numbers, α\alpha and γ\gamma drop sharply at S4S_4 under the coarse labels, while at S1S_1, α\alpha stays where it was, and γ\gamma falls far less. We should emphasize that the Tables B2 and B3 together show what either table alone could not. When the task stops asking for the fine categories, S4S_4 holds fewer of them and, at the same time, sustains its memory for a shorter time and lets it re-enter more weakly, whereas S1S_1 is left almost as it was. The early memory is thus brief and generic and the late memory long-lasting and category-specific. We find it striking that the two ends, told apart in the paper thus far only by their learned α\alpha and γ\gamma, also differ in how long their memory remains and in what that memory retains.

Table B3.\hspace{12em}\textbf{Table B3.}

Labels (same images)α\alpha S1 / S2 / S3 / S4γ\gamma S1 / S2 / S3 / S4
fine (Table 1)0.08 / 0.11 / 0.22 / 0.960.64 / 0.38 / 0.36 / 0.97
coarse0.09 / 0.15 / 0.14 / 0.140.52 / 0.39 / 0.26 / 0.46
AuthorsOpenReview ↗

[3/3] Rebuttal by Authors

Learned decay

The two controls of Table B4 are a fixed decay and a simplified gate, both of which retain the IconicBlock\textbf{IconicBlock}'s parameters, projection, normalization, re-entry and other training settings. Losing its temporal accumulation, as a putative analogue to psychophysical masking, the single gate block has the early γ\gamma tail (S1S_1) disappear, with the terminal one (S4S_4) still intact, and loses the U-shape altogether. The fixed-retention variant, on the other hand, has surprised us by also losing the S1S_1 peak and following a completely different shape, which we interpreted as arising from a need for continually learned and synergistic allocation of demand and persistence.

Table B4.\hspace{18em}\textbf{Table B4.}

ModelAdded paramsΔ cleanΔ mCAγ\gamma S1 / S2 / S3 / S4γ\gamma profile
Single gate (one read)1,395,528+1.33+1.930.42 / 0.41 / 0.66 / 0.98no U
Fixed retention (α\alpha frozen)1,395,528+0.91+2.190.40 / 0.42 / 0.56 / 0.95monotonically increasing
IconicBlock\textbf{IconicBlock}1,395,528+2.74+4.130.64 / 0.38 / 0.36 / 0.97U-shape

Replacing the passive decay pertinent to iconic memory with a recurrent cell, while changing nothing else, makes γ\gamma rise monotonically with depth instead (Table B5). Aside from any denotation of persistence, the gauge of memory demand too holds, and all the more strongly now, since it has also been able to measure the recurrent memory demand predicted by Chaudhuri et al. (2015) [1].

Table B5.\hspace{18em}\textbf{Table B5.}

ModelAdded paramsΔ cleanΔ mCAγ\gamma S1 / S2 / S3 / S4γ\gamma profile
Recurrent cell1,395,524+1.38+2.440.52 / 0.57 / 0.87 / 0.99monotonically increasing
IconicBlock\textbf{IconicBlock}1,395,528+2.74+4.130.64 / 0.38 / 0.36 / 0.97U-shape

The single gate block, in our setting, is a depth-dependent static gate in the manner of Hu et al. (2018) [4], while our recurrent cell, which rewrites its state at every step, is closer to the models of Nayebi et al. (2018) [5]. Since neither forms any sort of U-shape, we are now more confident that this new knowledge does specifically depend on our decay design—a trace that is written once, never transformed and left to fade. The fine/coarse-label response, on the other hand, concerns the late peak—the peak which every control preserved and which has thus far been affected chiefly by task demand, attesting more to its categorical content than to its visual. The performance gains also drop whenever the decay is altered (Tables B4, B5).


Double dissociation

The two readouts now give the dissociation a functional measure (Tables B1, B2). Of the two interventions, the change of labels is the selective one (Table B3; Fig. 5), whereas phase scrambling dissolves the early ordering (Fig. 4) with effects not confined to early stages. The class-count sweep of Fig. 7 raises γ\gamma at every stage as classes accumulate, so it shows how the demand for memory grows with the task, which the revision will make sure to set apart from the dissociation.


Evidential hierarchy

We will reorganize the abstract and Section 4 so that the stage-wise pattern and its new controls come first, the interventions that alter it second, and the robustness, brain-similarity and audio results last, as supporting observations, each with the boundary the paper already attributes to (a descriptive RSA, audio left as future work). The two readouts will be placed between the γ\gamma profile and its persistence interpretation, where, as the reviewer notes, Section 4 moved relatively quickly.


Applicability conditions

Taken together, our results suggest that the U-shape emerges under three conditions: Natural image structure (Fig. 4), a task demanding fine categories (Fig. 5) and a decay learnable (Tables B4, B5). The decay and natural image structure are what the early end requires, whereas the task complexity drives the late end. We will state these conditions more clearly in camera-ready and present label granularity, class count and images per class as separate probes instead of a single gradient of task complexity, since they probe discrimination requirements and data availability differently. Moreover, as the reviewer has aptly put, the natural-image condition spans two different confounds, a change of dataset (MNIST, SVHN) and the removal of structure from the same images (phase scrambling). Our emphasis will consist of the latter, on which the natural-image claim should rest, since structural difference is empirically more apparent there.

What began as a profile of two parameters now rests on readouts, controls and conditions, all attesting to one initial claim—that, given this tool, the network can tell visual from informational persistence apart.

Reviewer_ugGROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks how briefly appeared visual information is retained across a processing hierarchy and whether distinct forms of persistence can emerge through task-driven learning. The paper adds fading trace blocks to successive stages of image classifiers, with learnable retention and re-entry gains. Across several backbones, retention generally increases with depth, while re-entry gain is higher at early and final stages. The authors interpret these patterns as counterparts of visible and informational persistence. They also report improvements in clean classification and corruption robustness.

Critical strengths
  1. Broad architectural evaluation. The method shows reported improvements across convolutional, transformer, and recurrent backbones, suggesting applicability beyond a single architecture.

  2. Useful experimental controls. Image scrambling, label granularity, visual front ends, and stage-placement ablations help probe how the learned profiles depend on input structure and task demands.

Critical weaknesses

The mechanism’s contribution is not isolated. Comparisons with the unmodified backbone do not separate the benefits of decay from those of added capacity, gating, and normalization.

The biological interpretation remains to be validated. The interventions and exploratory neural alignment provide useful evidence, but do not specifically establish visible versus informational persistence.

Questions for authors
  1. Could a trainable recurrent module achieve similar gains and stage profiles under comparable parameter and computational budgets?

  2. With the other components preserved, do single-gate or fixed-retention controls reproduce the accuracy gains and re-entry gain profiles?

  3. What testable prediction distinguishes the proposed persistence interpretation from general differences between early feature processing and late classification?

4 comments
AuthorsOpenReview ↗

[1/3] Rebuttal by Authors

We thank the reviewer for their insightful remarks and questions. Some of them proved especially useful, and we think their answers have bolstered the paper's main claims considerably further.


Recurrence vs. passive decay

To differentiate between iconic memory and recurrent excitation, we replaced IconicBlock\textbf{IconicBlock}'s passively decaying buffer with a recurrent cell, changing nothing else. The cell rewrites its state at every step through a learned gate and, since it does not sustain a copy in time, renders α\alpha redundant, due to which we only report the network's γ\gamma.

With equal parameters, step-size and re-entry (γ\gamma), the cell reaches neither the gain nor the telltale U-shape of IconicNet\textbf{IconicNet} (Table A1, n=3n = 3); instead, its γ\gamma rises monotonically with depth. An opportune insight for us, since experiments thus far have only been able to shed light onto what drives the late stage jump—which turns out is unaffected by this recurrent change—and only partly the early tail that we argue should reflect visual persistence.

We additionally found this to be a new strong correspondence because, trained only on the classification task, the re-entry gain γ\gamma becomes a marker of demand again and, this time, the demand for recurrence. IconicBlock\textbf{IconicBlock}, when the buffer supplied intra-stage was passive, had already demonstrated two types of competing demand via re-entry. Now, given instead a recurrent cell that actively transforms the input, γ\gamma follows a monotonic increase and recovers, quite strikingly, the cortical gradient of recurrent excitation proposed by Chaudhuri et al. (2015) [1]. These results, taken together, help delineate what iconic memory warrants from what recurrent memory, or any other type of memory, warrants. Ultimately, we have now further evidence for the paper's initial position that γ\gamma measures memory demand and thank the reviewer for this especially revelatory question.

Table A1.\hspace{18em}\textbf{Table A1.}

ModelAdded paramsΔ cleanΔ mCAγ\gamma S1 / S2 / S3 / S4γ\gamma profile
Recurrent cell1,395,524+1.38+2.440.52 / 0.57 / 0.87 / 0.99monotonically increasing
IconicBlock\textbf{IconicBlock}1,395,528+2.74+4.130.64 / 0.38 / 0.36 / 0.97U-shape

As Table A1 shows, recurrence also costs robustness gains, possibly because the recurrent cell rebuilds its state from corrupted features at every step, propagating the corruption, whereas the IconicBlock\textbf{IconicBlock} gates can choose to sustain the needed features from the initial snapshot—a sort of, "you can pick what you want but from only the original copy, not your interpretation of that copy in a subsequent step," approach. We must also note that, since nothing else was changed other than the intra-stage effect rendered now recurrent, this variant inherits exactly the hyperparameters of the IconicBlock\textbf{IconicBlock}, so its gains are best read as a floor and not a ceiling.


[1] Chaudhuri et al. (2015). A large-scale circuit mechanism for hierarchical dynamical processing in the primate cortex.

AuthorsOpenReview ↗

[2/3] Rebuttal by Authors

Single-gate and fixed-retention controls

To isolate decay from the rest of the IconicBlock\textbf{IconicBlock}, we preserved every component as is but reduced the memory steps to one (single gate). To isolate the decay's learned rate instead, we froze α\alpha at its initial value while letting decay stay untouched (fixed retention). Both controls keep the IconicBlock\textbf{IconicBlock}’s parameters, gate and normalization, so, touching on the reviewer's first concern, they separate the decay proper from added capacity, gating or any other intervention.

ModelAdded paramsGateNormalizationDecayLearned decay rate
IconicBlock\textbf{IconicBlock}1,395,528++++
Single gate (one read)1,395,528++−−
Fixed retention (α\alpha frozen)1,395,528+++−

Losing its temporal accumulation, the single gate block has the early γ\gamma tail (S1S_1) disappear, with the terminal one (S4S_4) staying intact and, stage-wise, loses the U-shape altogether. Not a surprising result since, at S4S_4, the trace barely fades, and the decayed reads remain nearly identical to the first read. Thus, keeping only the initial trace changes the end stage very little, which has so far been affected mainly by task complexity—and that putatively attests to its informational function.

The fixed-retention variant, on the other hand, has surprised us. Because the learned α\alpha at S1S_1 and S2S_2 already settle near their initial value in normal training, we expected no change in these stages to occur when we merely fixed α\alpha to init from the beginning ourselves. Nevertheless, the fixed α\alpha variant loses both the S1S_1 peak and the U-shape, and, as a whole, follows a monotonic increase. Since this was an unexpected result, we wanted to further investigate—maybe, because we fixed the α\alpha of all stages at the same time, S4S_4's was making the whole shape collapse here?

To discern this possible stage-wise confound, we made two additional variants: First, where the α\alpha of S1S_1, S2S_2 and S3S_3 were fixed and that of S4S_4 learned and the second, vice versa. Alas, even in these configurations, the U-shape was dismantled! A parsimonious explanation we want to offer is, because α\alpha and γ\gamma co-adapt during training, the U is possibly an allocation the network has to arrive at by itself, emerging only when every stage is free to set its own persistence—a reading we think is further corroborated by the single-gate version, where the γ\gamma tail is subsumed once α\alpha drops out of the computation—a theory that warrants further experimentation as it stands.

Table A2.\hspace{22em}\textbf{Table A2.}

ModelAdded paramsΔ cleanΔ mCAγ\gamma S1 / S2 / S3 / S4γ\gamma profile
Parameter-matched ResNet-181,399,246+0.72+0.44——
Single gate (one read)1,395,528+1.33+1.930.42 / 0.41 / 0.66 / 0.98collapsed U
Fixed retention (α\alpha frozen)1,395,528+0.91+2.190.40 / 0.42 / 0.56 / 0.95monotonically increasing
IconicBlock\textbf{IconicBlock}1,395,528+2.74+4.130.64 / 0.38 / 0.36 / 0.97U-shape

In accuracy, too, both controls fall short of IconicNet\textbf{IconicNet}, and a parameter-matched ResNet-18, with the block discarded altogether, performs the worst of all (Table A2), which is not surprising for a change in size this meager. The paper had posited that a fading percept confers benefits beyond what a mere expansion in parameter space warrants, and we can now confirm once more that it indeed does.

AuthorsOpenReview ↗

[3/3] Rebuttal by Authors

Early vs. late persistence after stimulus offset

The prediction that sets the persistence interpretation apart from any possibly generic differences between early feature processing or late classification is a temporal one; namely, that the two ends should sustain their proffered memory for different durations after stimulus offset. We have accordingly conducted new tests with a different measurement, closer now in vein to psychophysical experiments and that largely corroborates their account.

Since, biologically, the two persistences are differentiated by delay-readouts—where the visual persistence spans a shorter lifetime than its informational, late-comer counterpart—the analogical test we employed is removing the image after the first step and letting each stage rely on the fading block alone, and then to measure how much of the memory's contribution remains one or three steps later (Table A3).

Table A3.\hspace{6em}\textbf{Table A3.}

DelayS1S2S3S4
one step later0.060.090.160.94
three steps later0.000.000.000.90

From the first moment the iconic image is removed, the early stages lose increasingly, and in the end almost all, of the fading memory's contribution, while the final stage (S4S_4) still preserves its own nearly intact even three steps later. Regardless of any labels to denote which end is which, this is an important result demonstrating that, inside the two ends of the network, opposite delay-dependent predictions emerge.

We would also be remiss not to mention that the ambiguity in some biological justifications has arisen less from any choice of ours than from the fact that neuroscience itself is yet to reach a consensus on what constitutes iconic memory in the first place. Indeed, while researchers themselves are still debating via what mechanisms or from what reaches of the cortex this peculiar memory arises, we are here trying to demonstrate its emergence, or a semblance of its emergence, and in a neural network at that. While this is a challenging task, and any sort of evidence even for its incipience entails its sundry share of speculative interpretations, we should emphasize that our main argument has never been that, iconic memory is a single mechanism that can emerge and can be pinpointed to, but, instead, that a plethora of evidence exists which, altogether, can point to that emergence. The recurrent cell, the single gate and the fixed retention each took away the early tail and left the late peak in place, and the delay test then found the same two ends behaving differently in time.

That each new emerging piece of evidence keeps agreeing with the last is, we argue, more than one can ask of a coincidence, and a confirmation that something common and special does emerge across all these varied networks. In some putative biological wording, we had named that something a tug-of-war between the two types of persistence, each holding its own end and each keeping its own kind of content—and these rebuttals have now shed more light on just how different the two truly are. We owe many of these new insights to the reviewer's well-placed questions and hope that it all reads as well to them as it did to us.

AuthorsOpenReview ↗

Author Comment

We hope you have had a chance to go through our rebuttal and the revised paper, where every change is now marked in red. We would like to elaborate on our answer to your question regarding a trainable recurrent module, since it proved one of the most revealing of the discussion. Given a recurrent cell in place of the passive buffer and nothing else altered, γ\gamma rises monotonically with depth and recovers the cortical gradient of recurrent excitation proposed by Chaudhuri et al. (2015), so that the same gauge, given a different memory, successfully measures a different sort of demand---exactly as it should.

It is through your questions that now we can confidently delineate what iconic memory warrants from what recurrent memory warrants, and we are grateful for this apt remark. Alas, we would still be happy to clarify any points may not have been sufficiently addressed before the discussion period ends. Thank you.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their insightful remarks and questions, many of which proved especially useful and, we believe, helped strengthen the paper's main claims in a different light. In summary, each requested control now further helps distinguish the two ends of the U-shaped curve, as do two readouts that probe the memory’s duration and its content. We have uploaded a revised version including all the requested experiments, with every change marked in red\color{red}{\text{red}}.

It is also worth clarifying some of the decisions we made designing our experiments and touching upon some commonly raised remarks. Since standard deep neural networks lack a time axis altogether, any temporal response we wished to elicit had to be configured, as best as possible, along a spatial one, using a trace that fades within each stage. Yet, since all mnemonic activity in cortex eventually fades, fading alone would be of no help in evincing iconic memory. How, then, can we even gauge it specifically?

No single result or experiment exists that can assuredly point and say, here emerges iconic memory proper, but many of them, taken together, can make a case for it more convincingly. And since each new test we conducted separated the early end of the network from the late one in the same way our titular claim predicts, it is this convergence that, we argue, can most compellingly attest to the emergence of iconic memory.

The recurrent cell, the single gate and the fixed retention each dismantles the early tail of the U and leaves its late peak in place, and with the recurrent cell replacing the passive buffer, γ\gamma surprisingly recovers the cortical gradient of recurrent excitation proposed by Chaudhuri et al. (2015). Furthermore, two new readouts now measure at every stage the two properties by which the persistences have been told apart in iconic nomenclature—how long a memory survives once the stimulus is gone and what that memory retains.

What our paper contributes to the field, beyond the telltale profile itself, is a possible explanation for why iconic memory, and its resultant dissociation, has to arise in the first place. Five architectures, sharing no temporal parameter at initialization, given only a fading trace and a classification objective, all arrive at the same two ends of the same U-shape and draw between them a division, much like the one psychophysics drew between visible and informational persistence, decades since.

In the end, no network was told how long to use its memory for, or whether to use it at all, yet, left with only a transitory trace, they worked both out on their own. So that a form of persistence, long studied as an intrinsic property of the biological visual system, emerges here solely from an image classifier having to recognize complex and natural images in the midst of impermanence.

We thank the reviewer for their response. In the IconicBlock, the residual branch carries nothing but the readout of the fading trace, x^=x+γN(u;x)\hat{x} = x + \gamma \mathcal{N}(u; x) (Eq. 3), where uu is the content admitted at each step by a gate that the fading trace itself drives (Eq. 2). A stage that makes more use of this residual branch therefore makes more use of its memory, which is exactly what we take γ\gamma to measure. Because the readout is normalized to the activation of its own stage before γ\gamma admits it back (Appendix C), γ\gamma measures this use on the same scale at every depth, so that a higher γ\gamma at the two ends cannot arise from their activations simply differing in scale.

What remains is whether the first and last stages favor this branch for some reason other than memory, as a generic preference for residual processing at the ends of a network. If that were so, the U-shape should hold whatever the network is trained on, since the architecture and its residual branches stay the same. Alas, it does not. On MNIST and SVHN the U-shape is absent and γ\gamma declines toward zero at S4S_4, phase scrambling dissolves the early ordering while the late rebound survives, a VOneNet front end takes over the S1S_1 peak, and coarse labels shrink the S4S_4 rebound to a fraction of itself (Figs. 4--6). Each end thus follows what its memory has to hold, the early one the structure of the input and the late one the structure of the required output, while the residual branch itself never changes.

The controls of the revised manuscript lead to the same conclusion. The recurrent cell, the single gate and the fixed-retention variant keep the same residual branch, gain and normalization, and change only what the branch carries, yet none of them draws the U-shape, which appears only when the branch carries a trace that fades at a learned rate:

ModelClean gain (%)mCA gain (%)γ\gamma S1S_1γ\gamma S2S_2γ\gamma S3S_3γ\gamma S4S_4
Param.-matched ResNet-18+0.72+0.44––––
Single gate+1.33+1.930.420.410.660.98
Fixed retention+0.91+2.190.400.420.560.95
Recurrent cell+1.38+2.440.520.570.870.99
IconicBlock+2.74+4.130.640.380.360.97

The two ends, moreover, behave as memories when measured directly. When the image is removed after the first step, the early stages lose almost all of what their memory contributed, while S4S_4 preserves it nearly intact. A linear probe for the fine categories scores close at S1S_1 and S2S_2 whichever labels the network was trained on, and differs considerably at S4S_4:

S1S_1S2S_2S3S_3S4S_4
Memory left, one step later0.060.090.160.94
Memory left, three steps later0.000.000.000.90
Fine-category probe (%), fine labels13.423.638.156.5
Fine-category probe (%), coarse labels13.622.231.540.1

The two ends that use the residual branch most are thus the ones whose memory differs in how long it survives the image and in what it holds. Each of these results separates the two ends in the same way, and it is this convergence, more than any single result, that ties γ\gamma to memory demand. We hope this settles the remaining uncertainty about the mechanism, and we would be grateful if the reviewer took it into account in their final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.