- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces an ICONICBlock, a gated residual module designed to model iconic memory, at different stages of CNNs and vision transformers. It has two learned parameters: α controls the decay of a memory trace over four artificial internal steps, while γ controls how strongly the aggregated memory signal is added back to the stage activation. The authors find that α generally increases with network depth, while γ is stronger at early and late stages than at intermediate stages. They interpret this as the spontaneous emergence of visual persistence in early processing and informational persistence in later stages. Phase scrambling and a V1-like Gabor front end reduce the early-stage enhancement, while increasing classification complexity strengthens the late-stage effect. The ICONICBlocks also improve classification accuracy and robustness. The findings are interesting, but a simpler interpretation is that different stages benefit differently from additional gated nonlinear processing, depending on image statistics and classification demands. The experiments do not establish that these computations represent temporal persistence or iconic memory.
Critical strengths
- The depth-dependent α and γ profiles are observed across several CNN and transformer architectures, although their magnitudes vary substantially.
- The phase-scrambling, coarse/fine classification, and Gabor front-end experiments suggest that early and late stages respond to different computational demands.
- The additional ICONICBlocks improve classification accuracy, corruption robustness, and adversarial robustness.
Critical weaknesses
- The connection to iconic memory is not convincing. The ICONICBlock appears to be a complicated gated residual function operating within a single feedforward pass, without actual temporal memory. Although the α and γ profiles are interesting, they do not demonstrate visual or informational persistence. The biological interpretation is therefore overstated: the experiments reveal differences in how much stages benefit from additional processing, rather than differences in how long sensory or categorical information remains available.
- The computational benefits of the proposed mechanism are not adequately controlled. ICONICBlocks introduce additional learned projections and nonlinear operations, but the authors compare them with the original backbones without matching parameter count or computational cost. It is unclear whether the improvements come from the proposed decay mechanism or simply from increased representational capacity. A parameter-matched gated residual baseline without artificial decay would be an important control.
Questions for authors
What evidence shows that the learned α and γ reflect temporal persistence rather than simply the optimization of stage-dependent nonlinear gating?
3 comments
[2/2] Rebuttal by Authors
Temporal persistence
The persistent U-shape of could have two origins: The stage each block occupies, or the kind of memory that it is assigned to. To tell them apart, we replaced the 's passively decaying buffer with a recurrent cell without any other change. The shape did not survive— no longer forms a U and rises monotonically with depth instead, surprisingly recovering the cortical gradient of recurrent excitation proposed by Chaudhuri et al. (2015) [1] (Table C2).
Since the same gate, in the same places, yields two different profiles for two distinct types of memories, we posit more confidently now that follows the memory it is assigned more than the stage that it occupies. Each memory, moreover, yields an intrinsic profile—a cell that rewrites its state at every step yields the biological gradient known for recurrence, whereas the U-shape appears only with a buffer that fades passively and is accessed, which is how the icon functions.
| Model | Added params | Δ clean | Δ mCA | S1 / S2 / S3 / S4 | profile |
|---|---|---|---|---|---|
| Recurrent cell | 1,395,524 | +1.38 | +2.44 | 0.52 / 0.57 / 0.87 / 0.99 | monotonically increasing |
| 1,395,528 | +2.74 | +4.13 | 0.64 / 0.38 / 0.36 / 0.97 | U-shape |
Since, biologically, the two persistences are differentiated by delay-readouts—where the visual persistence usually spans a shorter lifetime than its informational, categorical counterpart—we wanted to construct an analogous test on , removing the image after the first step, letting each stage rely on the fading block alone and measuring finally how much of the memory's contribution remained one or three steps later (Table C3).
| Delay | S1 | S2 | S3 | S4 |
|---|---|---|---|---|
| one step later | 0.06 | 0.09 | 0.16 | 0.94 |
| three steps later | 0.00 | 0.00 | 0.00 | 0.90 |
From the moment the iconic image is removed, the early stages lose increasingly, and in the end almost all, demand put to fading memory, while the final stage () still preserves its own demand intact even three steps later. Each stage's memory thus outlasts the stimulus as its own learned retention allows, making denote the memory lifespan following stimulus offset—the very difference in how long information remains available that the reviewer found missing. Regardless even of any labels that denote which end is which, this is an important result demonstrating that, inside the two ends of the network, opposite delay-dependent predictions do emerge.
Inspecting memory content
Visual content, being generic, should remain largely invariant to the task at hand, whereas informational content, being specific to the task itself, should vary with it. To discern more clearly which kind of content a stage holds, we took the two networks of Fig. 5, trained on the same images with either the fine (200) or the coarse (20) labels, and measured with a linear probe how well the fine categories can be read off from each stage.
| Training labels | S1 | S2 | S3 | S4 |
|---|---|---|---|---|
| fine (200) | 13.4 | 23.6 | 38.1 | 56.5 |
| coarse (20) | 13.6 | 22.2 | 31.5 | 40.1 |
Both the fine and the coarse networks score nearly the same at and and differ most at (Table C4). The early stages remain invariant to the task, while the last varies with it the most. We find it striking that the two ends, told apart thus far in the paper by their learned parameters alone, also differ, when measured directly, in both how long their memory remains and in what that memory retains.
We must also note that some of the ambiguity in our biological wording is owed to the subject itself. Since neuroscience itself has yet to settle what constitutes iconic memory, let alone what its counterpart in a neural network should look like, what we can speak to with more confidence is its temporality and thereby the corresponding . Nevertheless, our claim still is not that a single evidence exists that can pinpoint an icon but, instead, that several results evince together its emergence.
The U-shape is lost when the fading is removed, the profile changes with the kind of memory behind the gate, and each stage's memory, after the iconic image is removed, lasts for as long as its . That each new emerging piece of evidence keeps agreeing with the last is, we argue, more than one can ask of a coincidence. We owe both the gated baseline and the delay test to the reviewer's constructive objections, for it is through them that we can now more confidently say, what unfolds within a single forward pass, given an icon, behaves as a memory in time.
[1] Chaudhuri et al. (2015). A large-scale circuit mechanism for hierarchical dynamical processing in the primate cortex.
We hope you have found our rebuttal and the revised paper, where all changes are marked in red, meaningful. The gated baseline, along with the delay readout your questions led us to, now resides in the manuscript---results suggesting strongly that the iconic profile is more than a coincidence of gating. We hope you will revisit your assessment in light of these results and would be glad to answer any remaining questions before the discussion period ends. Thank you.
[1/2] Rebuttal by Authors
We thank the reviewer for their attentive remarks and questions. The possibility that the profiles we report might be nothing more than an artifact of generic gating is the most natural one to raise and, admittedly, one we should have already dispelled. Therefore, we now provide two new tests (n=3) that inspect the iconic claims in this light. The first keeps the gate and takes away only its fading, and the U-shape no longer forms (Table C1), attesting that the gate alone cannot be what draws the U. The second removes the stimulus and sees each stage's memory last only as long as its α (Table C3). We also showcase, through the parameter-matched controls the reviewer requested, how IconicNet remains unsurpassed, and how a parameter-matched ResNet-18 that has no iconic parameters performs the worst out of all controls (Table C1), reconfirming once again the abstract's claim that what drives these gains is not a mere expansion in parameters but instead how that expansion is put to work.
Temporal memory within a single forward pass
Standard deep neural networks (DNN), as we mention in the paper, altogether lack a temporal axis, so any temporal response we wished to elicit had to be configured, as best as possible, along a spatial one, which is to say within a single forward pass itself. Indeed, our motive from the start has been that, while we cannot implement a clockwork of milliseconds into the layers of a DNN, we can very well implement what happens in that space as time passes—this is the gate. Each block hence writes its stage's activation once into a trace that fades over four internal steps, and the gate reads from what remains at each of those steps. A gated structure, we argue, while inhabits no time itself, reproduces as closely as possible what the passing of time entails.
Nevertheless, the concept of fading is not exclusive to iconic memory—all mnemonic activity in cortex fades, the short-term kind extremely rapidly so. If all memory fades, then, how can we even gauge iconic memory? What makes it proper is that it fades, is short-term, but is only accessed, and immediately, as an outside passive buffer. A trace that only faded would be no different from any other memory. It becomes iconic when something reads from it without any rewrite, which renders the implementation of the gate as a sort of access a must, which is also beneficial to separate persistence from a magnitudinal correspondence lest it collides with γ. How long that buffer then remains available at each stage is what α sets, and what the delay test of our following reply measures directly (Table C3).
The input is indeed a single static image, and we will make sure to state this and what it entails explicitly in the revised abstract, also adding as future work how the network would behave if it were not a static image but instead a video feed, where a temporal axis already exists and need not be manufactured.
A parameter-matched gated baseline without decay
To isolate decay from the rest of the IconicBlock, we preserved every component as is but reduced the memory steps to one (single gate). The control retains the IconicBlock's parameters, projections, gate and normalization, and is hence the parameter-matched gated residual baseline without any decay that the reviewer has requested.
Table C1.
With equal parameters, the single gate variant reaches neither the gain nor the telltale U-shape of the IconicNet (Table C1), so the improvements do not come simply from increased representational capacity. Losing its temporal accumulation, the single gate block has the early γ tail (S1) disappear, with the terminal one (S4) still intact. Not a surprising result since, at S4, the trace barely fades and the decayed reads are nearly identical to the first, so that keeping only the initial trace changes the end stage very little, which has so far been affected chiefly by task demand, as we demonstrated in the paper—attesting, putatively, to its informational function.
Moreover, that the parameter-matched ResNet-18 gains the least (Table C1) is expected for so meager a change in size. The paper had posited as much, that a fading percept confers benefits beyond what a mere expansion in parameter space warrants, and we can now confirm once again that it does.