acceptodds
Withdrawn from ICLR 2027

Two Routes to the Middle: Placement Search and Brain Readouts Converge on Where Continual Learners Should Specialize

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual learners that keep a task-specific adapter in every block of a pre-trained vision transformer accumulate storage linearly with the number of tasks; keeping task-specific adapters in only a few blocks curbs this growth but raises the question of where to place them. We investigate this question from two perspectives. Algorithmically, training all contiguous four-block placements yields an inverted U: final accuracy peaks at intermediate depth and varies by up to 3.5 percentage points (pp), while inexpensive criteria based on weight spectra or activation statistics favor the deepest blocks. From neuroscience, the hierarchical organization and intermediate-stage plasticity of the visual cortex motivate us to ask whether a measurement taken outside the learner can guide layer specialization without placement search. LS-B observes the first tasks through a frozen fMRI encoding model of twelve human visual areas and commits task-specific capacity once to the blocks whose readouts vary most across tasks relative to their stable structure. Across three ViT-B/16 backbones, LS-B yields stable, backbone-specific allocations. On the two backbones with placement search, AugReg and iBOT, the selected blocks overlap the intermediate-depth region identified by search. Under matched storage and observation budgets, the selected blocks outperform the shallowest and deepest four-block configurations. On Split ImageNet-R, LS-B uses 60% of full-BiLoRA adapter storage while remaining within 1.5 pp of its final accuracy. The allocation requires no labels or backpropagation, adds under 0.6% runtime, and exhibits backbone-specific cortical signatures.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Brain-Inspired Learning, Parameter-Efficient Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uhPMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies where to keep task-specific adapters in a frozen ViT for continual learning, when only some blocks can keep them. The authors first show that placement search finds the best position at intermediate depth, but simple internal criteria prefer the deepest blocks. Then they propose LS-B, which uses a frozen fMRI encoding model to see how each block responds differently across the first few tasks, and selects the task-specific blocks one time without labels or search. They claim the two approaches agree on the middle region, and LS-B saves adapter storage with small accuracy drop.

Critical strengths
  • I think the experimental protocol is careful. Slot counts are stated for every comparison, observation and placement are separated by matched-observation controls, failed registered criteria and negative supplementary results are reported.

  • The placement measurement itself is useful for parameter-efficient continual learning, independent of the brain part. Showing that four-block windows at identical storage differ by 2.4 to 3.5 pp in an inverted U, and that spectral and activation criteria systematically go to the deepest window, is a concrete finding that other works on shared/task-specific partition can directly use.

Critical weaknesses
  • My main concern is that the placement claim is tested only against the two extreme windows, 0-3 and 8-11. Since Sec. 5.1 already shows an inverted U, almost any intermediate allocation is expected to beat both extremes, so this test can not show that the readout chooses better blocks than a random draw or a trivial fixed middle window. This matters a lot because, if I compute correctly from Table 2 and Sec. 5.3, observation followed by committing 0-3 or 8-11 lies within about 0.2 pp of the shallow-first interpolation at 72 slots, so the whole storage-matched gain of LS-B (0.89/0.77 pp) comes from which blocks are committed. Other numbers in the paper do not show that the readout adds much beyond being in the middle. Its ranking does not track accuracy (Spearman +0.10 and +0.38), AugReg sets overlap the best window by 1.44 blocks on average against 1.33 expected for random four blocks, and the iBOT margin over the deepest blocks is 0.03 pp against the 16-seed mean.

  • The contribution of the fMRI encoding model itself is not isolated. The readout is a dispersion statistic of frozen backbone features projected by a per-backbone encoding model, but the non-brain versions of the same score (random and PCA feature groups) are run only at 48 slots without observation, and the registered response-shuffled control was not run (Appendix G). Without a null under matched observation, it is hard to say whether brain data or simply a projection that is not deep-biased drives the allocation, and the title and the neuroscience framing depend exactly on this point.

Questions for authors
  • Could you run matched observation (72 slots, K=3) with several random four-block allocations and with a fixed middle window (4-7 for AugReg, 3-6 for iBOT)?

  • Could you report the same pipeline with a null encoding model under matched observation, for example shuffled vertex responses, a random projection with similar layer-selection marginals, or an encoding model of other NSD subject?

  • The 0.42 pp yardstick is one pair of runs measured before an RNG fix, so it is not clear to me why it is an upper bound on seed noise. Can you report the seed standard deviation of the configurations with 16 seeds instead?

  • In Appendix G, what exactly was the registered criterion of backbone-specific placement, and why did it fail? The abstract claims stable, backbone-specific allocations, so I would like to understand the difference.

  • In Fig. 9, AugReg block 2, which is committed in all seeds, seems to have both m_l and s_l close to zero. How robust is the top-k ranking to epsilon and to resampling the observed images, and does removing the near-duplicate FBA/FFA vertices change the AugReg allocation?

Reviewer_p4sJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method to select layer specialization in continual learning settings using unlabeled images based on brain readouts. They show recovering near-optimal selection on Split ImageNet-R while reducing storage requirements.

Critical strengths

The authors propose a novel method to leverage brain readouts for layer specialization in continual learning settings. This is an interesting use of neuroscience grounding for continual learning. They demonstrate that brain readouts lead to intermediate layers which aligns with the high level algorithmic search findings, suggesting a potential link between brain activity and continual learning in AI.

Critical weaknesses

While both routes lead to intermediate layers on the ImageNet split, the actual layers are different in each route. Together with the mixed results on the other benchmarks provided in the Appendix F, this suggests that the link between the brain readout and optimal layer specialization is not yet sufficiently established. I believe further evidence and experiments are needed to support this claim.

Questions for authors

Can you propose a general claim that can hold beyond the split ImageNet settings?

Reviewer_RHVDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates where to retain task-specific adapters in pretrained vision transformers for rehearsal-free class-incremental learning. The proposed method, LS-B, uses a frozen fMRI encoding model to obtain task-dependent readouts during an initial observation stage. It then selects a subset of blocks for continued task-specific adaptation, while the remaining blocks use shared adapters alongside their frozen observation-stage parameters. On Split ImageNet-R, the default configuration retains 60% of full BiLoRA adapter storage while remaining within 1.5 percentage points of its final accuracy. The paper compares these allocations with placement search and several learner-derived criteria.

Critical strengths
  1. The paper addresses a concrete architectural decision: where to allocate limited task-specific capacity in a continual learner.

  2. Using an independently fitted neural encoding model to inform this decision is an interesting direction. The reported storage–accuracy trade-off demonstrates potential practical utility.

  3. The authors clearly report negative results and limitations, including the absence of detectable score–accuracy correlations, seed sensitivity, and mixed supplementary results. This transparency helps readers assess the scope of the findings.

Critical weaknesses
  1. The key placement comparison is incomplete. Most four-block allocation controls retain 48 slots, compared with 72 for LS-B. As the authors acknowledge in Section 5.3, much of the advantage comes from the observation stage. The matched-observation experiments address this confound only for the shallowest and deepest four-block allocations. They do not include a fixed intermediate allocation or random allocation under the same observation and storage budgets. Consequently, the current evidence does not establish whether the brain readout improves upon a simple intermediate-depth policy.

  2. The specific utility of the neural readout remains unclear. Section 5.2 reports no detectable association between the readout ranking and configuration accuracy. The authors appropriately interpret the readout as a coarse structural prior. However, its value beyond a generic depth preference or a non-neural transformation of backbone features still requires stronger evidence. The activation-based comparisons use different early-task parameter budgets. The planned response-shuffled control was not run because the authors report that it was numerically degenerate; an alternative informative control is therefore still missing.

  3. The robustness of the placement benefit is uncertain. The critical matched-observation comparisons use only three seeds. For iBOT, the LS-B mean changes from 69.60% over seeds 0–2 to 68.88% over all 16 seeds, substantially reducing the reported margin over the deepest-block control when the reference is changed. Since the matched controls were not similarly extended, the magnitude and uncertainty of this benefit remain unresolved. Appendix F also reports mixed storage–accuracy gains across datasets and backbones, leaving the conditions under which the allocation rule is useful insufficiently characterized.

Questions for authors
  1. Under the same observation stage, shared-adapter training settings, and 72-slot budget, how does LS-B compare with a fixed intermediate four-block allocation and a random four-block allocation?

  2. Given the numerical degeneracy of the planned response-shuffled control, can an alternative non-neural control, evaluated under matched observation and storage budgets, establish what additional information the neural encoding model contributes?

  3. Can you provide stronger paired evidence for the matched-observation placement benefit, particularly on iBOT, and clarify how the mixed results in Appendix F delimit the method’s applicability?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.