- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies where to keep task-specific adapters in a frozen ViT for continual learning, when only some blocks can keep them. The authors first show that placement search finds the best position at intermediate depth, but simple internal criteria prefer the deepest blocks. Then they propose LS-B, which uses a frozen fMRI encoding model to see how each block responds differently across the first few tasks, and selects the task-specific blocks one time without labels or search. They claim the two approaches agree on the middle region, and LS-B saves adapter storage with small accuracy drop.
Critical strengths
-
I think the experimental protocol is careful. Slot counts are stated for every comparison, observation and placement are separated by matched-observation controls, failed registered criteria and negative supplementary results are reported.
-
The placement measurement itself is useful for parameter-efficient continual learning, independent of the brain part. Showing that four-block windows at identical storage differ by 2.4 to 3.5 pp in an inverted U, and that spectral and activation criteria systematically go to the deepest window, is a concrete finding that other works on shared/task-specific partition can directly use.
Critical weaknesses
-
My main concern is that the placement claim is tested only against the two extreme windows, 0-3 and 8-11. Since Sec. 5.1 already shows an inverted U, almost any intermediate allocation is expected to beat both extremes, so this test can not show that the readout chooses better blocks than a random draw or a trivial fixed middle window. This matters a lot because, if I compute correctly from Table 2 and Sec. 5.3, observation followed by committing 0-3 or 8-11 lies within about 0.2 pp of the shallow-first interpolation at 72 slots, so the whole storage-matched gain of LS-B (0.89/0.77 pp) comes from which blocks are committed. Other numbers in the paper do not show that the readout adds much beyond being in the middle. Its ranking does not track accuracy (Spearman +0.10 and +0.38), AugReg sets overlap the best window by 1.44 blocks on average against 1.33 expected for random four blocks, and the iBOT margin over the deepest blocks is 0.03 pp against the 16-seed mean.
-
The contribution of the fMRI encoding model itself is not isolated. The readout is a dispersion statistic of frozen backbone features projected by a per-backbone encoding model, but the non-brain versions of the same score (random and PCA feature groups) are run only at 48 slots without observation, and the registered response-shuffled control was not run (Appendix G). Without a null under matched observation, it is hard to say whether brain data or simply a projection that is not deep-biased drives the allocation, and the title and the neuroscience framing depend exactly on this point.
Questions for authors
-
Could you run matched observation (72 slots, K=3) with several random four-block allocations and with a fixed middle window (4-7 for AugReg, 3-6 for iBOT)?
-
Could you report the same pipeline with a null encoding model under matched observation, for example shuffled vertex responses, a random projection with similar layer-selection marginals, or an encoding model of other NSD subject?
-
The 0.42 pp yardstick is one pair of runs measured before an RNG fix, so it is not clear to me why it is an upper bound on seed noise. Can you report the seed standard deviation of the configurations with 16 seeds instead?
-
In Appendix G, what exactly was the registered criterion of backbone-specific placement, and why did it fail? The abstract claims stable, backbone-specific allocations, so I would like to understand the difference.
-
In Fig. 9, AugReg block 2, which is committed in all seeds, seems to have both m_l and s_l close to zero. How robust is the top-k ranking to epsilon and to resampling the observed images, and does removing the near-duplicate FBA/FFA vertices change the AugReg allocation?