acceptodds
Under review as a conference paper at ICLR 2027

Learning to Compose Prompts with Hebbian Pathways for Continual Learning

Abstract

Parameter-efficient continual learning adapts a frozen pre-trained model with small modules, such as prompts, that hold the knowledge acquired over the task sequence. However, allocating modules to each task avoids interference but prevents reuse and enlarges the pool with every task, whereas sharing modules across tasks enables reuse but exposes the shared modules to interference from later tasks. This paper proposes a new method of Learning to Compose (L2C) that represents the task knowledge as a composition of prompts selected at multiple layers of a vision transformer, thereby lowering the unit of sharing and protecting from tasks to prompts. Under this representation, a new task can be expressed as a new composition that reuses the prompts of earlier tasks. L2C determines which prompts to protect from a selection history that is decoupled from the task loss, in which a Hebbian rule accumulates the selection probabilities of prompts chosen together at adjacent layers. The history serves as a prior that stabilizes compositions already in use and determines which prompts are frozen or re-initialized orthogonally to the frozen keys, recovering capacity without enlarging the pool. Experiments on three benchmarks with 5 to 40 tasks show that L2C attains the highest average incremental accuracy and the lowest forgetting against ten baselines in all nine settings, while training fewer than 0.2% of the backbone parameters.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, class-incremental learning, rehearsal-free continual learning, prompt learning, parameter-efficient fine-tuning, vision transformer, Hebbian learning, Oja's rule

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MiHaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

L2C selects prompts at three depths of a frozen ViT. A transition history updated by Oja's rule contributes a selection prior and determines which prompts are frozen or have their keys re-initialized. The method combines centered queries, prompt reuse, and a fixed-size pool, with results on CIFAR-100, ImageNet-R, and ImageNet-A.

Critical strengths

Selecting and protecting individual prompts across depths is a reasonable way to support reuse without allocating an entire new module per task. Table 1 reports consistently favorable average accuracy and forgetting across nine settings and five class-order seeds. The ablations include alternatives to Oja's rule and separate the history prior from similarity-based selection. The reported parameter count and throughput are also useful.

Critical weaknesses

The contribution of the Hebbian history is not fully separated from that of generic popularity-based protection. Proposition 1 and the discussion show that in-degree closely tracks selection frequency, while the prior is relatively stable across inputs. A matched baseline using per-prompt usage counts for freezing and a popularity bias for selection would test whether storing cross-stage pathways is necessary. Table 2(b) changes the transition update rule, which does not answer this specific question. The main baseline comparison uses a common five-epoch schedule, which need not give comparable optimization quality or computation across methods. The discussion refers to longer-schedule comparisons with HiDe-Prompt, MOS, and TUNA, but also changes L2C to six pools of twenty prompts. A comparison that keeps L2C's configuration fixed and reports both computation and accuracy would clarify the advantage.

Questions for authors

Does L2C outperform a matched usage-count freezing policy with an independent per-stage popularity prior? How often do old examples change their selected prompts after key recycling and subsequent training?

2 comments
AuthorsOpenReview ↗

Comment 1 of 2

Presentation

Apologies for the presentation, which made the method difficult to follow. The revision opens Section 3 with a three-step summary of the procedure (computing the queries from the frozen backbone; selecting prompts with the selection history; freezing and re-initializing prompts at task boundaries) and adds a table of notation to the supplementary material (Table 24).

W1 (Hebbian history versus popularity)

Thank you for this suggestion, which isolates the contribution of the Hebbian history. The requested baseline was run, together with a stricter one. Both remove the transition matrices and both are below L2C, which shows that storing the cross-stage pathways is necessary.

Variants of the history. In both baselines, the usage decides freezing and re-initialization and is added to the selection score with the same weight λ.

VariantUsage statisticPrior
Usage count (requested)Cumulative selection probability of each promptPer stage, independent of the input
Usage with Oja's decaySame recursion as the in-degree of L2C (Eq. (8) of the supplementary material)Per stage, independent of the input
L2CIn-degree of the transition matricesConditioned on the selection at the preceding stage

Results on the 20-task sequences (mean ± standard deviation over five seeds; Table 18 of the revision).

VariantIN-R AvgIN-R FgtIN-R ATA_{T}IN-A AvgIN-A FgtIN-A ATA_{T}
Usage count75.71 ± 1.415.37 ± 0.6570.61 ± 0.5359.98 ± 1.5011.37 ± 1.1853.53 ± 0.88
Usage with Oja's decay74.43 ± 1.365.62 ± 0.5569.98 ± 0.7660.11 ± 1.5210.90 ± 1.5355.48 ± 1.12
L2C77.41 ± 0.654.40 ± 0.8672.14 ± 0.7265.11 ± 1.508.99 ± 1.3860.51 ± 1.01
  • L2C is better than both baselines in every metric on both benchmarks.

  • The second baseline differs from L2C only in that its prior is not conditioned on the preceding stage, and the margin remains. The gain comes from the cross-stage transitions, not from recording the usage.

Conclusion. Popularity-based protection alone does not reproduce the results of L2C, so the contribution of the Hebbian history is separated from that of popularity. The results are added to Section 4.3 and to Section D.12 of the supplementary material.

W2 (five-epoch schedule)

The common schedule gives every method the same computation, and the comparison does not rely on it alone, since the methods that need longer schedules are also evaluated at 20 epochs.

  • Protocol. The common 5-epoch schedule follows continual learning under a computational budget (Prabhu et al., 2023; Seo et al., 2025): every method receives the same number of iterations per task.

  • Computation. It is comparable under this schedule: the training times on 10-task ImageNet-R lie within 18.0–21.1 minutes (table in W3), and the throughput of L2C lies between those of CODA-Prompt and L2P (Table 15 of the supplementary material).

  • Optimization quality. L2C has converged at 5 epochs, and the methods designed for longer schedules are also compared at 20 epochs (W3).

AuthorsOpenReview ↗

Comment 2 of 2

W3 (fixed configuration under a longer schedule)

Thank you for the suggestion; the experiment was run as requested. The default configuration (three pools of ten prompts) was trained for 20 epochs without any change, and parameters and training time are reported for every method. With its configuration fixed, L2C keeps its advantage at 5 epochs.

10-task ImageNet-R at 5 and 20 epochs per task (five seeds). Parameters are counted after the last task; time is the training time of one run.

MethodTrainable parameters5 epochs: Avg / FgtTime (min)20 epochs: Avg / FgtTime (min)
HiDe-Prompt937,16068.75 / 4.1221.174.22 / 12.8877.1
MOS3,197,00069.88 / 3.8919.383.07 / 5.0440.9
TUNA3,639,56070.15 / 3.5519.685.91 / 6.2044.1
L2C (default)292,04079.24 / 3.5618.079.53 / 5.6139.1
L2C-Large706,76077.15 / 6.9420.186.81 / 2.8146.8
  • Default configuration. It has converged at 5 epochs. At 5 epochs it exceeds all three methods in Avg, and it exceeds HiDe-Prompt trained for 20 epochs.

  • MOS and TUNA at 20 epochs. Both exceed the default configuration, but with 11 and 12 times its trainable parameters: they add an adapter per task, and the longer schedule trains this growing capacity.

  • A larger configuration under the longer schedule. L2C-Large (six pools of 20 prompts with transition matrices between all pairs of pools; 0.71M parameters, less than one quarter of those of MOS and TUNA and independent of the number of tasks) attains the highest Avg and the lowest Fgt at 20 epochs. This is consistent with Tables 12 and 17, which are measured at 5 epochs: under that budget, larger pools and more stages do not raise Avg (L2C-Large at 5 epochs: 77.15, below the default 79.24), whereas the longer schedule gives the additional prompts enough updates to be trained.

Conclusion. With its configuration fixed, L2C is the best at 5 epochs and does not need a longer schedule. The lead of MOS and TUNA over the default configuration at 20 epochs reflects the capacity that a longer schedule can train rather than a weakness of the method, since it disappears with L2C-Large. The default row and the columns of parameters and time are added to Table 10 of the supplementary material.

Q1 (usage-count baseline)

Yes. Against this baseline (“Usage count” in W1), L2C is 1.70 and 5.13 points higher in Avg on 20-task ImageNet-R and ImageNet-A, with lower Fgt and higher ATA_{T} on both.

Q2 (selection changes of old examples)

This was measured directly. Old examples change their prompts rarely where the history applies, and almost never because of key recycling.

Test inputs of each task whose selected set after a later task differs from the set selected right after their own task (20 tasks; averaged over all later tasks and five seeds). "Re-initialized" is the share of the changes in which a re-initialized prompt enters the set.

ImageNet-R: changedRe-initializedImageNet-A: changedRe-initialized
L2C, shallow17.1%–29.8%–
L2C, middle10.6%1.0%22.2%7.1%
L2C, deep16.1%0.6%23.7%3.1%
Single pool (M = 10, K = 5)61.8%–79.1%–
  • Subsequent training. At the middle and deep stages, to which the history applies, the selection is maintained in 84–89% of the cases on ImageNet-R and 76–78% on ImageNet-A; a single pool of the same size maintains it in 21–38%.

  • Key recycling. A re-initialized key is drawn from the orthogonal complement of the hub keys and starts at a neutral similarity to earlier queries (Proposition 3), so recycling leaves the selection of earlier inputs nearly unchanged.

Conclusion. Key recycling is not a source of forgetting. The table is added to Section D.14 of the supplementary material (Table 20).


Reviewer_Z9wnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits the allocation–sharing trade-off in PEFT-based continual learning by making decisions at the prompt level rather than the task level. Prompts are selected at three depths of a frozen ViT using query–key similarity and a transition prior learned. The resulting selection history also determines which prompts are preserved as hubs and which are reinitialized, allowing capacity to be recycled within a fixed-size pool. The evaluation covers three benchmarks, nine task configurations, ten baselines, and five seeds.

Critical strengths

1.Moving allocation and sharing decisions from tasks to individual prompts provides a clear conceptual contribution, supported by a concrete mechanism. 2.The ablations show substantial, cumulative gains: Avg accuracy increases from 65.94 to 77.41 in Table 2a. The stage-removal experiments also support the proposed functional distinction: shallow prompts contribute more to retention, while deeper prompts contribute more to input-specific adaptation.

Critical weaknesses
  1. The smallest gains need uncertainty estimates. On 20-task ImageNet-A, the improvement over APER-A is only 0.26 points (65.11 vs. 64.85). The main table omits standard deviations, referring instead to unavailable supplementary tables.
  2. The parameter-efficiency claim needs qualification. Prompt, key, and transition-state sizes remain fixed, but the classifier grows as classes are introduced. Please distinguish fixed adaptation overhead from total model size and report both over the task sequence.
  3. Sensitivity analysis and presentation could be improved. The choices of ρ\rho, λ\lambda, KK, and η\eta lack accessible sensitivity results. The manuscript also contains spacing errors, inconsistent citation capitalization, and a missing citation where DeiT is introduced. These issues should be corrected.
Questions for authors

see above.

4 comments
AuthorsOpenReview ↗

Comment 1 of 1

W1 (uncertainty estimates)

The standard deviations exist for every setting and are reproduced below. In the setting mentioned, the difference in Avg is within the variation across seeds, while Fgt and ATA_{T} separate the two methods.

Standard deviations over five seeds are reported for every setting in Table 7 (Avg, Fgt) and Table 8 (ATA_{T}) of the supplementary material, which was uploaded as a separate file. Table 1 keeps only the means because of the page limit. Both tables are reproduced in the two comments below.

20-task ImageNet-A (mean ± standard deviation over five seeds).

MethodAvgFgtATA_{T}
APER-A64.85 ± 0.1110.88 ± 0.0858.12 ± 0.86
L2C65.11 ± 1.508.99 ± 1.3860.51 ± 1.01
  • L2C is 1.89 points lower in Fgt and 2.39 points higher in ATA_{T}.

  • APER fixes its representation after the first task, hence its small variation across seeds; L2C trains prompts in every task.

Conclusion. The claim of the paper does not rest on this margin: the method closest to L2C in Avg changes with the setting (RanPAC, APER-S, EASE, APER-A), whereas L2C ranks first in both Avg and Fgt in all nine settings of Table 1.

W2 (parameter efficiency)

Thank you for this precise suggestion; it is adopted. The fixed adaptation overhead and the total size are now reported separately over the task sequence, and the claim of parameter efficiency holds under either accounting (0.16% and 0.34% of the backbone).

Parameters of L2C over the 40-task ImageNet-R sequence (Table 23 of the revision).

Tasks learned1102040
Fixed adaptation overhead138,440138,440138,440138,440
Classifier rows in use (769 per class seen)3,84538,45076,900153,800
Total in use142,285176,890215,340292,240
Total allocated292,240292,240292,240292,240
  • Fixed adaptation overhead. Prompts (115,200), keys (23,040), and transition matrices (200): 0.16% of the backbone, independent of the number of tasks.

  • Classifier. 769 parameters per class (D + 1), as in every compared method with a linear classifier. It is allocated once for all 200 classes and the outputs of classes not yet introduced are masked, so the allocated total (0.34% of the backbone) is the upper bound at any point of the sequence.

  • Reference. After 40 tasks, methods that add an adapter per task have 12.33M (MOS) and 12.78M (TUNA) trainable parameters.

  • Revision. The statement of "fewer than 0.2%" in the Abstract and Section 3.5 is restricted to the fixed adaptation overhead, and the table above is added to the supplementary material (Section D.16).

W3 (sensitivity and presentation)

Thank you for both points; both are addressed. Sensitivity results are now available for all four hyperparameters and show that the results do not depend on fine tuning, and the presentation issues are corrected.

Sensitivity. λ is covered in Table 12 of the supplementary material, and K, η, and ρ in additional runs that change one value at a time (Table 21, Section D.15 of the revision).

Sensitivity on 20-task ImageNet-R (Avg, mean over five seeds; * marks the default).

HyperparameterAvg for each valueRange
λ0: 75.48, 0.3: 76.48, 0.5*: 77.41, 1.0: 76.361.93
K1: 75.63, 3: 76.96, 5*: 77.41, 7: 77.141.78
η0.001: 77.52, 0.01*: 77.41, 0.1: 77.650.24
ρhub\rho_{\mathrm{hub}}0.1: 77.41, 0.2*: 77.41, 0.3: 77.410.00
ρdead\rho_{\mathrm{dead}}0.1: 77.56, 0.2*: 77.41, 0.3: 77.400.16
  • L2C is insensitive to η, ρhub\rho_{\mathrm{hub}}, and ρdead\rho_{\mathrm{dead}}. Over two orders of magnitude of η, the largest ηE[bj2]\eta\mathbb{E}[b_j^2] stays below 1.2×10−31.2\times10^{-3}, within the regime of Corollary 1. The results for ρhub\rho_{\mathrm{hub}} coincide because the number of hubs is set by the logarithmic cap.

  • K. Avg increases from K = 1 to K = 5 and saturates: K = 7 (77.14 ± 0.65) is within one standard deviation of the default (77.41 ± 0.65). The default K = 5 is fixed for all benchmarks, and a larger K only adds inserted tokens (LpL_{p} = 5 more per stage for each additional prompt).

In sum, Avg varies within 0.24 points for η and ρ, a single value of λ is used for all benchmarks, and Avg saturates at the default K.

Presentation. Corrected: the citation of DeiT (Touvron et al., 2021) is added in Section 4.1, the capitalization of the references is unified, and the spacing errors are fixed.


AuthorsOpenReview ↗

Tables 7 and 8 of the supplementary material (1/2)

Tables 7 and 8 of the supplementary material are reproduced below (mean ± standard deviation over five seeds), with ATA_{T} on 5-task CIFAR-100 added. Three typographical errors in the standard deviations of Table 8 (L2P on 20-task ImageNet-R; APER-V on 10- and 20-task ImageNet-A) are corrected here and in the revision.

CIFAR-100: Avg and Fgt (%).

Method5T Avg5T Fgt10T Avg10T Fgt20T Avg20T Fgt
L2P84.36 ± 1.437.23 ± 1.1781.94 ± 1.757.65 ± 1.9478.14 ± 1.868.97 ± 1.62
DualPrompt87.84 ± 0.788.54 ± 0.7886.45 ± 0.409.12 ± 1.0182.54 ± 0.6511.76 ± 1.25
CODA-P86.81 ± 0.4810.01 ± 0.4985.50 ± 0.4210.92 ± 0.6881.97 ± 0.8412.05 ± 1.68
RanPAC90.19 ± 0.123.66 ± 0.0588.58 ± 0.154.41 ± 0.0887.72 ± 0.114.98 ± 0.09
EASE90.18 ± 1.047.04 ± 1.1088.53 ± 0.897.67 ± 0.8487.10 ± 0.357.53 ± 0.68
LAE88.49 ± 0.477.32 ± 0.7786.81 ± 0.377.96 ± 0.8983.25 ± 0.509.89 ± 0.93
MEMO68.32 ± 1.2629.69 ± 2.1174.17 ± 1.7630.04 ± 2.1737.43 ± 2.5437.17 ± 2.01
APER-A89.31 ± 0.085.24 ± 0.0388.14 ± 0.096.00 ± 0.0586.69 ± 0.076.58 ± 0.06
APER-S89.46 ± 0.104.84 ± 0.0489.13 ± 0.115.45 ± 0.0688.00 ± 0.095.82 ± 0.05
APER-V89.53 ± 0.094.90 ± 0.0588.51 ± 0.105.61 ± 0.0486.25 ± 0.086.00 ± 0.07
L2C90.41 ± 0.903.29 ± 0.5889.71 ± 1.213.51 ± 0.6188.79 ± 1.224.16 ± 0.98

ImageNet-R: Avg and Fgt (%).

Method10T Avg10T Fgt20T Avg20T Fgt40T Avg40T Fgt
L2P72.07 ± 0.446.76 ± 0.6564.03 ± 0.308.54 ± 0.8063.80 ± 1.077.16 ± 0.96
DualPrompt74.11 ± 0.487.73 ± 0.8771.07 ± 0.3110.33 ± 0.4869.67 ± 0.468.34 ± 0.53
CODA-P74.41 ± 0.258.09 ± 0.5773.12 ± 0.919.13 ± 0.7568.53 ± 0.2510.79 ± 1.69
RanPAC78.12 ± 0.125.94 ± 0.0875.06 ± 0.116.22 ± 0.0972.24 ± 0.126.39 ± 0.09
EASE70.80 ± 0.236.98 ± 0.6270.69 ± 0.117.18 ± 0.6170.92 ± 0.137.47 ± 0.40
LAE74.01 ± 0.296.90 ± 0.5571.54 ± 0.398.43 ± 0.7868.67 ± 0.479.38 ± 0.42
MEMO69.51 ± 0.7723.28 ± 0.2969.43 ± 1.1025.43 ± 1.2650.49 ± 3.9829.44 ± 2.57
APER-A70.46 ± 0.096.10 ± 0.0670.63 ± 0.086.43 ± 0.0570.94 ± 0.086.71 ± 0.05
APER-S78.05 ± 0.115.34 ± 0.0576.62 ± 0.106.04 ± 0.0771.70 ± 0.106.37 ± 0.06
APER-V76.39 ± 0.105.13 ± 0.0674.39 ± 0.095.61 ± 0.0671.05 ± 0.096.02 ± 0.07
L2C79.24 ± 0.863.56 ± 0.4077.41 ± 0.654.40 ± 0.8673.10 ± 0.865.74 ± 0.43

ImageNet-A: Avg and Fgt (%).

Method10T Avg10T Fgt20T Avg20T Fgt40T Avg40T Fgt
L2P54.00 ± 0.5015.08 ± 1.4546.20 ± 1.599.75 ± 1.8331.00 ± 2.1710.11 ± 2.02
DualPrompt62.34 ± 0.939.42 ± 0.9160.16 ± 1.159.99 ± 1.4054.26 ± 0.9112.12 ± 1.15
CODA-P61.86 ± 0.7810.74 ± 1.9361.67 ± 1.0310.40 ± 2.1155.00 ± 2.5411.51 ± 2.58
RanPAC65.94 ± 0.129.38 ± 0.0863.47 ± 0.139.67 ± 0.0960.76 ± 0.1522.45 ± 0.08
EASE66.00 ± 0.1611.64 ± 0.8864.82 ± 0.4212.24 ± 0.9962.19 ± 0.5912.44 ± 0.97
LAE62.25 ± 1.1710.20 ± 1.0359.81 ± 1.6212.30 ± 1.1555.13 ± 0.7012.44 ± 0.88
MEMO31.79 ± 2.0934.21 ± 1.1525.79 ± 4.2532.28 ± 3.1514.93 ± 7.5221.54 ± 6.29
APER-A65.05 ± 0.0910.21 ± 0.0664.85 ± 0.1110.88 ± 0.0862.40 ± 0.0911.28 ± 0.05
APER-S65.99 ± 0.1010.23 ± 0.0563.29 ± 0.1011.00 ± 0.0761.31 ± 0.1211.46 ± 0.07
APER-V60.13 ± 0.1110.13 ± 0.0763.79 ± 0.0910.52 ± 0.0661.10 ± 0.1011.01 ± 0.06
L2C66.47 ± 1.697.49 ± 1.1965.11 ± 1.508.99 ± 1.3863.57 ± 1.339.78 ± 0.96
AuthorsOpenReview ↗

Tables 7 and 8 of the supplementary material (2/2)

Final accuracy ATA_{T} (%) on CIFAR-100 and ImageNet-R. The 5-task CIFAR-100 column is added in the revision.

MethodCIFAR-100 5TCIFAR-100 10TCIFAR-100 20TImageNet-R 10TImageNet-R 20TImageNet-R 40T
L2P79.91 ± 0.6476.74 ± 1.0370.62 ± 0.9866.39 ± 1.0356.44 ± 1.2756.24 ± 4.14
DualPrompt83.26 ± 0.8180.74 ± 0.6975.31 ± 0.7668.19 ± 0.4062.56 ± 6.4262.86 ± 0.39
CODA-P81.39 ± 0.4578.68 ± 0.4073.51 ± 0.8868.33 ± 0.2765.76 ± 1.3859.12 ± 7.09
RanPAC89.41 ± 0.1185.05 ± 0.3380.54 ± 0.0674.99 ± 0.2872.62 ± 0.2466.63 ± 0.12
EASE86.24 ± 0.3983.35 ± 0.1880.54 ± 0.1365.30 ± 0.0564.68 ± 0.0964.47 ± 0.10
LAE84.20 ± 0.4781.57 ± 0.3176.00 ± 0.8568.90 ± 0.2864.62 ± 3.4860.90 ± 0.68
MEMO59.22 ± 5.0965.53 ± 2.5230.76 ± 10.2160.39 ± 1.9659.34 ± 1.2841.40 ± 22.47
APER-A85.19 ± 0.1783.22 ± 0.2779.09 ± 0.1765.02 ± 0.1964.63 ± 0.0264.51 ± 0.02
APER-S85.07 ± 0.1083.91 ± 0.4080.53 ± 0.7872.17 ± 0.5370.56 ± 0.5467.53 ± 0.75
APER-V85.01 ± 0.1282.97 ± 0.5578.57 ± 1.4370.20 ± 0.7067.96 ± 1.1164.53 ± 1.77
L2C87.37 ± 0.3084.87 ± 0.3180.88 ± 1.1374.46 ± 0.5572.14 ± 0.7268.99 ± 0.70

Final accuracy ATA_{T} (%) on ImageNet-A.

MethodImageNet-A 10TImageNet-A 20TImageNet-A 40T
L2P41.66 ± 5.9535.26 ± 8.0118.36 ± 3.72
DualPrompt54.37 ± 0.9051.30 ± 1.1643.77 ± 3.67
CODA-P52.72 ± 1.3251.99 ± 0.5444.15 ± 2.33
RanPAC61.30 ± 0.4960.40 ± 0.4147.88 ± 6.44
EASE58.56 ± 0.2658.59 ± 0.3052.38 ± 14.02
LAE53.22 ± 0.8749.31 ± 2.3643.34 ± 2.10
MEMO20.47 ± 4.4215.05 ± 5.878.50 ± 10.99
APER-A60.59 ± 1.1258.12 ± 0.8655.54 ± 0.00
APER-S58.56 ± 1.4358.21 ± 0.8857.17 ± 1.39
APER-V51.24 ± 1.3055.75 ± 0.5356.56 ± 2.91
L2C62.80 ± 0.9760.51 ± 1.0159.72 ± 0.77

Further details are given in the supplementary material. Please let us know if it cannot be accessed.

Thank you for your response. I believe the score I assigned is reasonable at this stage, given that the reviewers’ scores are generally low overall.

Reviewer_8iJoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Learning to Compose (L2C) with Hebbian pathways for continual learning. The proposed method utilizes a stage-wise (stage=4 blocks of ViT layers) prompt pool, where the top-k prompt on each stage is selected based on similarity. In general, the paper has a unique idea, the proposed method is well formulated, and the writing is well presented. However, the paper has several issues that need clarification. Please see weaknesses and questions.

Critical strengths
  1. This paper proposes a unique idea from both prompt structure and selection mechanism.

  2. The proposed method is well formulated and is supported by a rigorous theoretical analysis.

  3. The writing is well presented and easy to follow.

Critical weaknesses
  1. Despite the uniqueness of stage-wise pool architecture, its significant impact is not proven yet compared to the existing layer-wise and multi-head-wise structure. In addition, it is still not clear to me how the prompt is attached to each stage feature. Does it change the dimension of the original (without prompt) feature?

  2. Baseline: Only few of the compared baselines are up-to-date. In addition, it has not included the advanced prompt-based methods such as ConvPrompt, LEAPGen, and NorGA. In addition, the included baselines need citations. I cannot easily find which references are for APER and MEMO.

  3. Metrics: The final accuracy (A_T) and final forgetting (F_T) should be included in each setting of the performance measurement. While the average incremental accuracy and forgetting describe the cumulative/historical performance of the methods, the final accuracy and forgetting represent the latest model performance after finishing all the tasks. Both final and incremental performance are important to observe.

  4. Performance: The small performance difference between the proposed method and the existing methods makes a doubtful impression on the significant impact of the proposed idea, despite its uniqueness and solid formulation. For instance, the difference of incremental accuracy and forgetting for CIFAR-100 5 tasks and 10 tasks and ImageNet-R 20 tasks between L2C and an existing method is less than 1%. Furthermore, looking at the final accuracy (shown in the supplemental), the difference is even smaller, and in several cases, L2C has lower performance than an existing method. Compared to ConvPrompt, LEAPGen, and NoRGa, which were published one or 2 years ago, L2C has a lower performance on CIFAR100 and ImageNet-R. From my perspective, it raises a serious question if the proposed L2C does offer a promising solution.

Questions for authors
  1. Performance vs. the size of trainable parameters: How much is the size of total trainable parameters in L2C? And with that size of parameters, is it reasonable to achieve such a small difference in performance? As the prompt is structured into stage-wise pools, it may have a bigger size of parameters than a single pool.

  2. A prompt within a pool may be trained on many tasks. Thus, the prompt that is already optimal for the previous task may become suboptimal due to being tuned in the current task. How does L2C handle such an issue? As it is likely to lead to forgetting.

  3. As L2C needs to check similarities on each stage, it may require extra computation compared to a single similarity check for prompted layers, as in L2P or Dual-Prompt. How is the performance and complexity (training and inference time) trade-off of the proposed method?

ConvPrompt: Convolutional Prompting Meets Language Models for Continual Learning [CVPR 2024]

LEAPGen: Vision and Language Synergy for Rhearsal-Free Continual Learning [ICLR 2025]

NoRGa: Mixture of Experts Meets Prompt-Based Continual Learning [NeurIPS 2025]

7 comments
AuthorsOpenReview ↗

Comment 1 of 4

W1 (stage-wise structure; prompt attachment)

Thank you for the careful reading of the architecture. Both points are answered below: (a) controlled experiments show that the stage-wise structure exceeds both a single pool with the same parameters and a layer-wise structure with more parameters, and (b) the prompts do not change the feature dimension.

(a) Impact of the stage-wise structure. The structure is what produces the gain. Three variants that change only the structure isolate it.

Structure of the prompt pools on 20-task ImageNet-R (mean ± standard deviation over five seeds; Table 17, Section D.11 of the revision).

StructureTrainable parametersAvgFgtATA_{T}
3 stages (L2C)292,04077.41 ± 0.654.40 ± 0.8672.14 ± 0.72
12 stages (one pool per block)430,28076.44 ± 1.337.20 ± 0.3370.83 ± 0.75
Single pool (M = 30, all components)292,04073.95 ± 1.275.12 ± 0.7667.30 ± 0.27
  • Single pool, same parameters. With an identical parameter count and every other component enabled, the single pool is lower in all three metrics. The gain comes from the composition across stages, not from capacity (see also Table 2(a)).

  • Layer-wise (12 stages). With 47% more parameters, the layer-wise variant forgets more. Four stages show the same tendency (Table 2(c); Table 4 of the supplementary material).

  • Existing layer-wise methods. DualPrompt and CODA-Prompt attach prompts to five layers. L2C is higher in Avg and lower in Fgt than both in all nine settings of Table 1.

  • Head-wise structures. NoRGa is compared in W2.

(b) Attachment and feature dimension. The feature dimension does not change. Prompts are inserted as tokens after [CLS], not as prefixes to the keys and values of attention; only the sequence length increases.

Token sequence of L2C (K = 5 prompts of length LpL_{p} = 5 per stage).

BlocksInserted prompt tokensSequence lengthToken dimension
Without prompts0197768
1–4 (shallow)25222768
5–8 (middle)50247768
9–12 (deep)75272768

The prompts of each stage are inserted at its first block and kept up to block 12. The [CLS] output fed to the classifier remains 768-dimensional. Section 3.2 of the revision states this explicitly.


AuthorsOpenReview ↗

Comment 2 of 4

W2 (baselines and citations)

Thank you for pointing to the recent prompt-based methods. The comparison is extended accordingly: NoRGa was run with its official code, and under the same training budget L2C is on par on ImageNet-R and better on ImageNet-A with about one twentieth of the parameters. ConvPrompt and LEAPGen rely on language and are discussed in Related Work, and the missing citations are added.

(a) Recent baselines. Four of the compared methods were published in 2024–2025, and HiDe-Prompt (NeurIPS 2023) is compared as well.

Recent methods compared in the submission.

MethodVenueLocation
EASECVPR 2024Table 1
APERIJCV 2025Table 1
MOSAAAI 2025Table 10, supplementary material
TUNAICCV 2025Table 10, supplementary material
HiDe-PromptNeurIPS 2023Table 10, supplementary material

(b) NoRGa, ConvPrompt, and LEAPGen. NoRGa (NeurIPS 2024) was run with its official code, both with 5 epochs for each of its three training stages and with its original schedule.

Comparison with NoRGa (mean ± standard deviation over five seeds; Table 19, Section D.13 of the revision). Original schedule: 20 / 150 / 30 epochs (20 / 80 / 30 on CIFAR-100).

SettingMetricNoRGa (5 / 5 / 5)NoRGa (original)L2C (5)
CIFAR-100, 10 tasksAvg91.24 ± 0.9393.80 ± 0.3689.71 ± 1.21
Fgt5.64 ± 0.712.26 ± 0.173.51 ± 0.61
ATA_{T}90.15 ± 0.9491.56 ± 0.4884.87 ± 0.31
ImageNet-R, 20 tasksAvg77.60 ± 0.8483.37 ± 1.0177.41 ± 0.65
Fgt4.44 ± 0.314.40 ± 0.254.40 ± 0.86
ATA_{T}71.08 ± 0.4980.89 ± 0.5572.14 ± 0.72
ImageNet-A, 20 tasksAvg60.62 ± 3.4061.11 ± 3.6565.11 ± 1.50
Fgt13.06 ± 0.4714.94 ± 0.758.99 ± 1.38
ATA_{T}53.17 ± 2.5253.69 ± 2.6860.51 ± 1.01

Resources used on 20-task ImageNet-R.

NoRGaL2C
Trainable parameters5.74M0.29M
Training stages per task31
Stored class statisticsYesNo
Capacity grows with the number of tasksYesNo
  • Same budget. L2C is higher on ImageNet-A and on par on ImageNet-R, with lower or equal Fgt in all three settings. NoRGa is higher only on CIFAR-100 (Avg and ATA_{T}), and it obtains this with task-specific prompts, stored class statistics, and three training stages, with about 20 times the trainable parameters of L2C (measured on ImageNet-R); even so, its Fgt is higher there.

  • Original schedule. The longer schedule raises NoRGa on CIFAR-100 and ImageNet-R but not on ImageNet-A, where it remains below L2C trained for 5 epochs. Its reported numbers therefore depend on a schedule that is not comparable to Table 1, in which every method is trained for 5 epochs per task.

  • ConvPrompt and LEAPGen. Both use language: ConvPrompt (CVPR 2024) sets the number of prompts per task from class descriptions generated by a large language model, and LEAPGen (ICLR 2025) generates prompts from language inputs. Every method in Table 1 uses images and a frozen vision backbone only, so the two belong to a different setting. Related Work of the revision describes both.

In sum for (b), the advantage reported for NoRGa comes from its schedule and resources, not from a stronger prompt structure.

(c) Citations. Added. Every baseline in Section 4.1 and Table 1 is now cited (APER: Zhou et al., 2025; MEMO: Zhou et al., 2023).

W3 (final accuracy and final forgetting)

Final and incremental performance are indeed both needed, and both are reported in the submission. Fgt in Table 1 is the final forgetting, in which L2C is the lowest in all nine settings, and ATA_{T} is given in Table 1 and in Table 8 of the supplementary material. The definition is now stated explicitly so that this is clear from the main text.

Metrics and where they are reported.

MetricReported asLocation
Average incremental accuracyAvgTable 1, all nine settings
Final forgetting (FTF_{T})FgtTable 1, all nine settings
Final accuracy (ATA_{T})ATA_{T}Table 1 (longest sequence of each benchmark); Table 8 of the supplementary material (with standard deviations; all nine settings in the revision)
  • Final forgetting. Fgt=1T−1∑i<Tmax⁡i≤t<T(at,i−aT,i)\mathrm{Fgt}=\frac{1}{T-1}\sum_{i<T}\max_{i\le t<T}(a_{t,i}-a_{T,i}) (Chaudhry et al., 2018) is computed after the last task. It is the final forgetting, not an average over the task sequence. Section 4.1 and the caption of Table 1 now state this.

  • Final accuracy. The page limit kept ATA_{T} of the shorter sequences in Table 8. The one missing setting, 5-task CIFAR-100, is added in the revision (L2C: 87.37 ± 0.30), so that ATA_{T} covers all nine settings.


AuthorsOpenReview ↗

Comment 3 of 4

W4 (size of the performance difference)

Thank you for the candid assessment. In short, the difference is not small once Avg and Fgt are considered together, the cases in which L2C is lower are confined to short sequences against a method with a fixed representation, and the higher numbers of NoRGa come from a longer schedule.

(a) Avg and Fgt. In the three settings mentioned, every baseline is behind L2C in both metrics, and a baseline that is close in one metric is clearly behind in the other; the single exception, RanPAC on 5-task CIFAR-100, is the shortest sequence, where its fixed representation is most favorable.

Closest baselines in the three settings mentioned (baseline − L2C, points; higher Avg and lower Fgt are better).

SettingBaselineAvgFgt
CIFAR-100, 5 tasksRanPAC−0.22+0.37
CIFAR-100, 10 tasksAPER-S−0.58+1.94
RanPAC−1.13+0.90
ImageNet-R, 20 tasksAPER-S−0.79+1.64
APER-V−3.02+1.21

Mean over the nine settings of Table 1.

L2CBest baseline
Avg77.0975.95 (APER-S)
Fgt5.667.21 (APER-V)
  • L2C is the only method ranked first in both metrics in all nine settings; the runner-up changes with the setting (RanPAC, APER-S, EASE, APER-A).

  • RanPAC and APER fix the representation after the first task, which limits forgetting by construction. L2C trains prompts in every task and still forgets less.

  • On CIFAR-100, the Avg of the leading methods lies within 86–90%, which leaves little room for differences. Over the best prompt-based baseline, the margin of L2C in Avg is largest on the longest sequences of CIFAR-100 and ImageNet-A: 2.57 (CIFAR-100, 5 tasks), 6.25 (CIFAR-100, 20 tasks), and 8.57 points (ImageNet-A, 40 tasks).

(b) Final accuracy. The cases in which L2C is lower are confined to RanPAC on shorter sequences, by at most 2.04 points, and the order is reversed as tasks accumulate, by up to 11.84 points. On the longest sequence of every benchmark, L2C attains the highest ATA_{T} of all methods.

ATA_{T} of L2C minus ATA_{T} of RanPAC (points).

Benchmark (number of tasks)ShortestMiddleLongest
CIFAR-100 (5 / 10 / 20)−2.04−0.18+0.34
ImageNet-R (10 / 20 / 40)−0.53−0.48+2.36
ImageNet-A (10 / 20 / 40)+1.50+0.11+11.84

The lead of RanPAC on short sequences comes from fixing the representation after the first task, not from better continual learning: the same design loses up to 11.84 points once the sequence is long, which is the regime continual learning targets.

(c) ConvPrompt, LEAPGen, and NoRGa. Under the same training budget, NoRGa is on par with L2C on ImageNet-R and below it on ImageNet-A, with 20 times the trainable parameters (W2). Its reported numbers come from a schedule of up to 200 epochs in total per task. ConvPrompt and LEAPGen use language information, which no method in Table 1 uses.

Conclusion. L2C attains the highest Avg and the lowest Fgt in all nine settings with a pool of fixed size, without a module per task, stored class statistics, or a fixed representation, and its margin is largest on the longest sequences of CIFAR-100 and ImageNet-A. L2C is therefore a promising solution precisely where continual learning is difficult.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.