acceptodds
Under review as a conference paper at ICLR 2027

CRL-FM: Calibrated Routing for Lifelong Learning on Foundation Models

Abstract

Continual learning (CL) in natural language processing requires sequentially mastering new task distributions without eroding previously acquired capabilities. Parameter-efficient fine-tuning and prompt-based methods reduce — but do not eliminate — interference from shared parameter updates, leaving models vulnerable to representational drift as task streams grow. We propose CRL-FM (Calibrated Routing for Lifelong learning on Foundation Models), a modular CL framework built on a frozen autoregressive foundation model backbone (GPT-2), chosen to keep the framework lightweight and fully reproducible on modest compute. CRL-FM trains a disjoint Low-Rank Adaptation (LoRA) module per task and retains each in an immutable Snapshot Vault, so that later tasks cannot overwrite earlier ones by construction — yielding zero backward interference at the adapter level. At inference time, an attention-mask-aware prototype router computes normalized semantic centroids over frozen contextual embeddings to identify the target task without retaining raw exemplars, and a calibrated selective-prediction layer abstains on low-margin queries whose top-two prototype similarities are ambiguous, trading coverage for precision under a tunable threshold. On a five-task sequential benchmark (SST-2, AG News, IMDb, DBpedia-14, Yahoo Answers Topics) across four random seeds, CRL-FM's task-specific adapters attain 82.18% ± 0.22% average accuracy under known task identity, substantially outperforming Sequential Fine-Tuning (43.42% ± 4.62%) and matching or exceeding Experience Replay and L2P while incurring no adapter-level forgetting. We further characterize the end-to-end, task-agnostic setting: the prototype router achieves 78.88% top-1 routing accuracy, and we show that selective prediction recovers high reliability (98.73% routing accuracy at 21.07% coverage) by abstaining on ambiguous queries — quantifying, rather than obscuring, the accuracy–autonomy tradeoff inherent to task-free continual deployment. We position CRL-FM as a transparent, resource-efficient reference point for parameter-isolated continual learning, and discuss its relationship to routing-based PEFT methods such as SAPT and orthogonal-subspace approaches such as O-LoRA.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, catastrophic forgetting, low-rank adaptation, parameter-efficient fine-tuning, rehearsal-free continual learning, task-free continual learning, selective prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_t7tTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CRL-FM trains separate LoRA adapters on a frozen GPT-2 backbone and selects them using nearest task prototypes from frozen BERT embeddings. On five sequential text-classification tasks over four seeds, it reports zero adapter-level forgetting, 82.17% oracle accuracy, and 71.45% accuracy with routing. The paper presents this gap as a negative result, supported by routing diagnostics, a prototype-sample sweep, and baseline comparisons.

Critical strengths
  1. Separating oracle from deployed accuracy addresses an important evaluation pitfall. Reporting the substantial routing penalty, rather than relying solely on isolated-adapter retention, could provide a useful caution for continual-learning evaluations. The explicit negative-result framing is appropriate to the reported end-to-end performance.
Critical weaknesses
  1. The central routing evidence is internally inconsistent. Figure 3 reports 99% correct routing for DBpedia, while Table 4 attributes 55 of 200 examples (27.5%) to misrouted classification errors; Section 5.3 also states 72.5% correct routing. Figure 3 uses 500 examples per task, whereas the stated evaluation uses 200 per seed. These results cannot substantiate one unified error decomposition without an explanation of their different evaluation populations or configurations.
  2. The deployed-system retention and baseline comparisons are not established. Freezing adapters guarantees retention when the adapter is supplied, but adding prototypes can change routing for earlier tasks; Table 2 does not demonstrate zero end-to-end forgetting. Furthermore, Table 7 compares CRL-FM's composed accuracy with EPI's 70.8%, apparently the oracle result from Table 1, rather than a documented routed result. These protocol ambiguities undermine both the zero-forgetting claim and the purportedly comparable deployed-system ranking.
Questions for authors
  1. Results clarification: Can you reconcile Figure 3, Table 4, and Section 5.3, specifying their checkpoints, seeds, sample counts, and routing configurations, and recompute the decomposition on identical examples?
  2. Evaluation clarification: What are the routed task-by-training-stage accuracy matrices for CRL-FM and EPI? What task information and candidate-label sets do all baselines receive at inference, and does Table 7 use EPI's oracle or routed predictions?
  3. Implementation clarification: Why does the Yahoo adapter contain four times as many parameters despite the stated common rank-eight configuration? Sequence length alone does not change LoRA parameter count; please specify the actual ranks and adapted modules used in the comparisons.
3 comments

3. Response to Reviewer t7tT

We thank Reviewer t7tT for asking for consistent routing counts, stage-wise retention evidence, and clarification of the baseline protocol and Yahoo adapter size.

Weakness 1 / Question 1: Routing Counts and Evaluation Protocol

The original figure and error table used different evaluation populations; the revised manuscript now uses one fixed set of 200 examples per task for both Figure 3 and Table 4. The table in the General Response reconciles wrong-task routes with benign correct outputs and Table 4 misrouted errors. The fixed router uses M=300M=300 support examples per task and the same benchmark across seeds 42, 123, 456, and 2026. The router is deterministic, so its predictions have zero between-seed variance; adapter-training seeds affect composed classification results.

For label scoring, shared-adapter methods (Sequential FT, ER, L2P) use true-task labels; composed CRL-FM and the EPI-style baseline use predicted-task labels; IncLoRA uses oracle task identity. This is not a task-information-matched comparison, and the revised manuscript states that distinction.

Weakness 2 / Question 2: Retention Matrices and EPI Comparison

Appendix A reports the following CRL-FM task-by-stage matrix (mean over four seeds):

StageSST-2AG NewsIMDbDBpedia-14YahooMean
Stage 0 (after SST-2)88.50%————88.50%
Stage 1 (after AG News)84.00%82.50%———83.25%
Stage 2 (after IMDb)76.50%81.50%83.62%——80.54%
Stage 3 (after DBpedia)76.50%81.00%83.62%70.50%—77.91%
Stage 4 (after Yahoo)76.50%81.00%83.62%70.25%45.88%71.45%

The current study does not contain an EPI task-by-stage matrix, so we make no claim about EPI's deployed retention over the task sequence. The EPI row in the main-results table is its composed, prototype-routed result, not an oracle score.

For clarity, the reported EPI numbers come from our project's EPI-style reimplementation, not author-provided results or a reproduction validated by the original authors. It uses per-class centroids formed from training-example embeddings and nearest-centroid routing. Its 44.75% SST-2 result is composed/routed, not oracle adapter accuracy; the result alone does not establish the cause of the low score. The paper compares this implementation as an in-project baseline and does not claim it is a direct reproduction of the ACL 2023 method.

Question 3: Yahoo Adapter Capacity

The actual rank is task-specific: task-adapter methods CRL-FM, the EPI-style implementation, and IncLoRA use rank 8 for SST-2, AG News, IMDb, and DBpedia-14, and rank 32 for Yahoo; all target GPT-2's fused c_attn projection. The shared-LoRA Sequential FT and ER baselines use rank 8, while L2P has no LoRA adapter. Thus, the comparison does not have a common per-task parameter budget. Yahoo's rank-32 choice was made for the harder 10-class label space; the training configuration notes that rank 16 had plateaued near 50% in tuning. Sequence length by itself does not change LoRA parameter count, and we do not offer it as the reason for the fourfold rank difference. We removed per-method latency estimates because consistent timing logs were unavailable.

The rank-16 observation is the tuning rationale recorded in the implementation, not a rank-sensitivity result included in the paper; no rank-sweep artifact is available in the package. We therefore do not claim rank 32 is optimal.

Thanks for the responses. To be honest, although I appreciate the motivation and idea, I think the paper needs a large amount of modification and experiments with frontier models and tasks. Thus I keep my original score.

Dear Reviewer t7tT,

Thank you for your follow-up and for acknowledging the motivation and clarifications in CRL-FM_revised.pdf. We agree that evaluating larger backbones and broader task suites is an important direction for future work, and we have explicitly documented our current GPT-2 (124M) scope and memory-constrained implementation notes in Section 6 and Appendix I. Thank you again for your time and review.

Reviewer_zqMBOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses rehearsal-free continual text classification, arguing that methods sharing PEFT parameters across tasks still suffer interference. CRL-FM trains one LoRA adapter per task on a frozen GPT-2. At test time it routes each query to the adapter whose prototype, a mean of frozen BERT embeddings, is nearest in cosine similarity (Eq. (1)), and scores labels by verbalizer likelihood. Evaluation uses one ordering of a five-task stream over four seeds against five baselines, with an oracle-versus-routed error decomposition, a routing confusion analysis and a prototype-size sweep. The authors conclude that isolation removes forgetting but moves the difficulty to task identification.

Critical strengths
  1. Section 5.6 treats routed accuracy as the deployed number and concedes the method trails three baselines (Table 7), and Section 4 documents the class-blocked IMDb/DBpedia split pitfall.
  2. The oracle-versus-routed decomposition (Tables 3 and 4, Fig. 2) is the right diagnostic for parameter-isolated methods and localises error per task.
Critical weaknesses
  1. Fig. 3 shows 99.0% and 93.4% correct routing for DBpedia and AG News. Yet Table 4 attributes 27.5% and 10.5% of their examples to misrouted errors, and Section 5.3 cites Fig. 3 for a 72.5% rate it does not show. Misrouted counts are ±0.0 across seeds for four tasks despite seed-dependent subsamples.
  2. The main router is built from all training data (Section 3.2) yet underperforms the M = 25 and M = 100 settings in Table 6.
  3. Table 7 compares routed CRL-FM with EPI’s oracle number, and with IncLoRA, whose zero forgetting implies it uses task identity at test time. The paper never says whether baselines score the true task’s label set. EPI’s oracle SST-2 accuracy is below chance.
  4. Sections 5.1, 6 and 7 claim the best oracle accuracy, but Table 1 has IncLoRA ahead overall (82.45 vs 82.17) and on four of five tasks.
  5. Per-task PEFT with training-free task identification is established. EPI (ACL 2023) follows this recipe yet appears as an undefined, uncited baseline; see also SAPT, MoCL and AdapterCL. The routing-bottleneck finding restates the within-task/task-id decomposition of Kim et al. (NeurIPS 2022).
  6. The title’s calibrated routing and the abstract’s selective-prediction results are absent from the PDF; the router is a plain argmax. GPT-2, a modified MbPA++ stream, a single task order and 200 test examples per task prevent comparison with published O-LoRA or SAPT results.
  7. The paper presentation is poor, including the spacing, tables, and figures.
Questions for authors
  1. For DBpedia and AG News routing, which is correct, Fig. 3 or Table 4? Why do the misrouted counts show zero seed variance?
  2. Does each baseline receive the true task’s candidate labels at test time, and does IncLoRA use task identity? Please report Table 7 with all methods in one setting.
  3. Is EPI the ACL 2023 method? If so, what distinguishes CRL-FM from it, and why is its oracle SST-2 accuracy below 50%?
  4. Where are the abstract’s selective-prediction results, and on which split was the threshold tuned?
2 comments

4. Response to Reviewer zqMB

We thank Reviewer zqMB for raising questions about the evaluation population, router support, baseline protocols, EPI, and presentation.

Weakness 1 / Question 1: Routing Results and Seed Variance

The corrected Figure 3/Table 4 evaluation uses the same fixed final-checkpoint set of 200 queries per task (1,000 total), not the earlier 500-per-task population. Final routing accuracy is SST-2 64.0%, AG News 86.0%, IMDb 99.0%, DBpedia 72.5%, and Yahoo 80.0%. Thus, the old 93.4% AG News and 99% DBpedia figures are superseded: 99% is IMDb, while DBpedia has 55 wrong routes out of 200. The fixed BERT encoder, prototypes, and queries make routes identical across seeds; adapter seeds change classification outcomes. The General Response table reconciles routes, benign outputs, and misrouted errors.

Weakness 2: Support Set

Each prototype uses a fixed sample of M=300M=300 training examples, shared across seeds. The MM sweep uses one support draw and is descriptive; it does not establish a statistically validated or universally optimal support size.

Weakness 3 / Question 4: Selective Prediction

Selective results are included in the abstract and paper. The threshold τ∗=0.03608\tau^*=0.03608 is calibrated on 1,250 examples and evaluated on 750 independent examples; the other 500 in the 2,500-example pool are unused. Test accuracy is 96.36% on accepted queries (440/750; 58.67% coverage; approximate 95% CI ±1.7\pm1.7 points). Counting abstentions as errors gives 56.53% over all 750 queries.

Weakness 4: Oracle Comparison

CRL-FM's oracle-task accuracy is 82.17% ±\pm 0.22%, versus IncLoRA's 82.45% ±\pm 0.65%. We say “matches within one standard deviation,” not “best” or “outperforms.” CRL-FM's autonomous composed accuracy is 71.45% ±\pm 0.58%.

Weakness 5 / Question 3: EPI

EPI results are from our EPI-style implementation, not author-provided results or an author-validated ACL 2023 reproduction. It trains isolated adapters and routes by nearest per-class BERT training-embedding centroids. Its 70.85% aggregate and 44.75% SST-2 result are composed/routed, not oracle adapter accuracy. A separate 79.12% ±\pm 3.35% oracle SST-2 centroid diagnostic is not the trained EPI model and does not explain that result. No EPI task-by-stage matrix was retained.

Question 2: Baseline Label-Set Access

Sequential FT, ER, and L2P use the true task to select candidate labels; IncLoRA receives oracle task identity. CRL-FM and EPI-style inference select a task and label set without true task identity. The primary comparison is therefore not information-matched. The supplementary 32-class joint-label run retrained ER and L2P only; it is not a unified rerun of all baselines.

Weakness 6: Prior Work, Scope, and Title

The revision cites EPI, AdapterCL, SAPT, MoCL, and prior task-identification error decomposition. We do not claim novel parameter isolation, a new general-purpose router, or unmeasured latency/asymptotic advantages. The deployed router is nearest-prototype argmax; confidence-margin thresholding is an optional abstention layer. Our empirical contribution is the routed-error decomposition, deployed-retention matrix, and selective-prediction analysis. The study is limited to one fixed-order English stream (SST-2, AG News, IMDb, DBpedia-14, Yahoo), GPT-2 124M, four seeds, and 200 primary test queries per task. We do not claim direct comparability with non-harmonized O-LoRA/SAPT results or reproduction of other benchmark variants. Query-order permutation does not test different training orders. The title's calibration wording refers to selective abstention; any title change would be considered for camera-ready only, subject to ICLR policy.

Weakness 7: Presentation

The revision moves the Figure 1 legend, adds the deployed M=300M=300 point, marks the selective operating point, clarifies label-set and deployed-retention tables, removes unsupported latency claims, and identifies the AI-use/citation-responsibility statement.

Thanks for the response. The revision fixes the Fig. 3/Table 4 inconsistency, states each baseline’s label-set protocol, corrects the oracle claims, cites EPI and other prior work, and adds the selective-prediction results. Three concerns remain. The comparison is still not information-matched, so the claim that isolation trails replay and prompting is unsupported. The shift from the original to the revised Fig. 3 (SST-2 24.6% vs 64.0%) is too large for sampling, and selective prediction still uses the original pool. Several results, including EPI’s, were relabelled with unchanged numbers. I raise my rating to 2 for your efforts during the rebuttal but cannot support acceptance of this paper.

Reviewer_rydtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual text classification using independent LoRA adapters on a frozen GPT -2 backbone and a frozen BERT prototype router. Across five tasks and four seeds, it reports 82.17% oracle accuracy, 71.45% composed accuracy, and zero adapter forgetting. Routing confusion, error decomposition, and a prototype sample sweep examine the performance loss from task identification.

Critical strengths
  1. The isolated adapters prevent parameter interference between tasks. Table 2 confirms unchanged accuracy when each task uses its own adapter.
  2. The paper reports both oracle and composed accuracy, explicitly acknowledging the performance loss when task identity must be inferred.
Critical weaknesses
  1. Figure 3 reports 99% routing accuracy for DBpedia, while Table 4 records 55 misrouted errors per 200 examples. Without an explanation of the differing evaluation populations, the central routing analysis is unreliable.

  2. Frozen adapters preserve individual classifiers, while adding task prototypes can redirect earlier queries. Table 2 does not show whether composed accuracy is preserved, so zero forgetting remains unverified under the actual router.

  3. The methodological contribution is limited to combining isolated LoRA adapters with a frozen nearest prototype router. No new routing or calibration mechanism is introduced, and the analysis provides limited insight beyond documenting failures of this particular combination.

Questions for authors
  1. Why does Figure 3 report 99% routing accuracy for DBpedia while Table 4 records 55 misrouted errors per 200 examples?

  2. How does each earlier task's composed accuracy change as new task prototypes are added?

2 comments

2. Response to Reviewer rydt

We thank Reviewer rydt for highlighting the routing discrepancy, the scope of the forgetting claim, and the need to identify the methodological contribution precisely.

Weakness 1: Figure 3 and Table 4 Routing/Error Counts

The original Figure 3 and Table 4 used different evaluation populations: the original figure used a larger 2,500-query pool, whereas Table 4 used 200 queries per task. In the revised manuscript, both use the same fixed 1,000-query benchmark (200 per task). Figure 3 now reports 80.30% mean routing accuracy, with per-task accuracies of 64.0% (SST-2), 86.0% (AG News), 99.0% (IMDb), 72.5% (DBpedia), and 80.0% (Yahoo).

Table 4 counts a wrong-task route as a misrouted error only if the resulting label is incorrect. The reconciliation table above gives the per-task route counts and benign outcomes. For AG News, 28 routes are to the wrong task, seven yield a correct label, and 21 are misrouted classification errors. The label scorer uses only the predicted task's label set and never receives the true task identity.

Weakness 2: Zero Forgetting vs. Deployed Retention

“Zero forgetting” refers to adapter-level parameter retention: task adapters are isolated and frozen after their task is trained. It does not imply constant deployed accuracy as the router's prototype set grows. Appendix A reports CRL-FM's task-by-stage deployed accuracy:

StageSST-2AG NewsIMDbDBpedia-14YahooMean
Stage 0 (after SST-2)88.50%————88.50%
Stage 1 (after AG News)84.00%82.50%———83.25%
Stage 2 (after IMDb)76.50%81.50%83.62%——80.54%
Stage 3 (after DBpedia)76.50%81.00%83.62%70.50%—77.91%
Stage 4 (after Yahoo)76.50%81.00%83.62%70.25%45.88%71.45%

Thus, adding prototypes can reduce earlier-task deployed accuracy even though the adapters themselves do not forget.

Weakness 3: Contribution Beyond a New Routing Algorithm

We do not claim a new general-purpose router. The revised paper makes the empirical contribution more explicit through (i) a task-identity-free label-set protocol, (ii) a quantitative decomposition of routed errors versus adapter errors, (iii) a deployed task-by-stage retention matrix, and (iv) confidence-margin selective prediction. On a separate held-out pool, thresholding at τ∗=0.03608\tau^*=0.03608 yields 96.36% accuracy among accepted queries at 58.67% coverage. These analyses characterize the routing bottleneck and its coverage trade-off; they do not establish that the underlying nearest-prototype router is novel.

Questions for Authors

  1. Why does Figure 3 report 99% routing accuracy for DBpedia while Table 4 records 55 misrouted errors per 200 examples?
    The original figures used different evaluation populations. The revised manuscript uses the same fixed final-checkpoint set of 200 queries per task for both Figure 3 and Table 4. The 99% value is actually the IMDb routing accuracy, not DBpedia; DBpedia's routing accuracy is 72.5%, with 55 wrong-task routes and 55 corresponding misrouted classification errors. The reconciliation in the General Response gives the full task-level counts.

  2. How does adding new task prototypes affect earlier task accuracy?
    Adding prototypes can redirect earlier-task queries and reduce deployed accuracy even though the adapters remain isolated and frozen. Appendix A reports the resulting task-by-stage matrix; it shows, for example, SST-2 deployed accuracy falling from 88.50% after Stage 0 to 76.50% after later tasks are added, while adapter-level SST-2 retention remains unchanged. This deployed-system result is distinct from the “zero forgetting” claim.

Dear Reviewer rydt,

Thank you again for your constructive review and for engaging with our revision (CRL-FM_revised_2.pdf). As discussion concludes, we want to briefly highlight how the updated manuscript addresses each of your points:

  1. Reconciled Routing & Error Counts: Figure 3 and Table 4 now evaluate the exact same fixed 1,000-query benchmark (N=200N=200 per task across seeds 42, 123, 456, and 2026), with benign cross-task sentiment overlaps accounted for (Section 5.3 and Appendix B).
  2. Deployed Retention Matrix (RijR_{ij}): Appendix A (Table 8) separates zero adapter-level parameter forgetting (0.00 pp0.00\text{ pp}) from deployed system accuracy across stages, showing how prototype competition stabilizes after Stage 2.
  3. Contribution & Information-Matched Context: We explicitly scope our contribution around error decomposition, deployed retention, and confidence-margin selective prediction (τ∗=0.03608\tau^* = 0.03608, 96.36% accepted accuracy at 58.67% coverage), and include a supplementary 32-class joint-label evaluation in Appendix F (Table 13).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Comment

We thank reviewers rydt, t7tT, and zqMB for their careful feedback. The revised manuscript clarifies the evaluation populations, label-set protocols, router support size, adapter-level versus deployed-system retention, and limitations of the baseline comparisons. Figure 3 and Table 4 use the same fixed 1,000-query benchmark.

Evaluation and Error-Count Reconciliation

The router uses a fixed BERT encoder and M=300M=300 training-support examples per task. Figure 3 reports routing on the fixed held-out benchmark of 200 queries per task (1,000 total) at the final five-task checkpoint (after Yahoo/Stage 4); its predictions are deterministic and identical across the four adapter-training seeds (42, 123, 456, 2026). Table 4 averages composed classification outcomes over those four seeds at the same final checkpoint. A “benign” route selects the wrong task but still predicts the correct label string under that task's label set.

True taskWrong-task routes (Fig. 3)Benign correct outputsTable 4 misrouted errors
SST-27239.232.8
AG News28721
IMDb202
DBpedia-1455055
Yahoo Topics40040

Wrong-task route totals are read from the fixed router matrix; benign outputs and Table 4 errors are averaged over the four adapter-training seeds. The seven benign AG News outputs are reported in aggregate; we do not assign them to a particular destination task.

Other Clarifications in This Revision

  • Sequential FT, ER, and L2P score the true task's label set; composed CRL-FM and EPI-style routing use the predicted task's label set; IncLoRA receives oracle task identity.
  • The selective-prediction threshold is fitted on 1,250 calibration examples and evaluated on 750 held-out examples. The remaining 500 examples in the 2,500-example pool were not used for threshold selection, evaluation, or reported results.
  • CRL-FM's task-by-stage matrix reports deployed accuracy separately from zero adapter-level forgetting. The paper's oracle comparison says CRL-FM matches IncLoRA within one standard deviation (82.17% ± 0.22% vs. 82.45% ± 0.65%).
  • The paper reports no per-method latency because consistently measured timing logs are unavailable.
  • The four seeds are 42, 123, 456, and 2026. Saved result summaries and generated table captions use the corrected fourth-seed identifier; the underlying metric values are unchanged.
  • The cascade is exploratory and uses a separate subset: 50 examples per task for gate calibration and the following 200 per task for evaluation. Its task-plus-label accuracy and post-cascade routing accuracy are not paired comparisons with the primary benchmark. The reported 74.02% label-only value ignores task identity; 84.50% is routing accuracy after the cascade, not Stage 1 routing accuracy.
  • The fixed-model query-order check permutes inference batches over already-trained, frozen adapters; it does not test retraining under different task-arrival orders. The cascade's historical latency estimates are inconsistent (+5.85 ms/query in one seed-42 record, +0.90 ms/query in a later paired summary, and +2.26 ± 2.53 ms/query in the aggregate), so no reliable latency claim is made.
AuthorsOpenReview ↗

Follow-up Supplementary Analyses and Implementation Update

5. Follow-up Supplementary Analyses and Implementation Update

The following work was completed after the manuscript corrections above. These supplementary protocols are distinct from the primary benchmark and do not replace or retroactively alter the main-table results.

Task-Qualified Joint-Label Evaluation

ER and L2P were newly trained and evaluated with 32 task-qualified classes over 1,000 held-out examples per seed (200 per task), using seeds 42, 123, 456, and 2026. ER achieved 72.98%±1.54%72.98\% \pm 1.54\% with oracle-task label restriction and 64.50%±7.11%64.50\% \pm 7.11\% when all 32 labels were scored jointly. L2P achieved 78.45%±1.48%78.45\% \pm 1.48\% and 22.95%±1.17%22.95\% \pm 1.17\%, respectively.

These are not post-hoc conversions of the published baseline values. The joint-label ER runner uses one epoch per task and a 200-example per-task replay budget; the joint-label L2P runner uses two epochs per task. The historical ER and L2P figures were produced by their own task-incremental baseline recipes. Thus, this supplementary rerun demonstrates the measured outcomes under the new task-qualified protocol but does not isolate label-set access as the sole cause of the difference from the historical scores.

Centroid and Prototype Geometry Diagnostic

Across four seeds, EPI-style nearest-centroid accuracy over 32 classes was 65.65%±0.61%65.65\% \pm 0.61\%, implied task routing was 85.18%±0.34%85.18\% \pm 0.34\%, and CRL-FM-style task-prototype routing was 84.28%±0.96%84.28\% \pm 0.96\%. Restricting the EPI-style centroid comparison to SST-2 yielded 79.12%±3.35%79.12\% \pm 3.35\% oracle nearest-centroid accuracy. The SST-2 and IMDb positive/negative centroid cosine similarities were 0.98797±0.000670.98797 \pm 0.00067 and 0.99446±0.000390.99446 \pm 0.00039.

This is a representation-geometry diagnostic, not a retrained EPI baseline or the exact deployed CRL-FM router evaluation. High centroid cosine similarity alone does not establish classification collapse, and these results do not explain the historical EPI SST-2 result of 44.75%44.75\%. No causal explanation is claimed.

Replay-Buffer Sweep

The sweep has four measured seeds for B=0,50,100,250,500B=0,50,100,250,500, with mean final accuracy respectively 45.25%±1.66%45.25\% \pm 1.66\%, 69.42%±3.55%69.42\% \pm 3.55\%, 73.10%±1.27%73.10\% \pm 1.27\%, 75.83%±1.31%75.83\% \pm 1.31\%, and 76.47%±1.46%76.47\% \pm 1.46\%. The corresponding average forgetting values are 41.31±1.7841.31 \pm 1.78, 11.78±3.8011.78 \pm 3.80, 5.81±2.085.81 \pm 2.08, 3.62±2.173.62 \pm 2.17, and 2.46±1.032.46 \pm 1.03 percentage points. For B=1000B=1000, only seed 42 is measured (78.10%78.10\% accuracy; 1.751.75 pp forgetting); the values previously forecast for the other three seeds remain projections and are not included in measured aggregates.

All sweep runs use up to 3,000 training examples per task and a fixed replay-batch fraction of 0.5. This runner uses one epoch per task, whereas the primary ER recipe states four epochs per task. The sweep is therefore exploratory and is not a direct reproduction of the primary ER baseline. Storage values are compact UTF-8/JSON record-size estimates, not measured system memory. A zero replay budget means no examples are retained between task stages; it does not mean raw examples are never read or temporarily processed.

Backbone Scaling and Memory Handling

The GPT-2 Medium benchmark implementation was updated for memory-constrained execution: four-example micro-batches with four-step gradient accumulation preserve an effective batch size of 16; CUDA mixed precision is enabled; completed task models are offloaded to CPU; evaluation moves one task model to the accelerator at a time; and the BERT routing model is loaded after adapter training. These changes are implementation measures only. Following the authors' decision, GPT-2 Medium was not run and no larger-backbone results or scaling claims are reported. The paper's measured backbone remains GPT-2 (124M).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.