- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CRL-FM trains separate LoRA adapters on a frozen GPT-2 backbone and selects them using nearest task prototypes from frozen BERT embeddings. On five sequential text-classification tasks over four seeds, it reports zero adapter-level forgetting, 82.17% oracle accuracy, and 71.45% accuracy with routing. The paper presents this gap as a negative result, supported by routing diagnostics, a prototype-sample sweep, and baseline comparisons.
Critical strengths
- Separating oracle from deployed accuracy addresses an important evaluation pitfall. Reporting the substantial routing penalty, rather than relying solely on isolated-adapter retention, could provide a useful caution for continual-learning evaluations. The explicit negative-result framing is appropriate to the reported end-to-end performance.
Critical weaknesses
- The central routing evidence is internally inconsistent. Figure 3 reports 99% correct routing for DBpedia, while Table 4 attributes 55 of 200 examples (27.5%) to misrouted classification errors; Section 5.3 also states 72.5% correct routing. Figure 3 uses 500 examples per task, whereas the stated evaluation uses 200 per seed. These results cannot substantiate one unified error decomposition without an explanation of their different evaluation populations or configurations.
- The deployed-system retention and baseline comparisons are not established. Freezing adapters guarantees retention when the adapter is supplied, but adding prototypes can change routing for earlier tasks; Table 2 does not demonstrate zero end-to-end forgetting. Furthermore, Table 7 compares CRL-FM's composed accuracy with EPI's 70.8%, apparently the oracle result from Table 1, rather than a documented routed result. These protocol ambiguities undermine both the zero-forgetting claim and the purportedly comparable deployed-system ranking.
Questions for authors
- Results clarification: Can you reconcile Figure 3, Table 4, and Section 5.3, specifying their checkpoints, seeds, sample counts, and routing configurations, and recompute the decomposition on identical examples?
- Evaluation clarification: What are the routed task-by-training-stage accuracy matrices for CRL-FM and EPI? What task information and candidate-label sets do all baselines receive at inference, and does Table 7 use EPI's oracle or routed predictions?
- Implementation clarification: Why does the Yahoo adapter contain four times as many parameters despite the stated common rank-eight configuration? Sequence length alone does not change LoRA parameter count; please specify the actual ranks and adapted modules used in the comparisons.
3 comments
Thanks for the responses. To be honest, although I appreciate the motivation and idea, I think the paper needs a large amount of modification and experiments with frontier models and tasks. Thus I keep my original score.
Dear Reviewer t7tT,
Thank you for your follow-up and for acknowledging the motivation and clarifications in CRL-FM_revised.pdf. We agree that evaluating larger backbones and broader task suites is an important direction for future work, and we have explicitly documented our current GPT-2 (124M) scope and memory-constrained implementation notes in Section 6 and Appendix I. Thank you again for your time and review.
3. Response to Reviewer t7tT
We thank Reviewer t7tT for asking for consistent routing counts, stage-wise retention evidence, and clarification of the baseline protocol and Yahoo adapter size.
Weakness 1 / Question 1: Routing Counts and Evaluation Protocol
The original figure and error table used different evaluation populations; the revised manuscript now uses one fixed set of 200 examples per task for both Figure 3 and Table 4. The table in the General Response reconciles wrong-task routes with benign correct outputs and Table 4 misrouted errors. The fixed router uses M=300 support examples per task and the same benchmark across seeds 42, 123, 456, and 2026. The router is deterministic, so its predictions have zero between-seed variance; adapter-training seeds affect composed classification results.
For label scoring, shared-adapter methods (Sequential FT, ER, L2P) use true-task labels; composed CRL-FM and the EPI-style baseline use predicted-task labels; IncLoRA uses oracle task identity. This is not a task-information-matched comparison, and the revised manuscript states that distinction.
Weakness 2 / Question 2: Retention Matrices and EPI Comparison
Appendix A reports the following CRL-FM task-by-stage matrix (mean over four seeds):
The current study does not contain an EPI task-by-stage matrix, so we make no claim about EPI's deployed retention over the task sequence. The EPI row in the main-results table is its composed, prototype-routed result, not an oracle score.
For clarity, the reported EPI numbers come from our project's EPI-style reimplementation, not author-provided results or a reproduction validated by the original authors. It uses per-class centroids formed from training-example embeddings and nearest-centroid routing. Its 44.75% SST-2 result is composed/routed, not oracle adapter accuracy; the result alone does not establish the cause of the low score. The paper compares this implementation as an in-project baseline and does not claim it is a direct reproduction of the ACL 2023 method.
Question 3: Yahoo Adapter Capacity
The actual rank is task-specific: task-adapter methods CRL-FM, the EPI-style implementation, and IncLoRA use rank 8 for SST-2, AG News, IMDb, and DBpedia-14, and rank 32 for Yahoo; all target GPT-2's fused
c_attnprojection. The shared-LoRA Sequential FT and ER baselines use rank 8, while L2P has no LoRA adapter. Thus, the comparison does not have a common per-task parameter budget. Yahoo's rank-32 choice was made for the harder 10-class label space; the training configuration notes that rank 16 had plateaued near 50% in tuning. Sequence length by itself does not change LoRA parameter count, and we do not offer it as the reason for the fourfold rank difference. We removed per-method latency estimates because consistent timing logs were unavailable.The rank-16 observation is the tuning rationale recorded in the implementation, not a rank-sensitivity result included in the paper; no rank-sweep artifact is available in the package. We therefore do not claim rank 32 is optimal.