acceptodds
Under review as a conference paper at ICLR 2027

RESOLUTION LIMITS OF A CONTINUAL-LEARNING BENCHMARK FOR LANGUAGE MODELS

Abstract

Small changes in continual learning are difficult to reliably distinguish from score variation due to task order, retraining seed and evaluation sampling. We quantify comparison reliability in a five-task classification benchmark with three language model backbones, trained under the same adaptation procedure. Across changes within the order, pooled rankings remain consistent, but magnitude and direction changes. A frozen fresh-seed run continuation verifies development agrees transfers or not. At five seeds per disjoint group, 13 out of 16 continue to meet a criterion of 95% sign agreement across seeds, splitting this holdout into replay vs non-replay comparisons gives us 9/9 and 4/7 agreeing comparisons respectively. This endpoint concerns agreement within each cohort, not preservation of the old winner or a replication probability. In a separate experiment-60 trajectories study, we find examples of close non-replay comparisons that change within fixed execution environments. Mixing hosts is thus not the only source of variation. Conditioned on fixed trained models, jointly resampling evaluation datasets across all training and testing stages leads to a median evaluation-sampling standard deviation of 0.0119, versus 0.0158 when resampling at each stage independently. We provide as part of the audit an executable that will reproduce these paired effects, uncertainty, declarations, abstentions and the observed cost of selective reporting. Reported near-superiority on new runs cannot be certified without assessing each method’s difference and each reporting decision. These results hold for this research, but do not establish universal seed requirement.

Primary area:
datasets and benchmarks
Keywords:
continual learning, large language models, benchmark reliability, model evaluation, task-order sensitivity, seed variability, statistical uncertainty, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uVVXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an empirical audit of a widely used five-task text classification continual-learning (CL) benchmark for language models, quantifying the extent to which small reported improvements can be confounded by task order, seed-dependent retraining variability, and evaluation sampling. Using a shared-recipe experimental harness across multiple adapter-style methods and three backbones, the authors analyze within-cohort agreement of pairwise method differences, test whether such agreement transfers to fresh cohorts, probe execution-environment sensitivity, and dissect order effects, including a routed-adapter case study. The main findings are that pooled rankings are stable whereas close pairwise comparisons are fragile, order effects can change both magnitude and direction of differences, and seed/evaluation variance and execution details can meaningfully alter conclusions; the paper also releases an executable audit workflow that reproduces analyses, uncertainty estimates, and reporting decisions.

Critical strengths

1.This article is an experimental analysis type of paper. From the perspective of experimental design, the paper adopts a prospective validation approach (including testing with new seeds and testing in a fixed host environment); it provides a detailed analysis of the impact of different sequences on test results; additionally, a comprehensive process has been established to ensure the reliability of the results. The scale and coverage of the experiment are relatively adequate

2.The statistical report is relatively cautious and auditable. The paper employs analyses such as paired differences, permutation tests, Holm multiple comparison correction, Greenhouse-Geisser correction, and interval reporting, while repeatedly distinguishing between 'failure to reject differences' and 'demonstrating equivalence.' The authors also proactively disclose adverse results, implementation biases, non-functional endpoints, post hoc analyses, and issues with missing historical records. The accompanying raw matrices, analysis scripts, and complete examples from input to reported conclusions offer a high degree of auditability.

3.Provides practical guidance and recommendations for the analysis of research results in the relevant field.

Critical weaknesses

Experimental limitations or concerns. As an experimental article, I still have concerns about the following experimental setup: 1.The 95% sign-agreement threshold, group sizes, and tie handling are heuristic; the framework would benefit from a formal variance-components or hierarchical model that yields interpretable uncertainty on effect sizes and minimal detectable differences. 2.Baseline comparisons are not entirely equivalent or faithful. All methods share the same set of hyperparameters, thus reflecting the 'implementation under the same training recipe' rather than the best performance of each method after reasonable tuning. 3.Fixed-host study covers only one order (o2) and three seeds per host; results show failures of the agreement criterion but remain underpowered to characterize host effects or scorer equivalence generally.

From the perspective of thesis writing: the number of experiments is relatively large, but there is a lack of unified modeling. The paper conducts a substantial amount of pairwise tests, permutation tests, and sensitivity analyses, yet the overall analysis is rather fragmented. There are interactions and repeated measurement structures among task order, seeds, methods, models, and hosts, which are more appropriately addressed using hierarchical models or mixed-effects models to estimate the variance contributed by each factor and their interactions in a unified manner. Although the current multiple-group tests are detailed, it is difficult to form a concise, transferable framework for resolution estimation.

Limited Contribution Scalability. The paper provides a prototype for continual learning experiment auditing and offers practical guidance and recommendations. However, there is no standardized, universal benchmark reliability toolkit to advance the field.

Questions for authors

1.What is the core conclusion that the paper seeks to promote? Is it that 'the five-task classification benchmark cannot stably distinguish minor differences,' or that 'language model continual learning experiments generally cannot reliably distinguish minor differences'? If it is the latter, are the existing tasks, model scales, and learning scenarios sufficient to support this generalization?

2.For the fixed-host study, why was only order o2 chosen, and do you anticipate the host effects or agreement failures to generalize across other orders? Could you extend to a small 2×2×m factorial (hosts × orders × seeds) for variance decomposition?

3.Can you justify the choice of the 95% sign-agreement threshold and group sizes from a power or decision-theoretic perspective? Would a hierarchical mixed-effects model (seed nested in order, method as fixed effect) better quantify minimal detectable effects and replication probabilities?

4.Since seeds conflate data sampling and initialization, have you run a targeted split where training initializations vary but evaluation sets are held fixed across seeds (and vice versa) at larger scale to quantify their separate contributions?

Reviewer_W3MTOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper empirically studies the resolution of a five-task text-classification continual-learning (CL) benchmark: whether small between-method improvements can be reliably distinguished from random variation due to task order, training seed, evaluation sampling, and execution environment.

Critical strengths

None.

Critical weaknesses

W1: Limited novelty. The paper is a purely empirical audit of an existing benchmark: it proposes no new method, metric, estimator, or theoretical framework. The contribution amounts to a set of cautionary measurements, and it is unclear what conceptual advance the community gains beyond "close comparisons on this benchmark are noisy".

  • W2: The writing is poor. The abstract and introduction are very hard to parse—key terms (cohort, arm, trajectory, qualification) are used before being defined, and the research question and motivation do not become clear until deep into the paper. Many readers will not be able to state what the paper asks or finds after reading the first page.

  • W3: The main conclusions and implications for future research are unclear. Despite the extensive experiments and detailed appendices, the paper never states what practitioners should do differently: how many seeds are enough, which effect sizes are resolvable, whether published small-margin claims on O-LoRA-style streams should be considered suspect, or what a better evaluation protocol would look like.

Questions for authors

Q1: Beyond the empirical audit, what is the paper's conceptual contribution? Is there a general principle, model, or diagnostic here that transfers to other benchmarks and scales, or are the findings specific to this five-task classification setup under one shared recipe?

Q2: What is your concrete recommendation to researchers evaluating on this benchmark?

Reviewer_d4aFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how random seed, task order, and test set re-sampling affect method comparisons in a continual-learning benchmark adapted to evaluate task forgetting. The paper shows that small differences in forgetting across methods which meet a 10-seed 95% agreement criterion may not hold up on 10 new seeds. The paper then shows that task order can change each method’s measured forgetting and, in a case study of two variants of the authors' own method, which variant forgets less.

Critical strengths

Assessing the rigor of evaluation practices in continual learning is an important research direction.

Critical weaknesses

The paper's clarity is very poor. The abstract and introduction are impossible to parse without first reading the main text. Several sentences in the abstract are ungrammatical. Figure 1 is cited in the introduction, but its caption relies on terms defined only on pages 3 and 4. Key terms such as the "frozen agreement criterion", "qualified" comparisons and "development" vs. "fresh" cohorts are used before being defined.

The paper does not study the benchmark as it is typically used in prior work. First, prior work primarily reports final average accuracy (e.g., MbPA++, IDBR, LFPT5, O-LoRA, N-LoRA), rather than the forgetting metric used in the paper. Second, the authors use 250 test examples per task for their main results, whereas prior work typically uses 7,600 per task (e.g., MbPA++, LAMOL, IDBR), roughly quadrupling evaluation noise according to the authors' own estimate. Apart from one example (O-LoRA's Table 2), the paper does not identify prior claims that its findings call into question. It is unclear why the headline results rely on the split-half 95% agreement criterion rather than paired differences with confidence intervals, as used in Section 6 and Fig. 4. It is unsurprising that comparisons whose differences are small relative to seed variance fail to hold-up on new seeds.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.