- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents an empirical audit of a widely used five-task text classification continual-learning (CL) benchmark for language models, quantifying the extent to which small reported improvements can be confounded by task order, seed-dependent retraining variability, and evaluation sampling. Using a shared-recipe experimental harness across multiple adapter-style methods and three backbones, the authors analyze within-cohort agreement of pairwise method differences, test whether such agreement transfers to fresh cohorts, probe execution-environment sensitivity, and dissect order effects, including a routed-adapter case study. The main findings are that pooled rankings are stable whereas close pairwise comparisons are fragile, order effects can change both magnitude and direction of differences, and seed/evaluation variance and execution details can meaningfully alter conclusions; the paper also releases an executable audit workflow that reproduces analyses, uncertainty estimates, and reporting decisions.
Critical strengths
1.This article is an experimental analysis type of paper. From the perspective of experimental design, the paper adopts a prospective validation approach (including testing with new seeds and testing in a fixed host environment); it provides a detailed analysis of the impact of different sequences on test results; additionally, a comprehensive process has been established to ensure the reliability of the results. The scale and coverage of the experiment are relatively adequate
2.The statistical report is relatively cautious and auditable. The paper employs analyses such as paired differences, permutation tests, Holm multiple comparison correction, Greenhouse-Geisser correction, and interval reporting, while repeatedly distinguishing between 'failure to reject differences' and 'demonstrating equivalence.' The authors also proactively disclose adverse results, implementation biases, non-functional endpoints, post hoc analyses, and issues with missing historical records. The accompanying raw matrices, analysis scripts, and complete examples from input to reported conclusions offer a high degree of auditability.
3.Provides practical guidance and recommendations for the analysis of research results in the relevant field.
Critical weaknesses
Experimental limitations or concerns. As an experimental article, I still have concerns about the following experimental setup: 1.The 95% sign-agreement threshold, group sizes, and tie handling are heuristic; the framework would benefit from a formal variance-components or hierarchical model that yields interpretable uncertainty on effect sizes and minimal detectable differences. 2.Baseline comparisons are not entirely equivalent or faithful. All methods share the same set of hyperparameters, thus reflecting the 'implementation under the same training recipe' rather than the best performance of each method after reasonable tuning. 3.Fixed-host study covers only one order (o2) and three seeds per host; results show failures of the agreement criterion but remain underpowered to characterize host effects or scorer equivalence generally.
From the perspective of thesis writing: the number of experiments is relatively large, but there is a lack of unified modeling. The paper conducts a substantial amount of pairwise tests, permutation tests, and sensitivity analyses, yet the overall analysis is rather fragmented. There are interactions and repeated measurement structures among task order, seeds, methods, models, and hosts, which are more appropriately addressed using hierarchical models or mixed-effects models to estimate the variance contributed by each factor and their interactions in a unified manner. Although the current multiple-group tests are detailed, it is difficult to form a concise, transferable framework for resolution estimation.
Limited Contribution Scalability. The paper provides a prototype for continual learning experiment auditing and offers practical guidance and recommendations. However, there is no standardized, universal benchmark reliability toolkit to advance the field.
Questions for authors
1.What is the core conclusion that the paper seeks to promote? Is it that 'the five-task classification benchmark cannot stably distinguish minor differences,' or that 'language model continual learning experiments generally cannot reliably distinguish minor differences'? If it is the latter, are the existing tasks, model scales, and learning scenarios sufficient to support this generalization?
2.For the fixed-host study, why was only order o2 chosen, and do you anticipate the host effects or agreement failures to generalize across other orders? Could you extend to a small 2×2×m factorial (hosts × orders × seeds) for variance decomposition?
3.Can you justify the choice of the 95% sign-agreement threshold and group sizes from a power or decision-theoretic perspective? Would a hierarchical mixed-effects model (seed nested in order, method as fixed effect) better quantify minimal detectable effects and replication probabilities?
4.Since seeds conflate data sampling and initialization, have you run a targeted split where training initializations vary but evaluation sets are held fixed across seeds (and vice versa) at larger scale to quantify their separate contributions?