- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Continual Learning Exam (CLE), a benchmark for evaluating the continual-learning ability of LLM agents in realistic human–AI cowork settings. CLE consists of 120 human-authored, five-level task ladders (600 levels) built by 50+ domain experts across 12 occupational domains. Levels are gated — an agent must pass each level to unlock the next — and later levels reuse knowledge from earlier ones along five dimensions: Tools & Skills, Data Dependency, Strategy, User Context, and Domain Knowledge. The environment ships with 23,981 real-world input files (77 file types) and 19 stateful mock applications exposing 1,024 APIs. Submissions are scored against 16,125 human-authored, human-verified rubric criteria by a multi-turn evaluation agent. The authors evaluate 17 configurations spanning 4 agent harnesses and 11 backbone models in two trials (K=1 single submission; K=2 one revision after diagnostic feedback). The experimental results show that even the best system (Claude Code with Opus 5) completes only 47.67% of levels, indicating continual learning as a key challenge for current LLM agents.
Critical strengths
Strength 1: A genuinely novel evaluation construct for agent continual learning.
Continual Learning Exam (CLE) operationalizes "learning from accumulated experience" in a more realistic and comprehensive way than prior agent benchmarks. It introduces gated five-level ladders with cross-level reuse of tools, data, strategy, user context, and domain knowledge — including stateful mock applications whose state persists across levels.
Strength 2: Substantial scale, realism, and rubric discipline of the benchmark.
The benchmark artifact is a substantive contribution. Its tasks are coherent professional workflows authored by 50+ domain experts across 12 occupational families rather than synthesized, supported by 23,981 real-world input files and 19 stateful mock applications (1,024 APIs), and graded against 16,125 human-verified rubric items, each specifying a unique expected outcome and its verification evidence.
Strength 3: A systematically comprehensive quantitative evaluation with well-designed metrics.
CLE benchmarks a broad set of frontier systems — 11 backbone model variants paired with 4 mainstream agent harnesses (17 configurations in total). The metrics cover effectiveness metrics (overall, per-level, and full-task completion, plus a separate Trial-1/Trial-2 measurement of diagnostic-feedback gains) and efficiency metrics (actor tokens, cumulative wall-clock time, and end-to-end USD cost). Joint reporting of these dimensions exposes accuracy–efficiency trade-offs that effectiveness-only leaderboards cannot capture.
Critical weaknesses
Weakness 1: Claims of poor continual-learning capability lack failure analysis or case studies.
The paper reports that even frontier configurations struggle — the best (Claude Code with Opus 5) completes only 42.17%/47.67% of levels in Trial 1/2, 23.33% at L5 — and concludes that continual learning is a key bottleneck for current LLM agents. However, it provides no attribution of why agents fail: no error taxonomy, no breakdown of failures by knowledge form (tools, data dependency, strategy, user context, domain knowledge) or by level, and no qualitative case studies showing where and how failures arise. Consequently, the benchmark quantifies the capability gap but cannot diagnose its causes, which limits its value for guiding continual-learning research.
Weakness 2: Absence of variance estimates or repeated runs for the agent under test undermines the reliability of comparative claims.
The experimental results are based on a single run per configuration and trial. However, LLM agents are known to exhibit substantial variance due to sampling temperature, non-deterministic tool responses, and API-level variability. Without repeated runs, readers cannot know whether the differences across configurations are statistically significant or within run-to-run noise.
Weakness 3: The "continual learning" framing does not distinguish experience-driven gains from cumulative task difficulty.
The paper claims that CLE "measures experience-driven improvement" and that "later levels build on knowledge and experience accumulated during earlier assignments." However, since the main experiment runs every task as a full sequential ladder, it cannot attribute the poor performance to a specific cause: agents may fail because they cannot learn from accumulated experience, or simply because the chained tasks are difficult. A straightforward control would separate the two — for example, running later levels in isolation, with predecessor artifacts provided but without the agent having performed the prior work. The block-based evaluation mechanism (Section 3.1) already supports this comparison, yet no such experiment is reported. Without it, the central claim that CLE measures continual learning, rather than cumulative task difficulty, remains unsubstantiated by the experiments.
Questions for authors
Please refer to the Weaknesses.