acceptodds
Under review as a conference paper at ICLR 2027

Continual Learning Exam

Abstract

Large language model agents are increasingly designed for continual learning beyond parameter updates, acquiring and updating knowledge from sequential experience while retaining useful prior knowledge. Current evaluations offer limited insight into this capability, as they largely assess individual task performance rather than improvement through accumulated experience. We introduce Continual Learning Exam (CLE), a benchmark developed by 50+ domain experts for measuring experience-driven improvement in realistic professional workflows. CLE comprises 120 human-authored tasks, each organized as a five-level ladder, for a total of 600 levels spanning 89 professional roles across 12 occupational domains. Agents must pass each level to unlock the next, with later levels building on experience accumulated during earlier assignments. Each submission is assessed by an evaluation agent against human-authored rubrics. We compare a single submission per level with a setting that also provides diagnostic feedback on unmet requirements after failure, allowing the agent to reflect on the feedback and resubmit once. Across 17 harness-model configurations, this feedback and revision opportunity yields a median completion gain of 5.3 percentage points. Yet even the best-performing system clears only 47.7% of levels and completes all five levels in just 23.3% of tasks, highlighting continual learning as a key challenge for current LLM agents.

Primary area:
datasets and benchmarks
Keywords:
Large Language Model, Large Language Model Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VFTvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Continual Learning Exam (CLE), a benchmark for evaluating the continual-learning ability of LLM agents in realistic human–AI cowork settings. CLE consists of 120 human-authored, five-level task ladders (600 levels) built by 50+ domain experts across 12 occupational domains. Levels are gated — an agent must pass each level to unlock the next — and later levels reuse knowledge from earlier ones along five dimensions: Tools & Skills, Data Dependency, Strategy, User Context, and Domain Knowledge. The environment ships with 23,981 real-world input files (77 file types) and 19 stateful mock applications exposing 1,024 APIs. Submissions are scored against 16,125 human-authored, human-verified rubric criteria by a multi-turn evaluation agent. The authors evaluate 17 configurations spanning 4 agent harnesses and 11 backbone models in two trials (K=1 single submission; K=2 one revision after diagnostic feedback). The experimental results show that even the best system (Claude Code with Opus 5) completes only 47.67% of levels, indicating continual learning as a key challenge for current LLM agents.

Critical strengths

Strength 1: A genuinely novel evaluation construct for agent continual learning.

Continual Learning Exam (CLE) operationalizes "learning from accumulated experience" in a more realistic and comprehensive way than prior agent benchmarks. It introduces gated five-level ladders with cross-level reuse of tools, data, strategy, user context, and domain knowledge — including stateful mock applications whose state persists across levels.

Strength 2: Substantial scale, realism, and rubric discipline of the benchmark.

The benchmark artifact is a substantive contribution. Its tasks are coherent professional workflows authored by 50+ domain experts across 12 occupational families rather than synthesized, supported by 23,981 real-world input files and 19 stateful mock applications (1,024 APIs), and graded against 16,125 human-verified rubric items, each specifying a unique expected outcome and its verification evidence.

Strength 3: A systematically comprehensive quantitative evaluation with well-designed metrics.

CLE benchmarks a broad set of frontier systems — 11 backbone model variants paired with 4 mainstream agent harnesses (17 configurations in total). The metrics cover effectiveness metrics (overall, per-level, and full-task completion, plus a separate Trial-1/Trial-2 measurement of diagnostic-feedback gains) and efficiency metrics (actor tokens, cumulative wall-clock time, and end-to-end USD cost). Joint reporting of these dimensions exposes accuracy–efficiency trade-offs that effectiveness-only leaderboards cannot capture.

Critical weaknesses

Weakness 1: Claims of poor continual-learning capability lack failure analysis or case studies.

The paper reports that even frontier configurations struggle — the best (Claude Code with Opus 5) completes only 42.17%/47.67% of levels in Trial 1/2, 23.33% at L5 — and concludes that continual learning is a key bottleneck for current LLM agents. However, it provides no attribution of why agents fail: no error taxonomy, no breakdown of failures by knowledge form (tools, data dependency, strategy, user context, domain knowledge) or by level, and no qualitative case studies showing where and how failures arise. Consequently, the benchmark quantifies the capability gap but cannot diagnose its causes, which limits its value for guiding continual-learning research.

Weakness 2: Absence of variance estimates or repeated runs for the agent under test undermines the reliability of comparative claims.

The experimental results are based on a single run per configuration and trial. However, LLM agents are known to exhibit substantial variance due to sampling temperature, non-deterministic tool responses, and API-level variability. Without repeated runs, readers cannot know whether the differences across configurations are statistically significant or within run-to-run noise.

Weakness 3: The "continual learning" framing does not distinguish experience-driven gains from cumulative task difficulty.

The paper claims that CLE "measures experience-driven improvement" and that "later levels build on knowledge and experience accumulated during earlier assignments." However, since the main experiment runs every task as a full sequential ladder, it cannot attribute the poor performance to a specific cause: agents may fail because they cannot learn from accumulated experience, or simply because the chained tasks are difficult. A straightforward control would separate the two — for example, running later levels in isolation, with predecessor artifacts provided but without the agent having performed the prior work. The block-based evaluation mechanism (Section 3.1) already supports this comparison, yet no such experiment is reported. Without it, the central claim that CLE measures continual learning, rather than cumulative task difficulty, remains unsubstantiated by the experiments.

Questions for authors

Please refer to the Weaknesses.

Reviewer_1RfLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CLE, a benchmark for continual learning of LLM agents. It has 120 expert-written tasks, each with five levels, where later levels are supposed to depend on knowledge gained in earlier ones. Agents work in stateful mock apps and are graded by an evaluation agent against human-written checklists. The paper tests 17 harness–model setups under two settings: one submission per level, or one extra submission after diagnostic feedback.

Critical strengths

The benchmark is a large and careful engineering effort. Tasks are written by domain experts with real files, go through agent and human review, and run in stateful environments.

Critical weaknesses
  1. When we want to measure continue learning, we should disentangle the effect of difficulty and make tasks connected so that experience can be shared. The results do not show that later levels are harder because of failed learning. The main evidence is that cumulative pass rates drop from L1 to L5 (median 62.5% to 8.33%). But this drop is expected from gating alone: if every level had about a 60% pass rate, passing all five would give about 8%. The paper’s own conditional pass rates (Figure 11) actually go up at later levels (median 55% to 80% in Trial 2). So the data does not show “early levels are fine, later levels fail.”

  2. There is no analysis linking failures to earlier levels. To claim the benchmark measures continual learning, the paper needs to show that agents fail at later levels because they do not use what they learned before. It doesn’t. There is no comparison of the same level with and without prior experience (the block setup could do this directly), no check of whether the failed checklist items are the ones that depend on earlier levels, and no results broken down by the five knowledge types. Without this, low scores could simply mean the tasks are hard. Also, the main reported gain (K=2) is retrying within the same level after feedback, which is self-correction, not learning across tasks.

Questions for authors
  1. If you reset the agent’s memory at each level but still provide the needed files from earlier levels, how much does performance drop? This gap is what the benchmark actually measures as learning.
  2. At later levels, are the failed checklist items mostly ones that depend on earlier levels, or ones that only involve the current level?
Reviewer_16fKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the continual learning exam, a benchmark of 120 professional-workflow task ladders in which 5 sequential levels are connected through reusable artifacts, information, and procedures. The proposed benchmark provides stateful applications and rubric-based agentic grading, and evaluates 17 model-harness configurations under one vs. two submission attempts with diagnostic feedback.

Critical strengths
  • This paper proposed a substantial, executable benchmark for long-horizon professional workflows.
  • The grading design is concrete and receives a meaningful validation check.
Critical weaknesses
  • The reported K=1K=1 vs. K=2K=2 comparison primarily measures the value of immediate diagnostic feedback and an additional attempt, not continual transfer. So what is observed is one revision step rather than an iterative learning curve. It does not establish that the agent retained what it learned and applied it to later levels or later tasks.

  • There is no level design guideline. LkL_k is only an ordinal within a task, yet it is aggregated across twelve domains in Tab. 1. The five transferable-knowledge types are never counted, annotated, or analysed. Fig. 8 also shows that 32 of 120 ladders reference no earlier-level file at any level, leaving the claimed experience dependence unspecified for a quarter of the benchmark.

Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.