acceptodds
Under review as a conference paper at ICLR 2027

Can Large Language Model Agents Learn by Watching Tutorial Videos?

Abstract

Humans continuously acquire knowledge from carefully crafted materials such as books and tutorial videos, carrying that understanding into future tasks. Millions of such instructional videos are uploaded online every day, yet LLM agents remain unable to exploit this ever-expanding wealth of human teaching. Their capabilities are frozen after training, while parameter updates at scale remain prohibitively expensive. This raises the central question: can LLM agents learn from this vast instructional resource without parameter updates? We introduce Tutorial Agentic Learning (TAL), a nonparametric continual-learning paradigm that consolidates tutorial videos into a persistent, structured tutorialbook memory through two-stage adversarial consolidation. At inference time, TAL retrieves through a Table-of-Contents (TOC) tree, tying retrieval directly to the model's reasoning rather than to embedding similarity. To study this setting, we introduce VidLearn, a multimodal QA benchmark built from 72 chess tutorial videos spanning 12 openings. On VidLearn, TAL achieves up to 113% and 91% relative gains on the training and test splits, improves as more videos are consumed, and yields stronger opening positions in 69% of head-to-head games. It also outperforms the evaluated retrieval and memory baselines, while a separate software-tutorial evaluation demonstrates cross-domain generalization. Together, these results highlight instructional videos as a scalable resource for parameter-update-free continual agent learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Large Language Model Agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YYVNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether large language model (LLM) agents can continuously acquire knowledge from tutorial videos without updating model parameters, and proposes Tutorial Agentic Learning (TAL). TAL organizes video content into a structured and continually updated external “tutorialbook” memory. It first generates and filters difficult questions that the model cannot answer independently in order to identify knowledge gaps, and then iteratively refines the tutorialbook based on these failures. At inference time, TAL uses TOC-based retrieval to select relevant chapters for answering queries. The authors introduce VidLearn, a benchmark consisting of 72 chess tutorial videos, and evaluate TAL across multiple mainstream models against Transcript-RAG, Video-RAG, mem0, and ReasoningBank. TAL achieves up to 71.8% accuracy on the held-out chess test set, outperforming the strongest baseline at approximately 64.5%, and also shows cross-domain improvements on Photoshop tutorials. Overall, the paper demonstrates that frozen LLMs can improve downstream task performance by continually constructing and refining structured external memory.

Critical strengths

Strength 1: Important Problem and Coherent Method Design

The paper addresses the practically relevant problem of enabling LLM agents to continuously acquire knowledge from tutorial videos without parameter updates. TAL realizes this through a coherent loop of knowledge-gap identification, structured memory refinement, and inference-time retrieval. In particular, the hard-question harvesting mechanism selectively targets knowledge that the model currently lacks, giving the overall framework a clear motivation and some methodological novelty.

Strength 2: Relatively Comprehensive Experimental Evaluation

The paper compares TAL not only with Transcript-RAG, but also with Video-RAG, mem0, and ReasoningBank, and further includes multi-model evaluation, cross-domain experiments, visual ablations, and retrieval ablations. With 48 videos, full TAL achieves 71.8% held-out accuracy compared with 62.1% for transcript-only TAL, supporting the contribution of visual information. The authors also examine potential train-test leakage. Overall, the experimental evaluation is reasonably comprehensive and provides solid support for the effectiveness of the proposed system.

Critical weaknesses

Weakness 1: Conceptual Gap Between the Learning Framing and the Actual Evidence

The paper frames TAL as enabling LLM agents to “learn by watching tutorial videos,” yet the underlying model remains frozen throughout, and all newly acquired knowledge is stored in an external tutorialbook that remains accessible at test time. Therefore, the current experiments more directly demonstrate that better external memory improves downstream performance, rather than that the model itself acquires persistent new capabilities. Since this issue directly concerns the paper’s central research question and contribution framing, it is my primary concern.

Weakness 2: Lack of a Strong Compute-Matched Structured-Memory Baseline

TAL requires substantial additional LLM computation during memory construction, including question generation, grounding judgment, chapter rewriting, and repeated verification, whereas the main RAG baselines use much lighter preprocessing. Without a compute-matched structured-memory baseline using a comparable LLM-call/token budget, it is difficult to determine whether the gains come from TAL-specific components such as hard-question harvesting, adversarial consolidation, and TOC retrieval, or mainly from the larger preprocessing budget. This weakens the evidence for the claimed superiority of TAL’s specific mechanisms.

Questions for authors

Q1. How do the authors define learning in this paper?

The authors treat the tutorialbook as part of the agent, so updating external memory can be interpreted as nonparametric agent learning. However, since the underlying LLM remains frozen and the performance gains continue to depend on inference-time access to the tutorialbook, I would like the authors to clarify: what is the essential distinction between learning and constructing and consulting an external knowledge resource in this setting? Which experimental results support knowledge acquisition rather than simply improved access to external knowledge?

Q2. To what extent can TAL’s performance gains be explained by additional LLM compute?

TAL requires substantial LLM computation for each video, including question generation, judging, memory rewriting, and verification, whereas the preprocessing used by the main RAG baselines is much lighter. Could the authors provide a compute-matched structured-memory baseline that uses a comparable LLM-call/token budget to summarize, deduplicate, and organize the tutorial videos into structured chapters, but without TAL’s hard-question adversarial loop, while using the same or a similar retrieval mechanism? Such a comparison would help determine whether the gains primarily come from TAL’s proposed mechanisms or from the larger preprocessing budget.

6 comments
AuthorsOpenReview ↗

Author Rebuttal (Part1)

Thank you for your constructive feedback and for recognizing the significance of our work and its empirical results. We address your concerns below.

W1 / Q1. Definition of Agent Learning and Supporting Evidence.

Thank you for this valuable question. Our research question concerns the complete LLM agent rather than the underlying LLM alone. In TAL, the agent consists of a frozen LLM and an agent harness that manages persistent memory, retrieval, and context construction.

We define agent learning as acquiring knowledge from instructional materials, retaining it in the agent’s persistent memory, and using it to improve performance on subsequent tasks. Formally, we represent the agent as At=(fθ,Ht)A_t=(f_\theta,\mathcal{H}_t), where fθf_\theta is the frozen model and Ht\mathcal{H}_t is the agent harness, whose persistent memory is instantiated as the tutorialbook BtB_t in TAL. The model parameters remain fixed, while BtB_t evolves as the agent learns from instructional materials. This formulation is consistent with prior work on nonparametric agent learning, including Reflexion [1], ExpeL [2], and ReasoningBank [3].

TAL’s distinction from conventional knowledge-base construction and retrieval lies in its learner-dependent diagnostic feedback, which guides what knowledge should be acquired and refined. Specifically, TAL autonomously identifies the agent’s knowledge gaps through tutorial-grounded diagnostic questions (Stage I) and uses these signals to guide memory refinement (Stage II). The acquired knowledge is subsequently reused through TOC retrieval. In this agent-level formulation, inference-time access to the tutorialbook is how the agent utilizes its acquired knowledge, rather than an alternative to learning.

We provide three complementary results supporting this learning formulation.

First, the stage ablation shows that knowledge-gap-guided refinement improves held-out accuracy from 56.36% to 71.82% (+15.46 pp), with both settings using external memory under the same retrieval framework. With the backbone model fixed, accuracy increases from 48.18% to 64.55% as the tutorial corpus grows from 12 to 48 videos, while downstream chess gameplay reduces mean centipawn loss from 162.81 to 125.74. Our structured-memory comparison under comparable computational budgets further examines whether knowledge organization and retrieval alone explain TAL’s gains, as detailed in W2/Q2.

Second, we evaluate cross-opening knowledge generalization through a leave-one-opening-out experiment. For each fold, we construct the tutorialbook using videos and diagnostic questions only from the other eleven openings. We then evaluate only the held-out questions corresponding to the excluded opening, covering all 110 test questions across the twelve folds at each video budget. As the learning corpus grows from 11 to 44 tutorials, accuracy increases from 59.09% to 64.85%, exceeding Vanilla (57.6%) by 7.25 pp at the largest budget. These results provide further evidence that TAL can generalize beyond the specific knowledge covered during memory construction.

Table 1: Leave-one-opening-out evaluation: held-out accuracy (%). Columns indicate tutorials per available opening. Gemini 3.5 Flash backbone; temperature =0.7; mean±SD over 3 seeds.

Setting1234
Vanilla57.6 ± 2.357.6 ± 2.357.6 ± 2.357.6 ± 2.3
TAL (target opening excluded)59.09 ± 1.2960.61 ± 2.3961.52 ± 1.1364.85 ± 2.27
TAL (target opening included)64.5 ± 2.064.8 ± 1.166.1 ± 0.471.8 ± 2.0

Third, we further evaluate TAL on Continual Learning Bench (CL Bench) [4], covering all six tasks in the official task environments. In this text-only setting, task descriptions, interaction trajectories, and environment feedback serve as instructional materials, replacing the video-based inputs while preserving TAL’s two-stage knowledge consolidation and TOC retrieval mechanisms. Using Claude Sonnet 4.6 as its backbone, TAL achieves an aggregate normalized reward of 0.3535 and gain of 0.3813 under CL Bench’s official evaluation protocol, ranking first among the 13 compared systems (Table 2) and outperforming the strongest competing system with the same backbone by 0.1304 and 0.1269, respectively.

Table 2: CL Bench results across six tasks (mean ± 95% CI). Baseline results are from the original CL Bench paper [4].

SystemModelAggregate reward ↑Aggregate gain ↑
TALClaude Sonnet 4.60.3535 ± 0.06720.3813 ± 0.0684
ICLClaude Sonnet 4.60.2231 ± 0.04100.2544 ± 0.0360
ICLGPT-5.40.2006 ± 0.09100.2006 ± 0.0910
Claude CodeClaude Sonnet 4.60.1901 ± 0.07100.2390 ± 0.0570
Mem0GPT-5.40.1505 ± 0.06400.2024 ± 0.0590

Together, these results support our formulation of nonparametric agent learning. We will clarify this formulation in the revised manuscript.

AuthorsOpenReview ↗

Author Rebuttal (Part2)

W2 / Q2. Structured-Memory Baseline Under Comparable Computational Budgets.

Thank you for this valuable suggestion. To examine whether additional LLM computation explains TAL’s performance gains, we introduce an Iterative RAG baseline that constructs structured tutorial chapters from the same video frames and full transcripts through extraction, summarization, and deduplication. Without TAL’s diagnostic QA harvesting, it iteratively refines these chapters by identifying omissions and factual inconsistencies and verifying revisions against the original tutorials.

Both methods use Gemini 3.5 Flash for memory construction and question answering, with the same top-3 TOC retrieval procedure. We evaluate four- and eight-round Iterative RAG checkpoints across 12, 24, 36, and 48 videos on the same 174 training and 110 held-out questions. For Iterative RAG, the training questions are used only for evaluation and do not guide memory refinement. To assess computational comparability, we report both construction-token usage and average per-query inference-token usage, including input and output tokens for retrieval and answer generation.

Table 3: Construction and inference token usage and training/held-out accuracy of TAL and Iterative RAG. Gemini 3.5 Flash backbone; temperature =0.7; accuracy reported as mean±SD over 3 seeds.

VideosMethodConstruction budget: Total tokens (M)Construction budget: Ratio to TALInference / query: Avg. total (K)Inference / query: Ratio to TALAccuracy (%) ↑: TrainAccuracy (%) ↑: Test
12Iterative RAG (4 rounds)4.3510.895×7.8841.296×60.0±0.753.9±1.7
12Iterative RAG (8 rounds)6.9521.431×7.7951.281×60.0±1.254.2±0.9
12TAL4.8591.000×6.0861.000×68.2±1.264.5±2.0
24Iterative RAG (4 rounds)8.8200.878×11.1101.154×64.8±0.557.6±1.1
24Iterative RAG (8 rounds)14.1421.408×11.3231.176×65.3±1.855.2±1.1
24TAL10.0471.000×9.6321.000×70.7±1.764.8±1.1
36Iterative RAG (4 rounds)14.2950.858×14.9031.065×66.1±0.960.0±1.3
36Iterative RAG (8 rounds)23.1431.390×15.0311.074×68.6±0.360.3±1.9
36TAL16.6551.000×13.9901.000×77.8±1.266.1±0.4
48Iterative RAG (4 rounds)18.4850.866×17.2610.981×64.6±1.452.4±2.4
48Iterative RAG (8 rounds)30.0361.406×17.4670.993×67.4±1.056.7±0.9
48TAL21.3571.000×17.5941.000×84.1±1.471.8±2.0

TAL achieves higher mean accuracy than both Iterative RAG variants across all four video budgets on both training and held-out questions. At 48 videos, TAL achieves 71.8% held-out accuracy, compared with 52.4% and 56.7% for four- and eight-round Iterative RAG, respectively. Notably, the eight-round variant uses 1.406× TAL’s construction-token budget with nearly identical per-query inference usage (17.467K vs. 17.594K tokens), yet remains 15.1 percentage points below TAL.

These results suggest that structured knowledge construction and iterative refinement without TAL’s knowledge-gap-driven learning mechanism did not match TAL’s performance in our evaluation, even with greater construction budgets and similar inference usage. We will include this comparison in the revised manuscript.

References.

[1] N. Shinn et al. Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS, 2023.

[2] A. Zhao et al. ExpeL: LLM Agents Are Experiential Learners. AAAI, 2024.

[3] S. Ouyang et al. ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory. ICLR, 2026.

[4] P. Asawa et al. Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments. arXiv:2606.05661, 2026.

Dear Reviewer YYVN,

Thank you very much for your time and effort in reviewing our paper. Your constructive comments have been very helpful in improving the quality of our work.

As the author–reviewer discussion period is approaching its end, we would be grateful to hear whether our responses have addressed your concerns or if any questions remain. We would be happy to provide further clarification or additional evidence if needed.

Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_VgiKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Tutorial Agentic Learning (TAL), a parameter-update-free framework that converts tutorial videos into structured, persistent memory through two-stage adversarial consolidation and Table-of-Contents (TOC) retrieval, and introduces VidLearn to evaluate knowledge acquisition from instructional videos.

Critical strengths
  1. Learning reusable knowledge from instructional videos without parameter updates is a practically relevant research direction, and the proposed tutorialbook memory provides an interesting approach to this problem.
  2. TAL consistently outperforms retrieval and memory baselines across multiple LLM backbones, with additional evidence from chess gameplay and software tutorials. The performance improvements are substantial, particularly as more videos are incorporated.
  3. The two-stage consolidation process and hierarchical retrieval mechanism are intuitive. The ablations on consolidation stages, visual information, and retrieval strategies provide useful evidence for the effectiveness of the main components.
Critical weaknesses
  1. Although TAL is presented as a new nonparametric continual-learning paradigm, its core mechanism is to extract information from videos, organize it into external memory, and retrieve relevant content at inference time. It remains unclear what fundamentally distinguishes this approach from iterative knowledge-base construction and retrieval-augmented generation, beyond the specific memory organization and refinement procedure.
  2. VidLearn contains only 72 chess tutorials covering 12 openings, and the held-out questions are derived from different videos covering the same openings. Therefore, strong performance may primarily reflect improved retrieval and consolidation of opening-specific knowledge rather than the acquisition of transferable reasoning capabilities. The additional software-tutorial evaluation helps, but remains limited in scale.
  3. In Stage II, the same harvested question bank is used to identify memory deficiencies, guide memory revisions, and determine whether revisions should be accepted. This directly optimizes memory against a fixed set of questions, making it difficult to distinguish genuine knowledge consolidation from question-specific adaptation. Although held-out evaluations demonstrate downstream improvements, the paper does not sufficiently isolate the contribution of generalizable knowledge from optimization toward the harvested questions.
Questions for authors
  1. How does TAL fundamentally differ from an iterative RAG system with LLM-generated knowledge summaries and feedback-driven memory refinement? Would a baseline that constructs a structured knowledge base directly from the same video evidence, without adversarial QA harvesting, achieve comparable performance under matched retrieval and inference budgets?
  2. Can TAL demonstrate generalization beyond the specific concepts used during memory consolidation? For example, how does it perform on questions requiring novel combinations of concepts across multiple tutorials, or on previously unseen chess openings, rather than different videos covering the same openings?
3 comments
AuthorsOpenReview ↗

Author Rebuttal (Part1)

Thank you for the careful and constructive review. We are grateful for your recognition of the practical relevance of our research direction, the intuitive design of TAL, and the strong empirical results. Below, we address your questions and concerns through additional experiments and clarifications.

W1 / Q1. Distinction from Iterative RAG and Comparison under Comparable Budgets.

Thank you for this valuable suggestion. TAL aims to enable agents to autonomously acquire and continually accumulate knowledge from human instructional materials. While TAL shares structured external memory and retrieval with iterative RAG, its key distinction lies in learner-dependent diagnostic feedback that guides knowledge acquisition and refinement. Specifically, TAL generates tutorial-grounded diagnostic questions and identifies those that the frozen solver cannot answer correctly without memory. These failures reveal agent-specific knowledge gaps that guide tutorialbook refinement. In Stage II, TAL evaluates the solver with the current tutorialbook and uses the remaining failures to refine relevant chapters. Unlike source-based refinement, which focuses on factual consistency and knowledge coverage, TAL uses the solver’s performance on diagnostic questions as an additional feedback signal for memory refinement.

Following your suggestion, we introduce an Iterative RAG baseline that constructs structured knowledge from the same video frames and full transcripts through extraction, summarization, and deduplication. Without TAL’s diagnostic QA harvesting, it iteratively refines the generated chapters using source-based feedback to identify omissions and factual inconsistencies, verifying revisions against the original tutorials.

We evaluate four- and eight-round refinement checkpoints across 12, 24, 36, and 48 videos under construction-token budgets comparable to TAL’s. Both methods use Gemini 3.5 Flash for memory construction and question answering, with the same top-3 TOC retrieval procedure. Evaluation uses the same 174 training and 110 held-out questions. For Iterative RAG, the training questions are used only for evaluation and do not guide memory refinement. To assess computational comparability, we report construction-token usage and average per-query inference-token usage, including input and output tokens for retrieval and answer generation.

Table 1: Construction and inference token usage and training/held-out accuracy of TAL and Iterative RAG. Gemini 3.5 Flash backbone; temperature =0.7; accuracy reported as mean±SD over 3 seeds.

VideosMethodConstruction budget: Total tokens (M)Construction budget: Ratio to TALInference / query: Avg. total (K)Inference / query: Ratio to TALAccuracy (%) ↑: TrainAccuracy (%) ↑: Test
12Iterative RAG (4 rounds)4.3510.895×7.8841.296×60.0±0.753.9±1.7
12Iterative RAG (8 rounds)6.9521.431×7.7951.281×60.0±1.254.2±0.9
12TAL4.8591.000×6.0861.000×68.2±1.264.5±2.0
24Iterative RAG (4 rounds)8.8200.878×11.1101.154×64.8±0.557.6±1.1
24Iterative RAG (8 rounds)14.1421.408×11.3231.176×65.3±1.855.2±1.1
24TAL10.0471.000×9.6321.000×70.7±1.764.8±1.1
36Iterative RAG (4 rounds)14.2950.858×14.9031.065×66.1±0.960.0±1.3
36Iterative RAG (8 rounds)23.1431.390×15.0311.074×68.6±0.360.3±1.9
36TAL16.6551.000×13.9901.000×77.8±1.266.1±0.4
48Iterative RAG (4 rounds)18.4850.866×17.2610.981×64.6±1.452.4±2.4
48Iterative RAG (8 rounds)30.0361.406×17.4670.993×67.4±1.056.7±0.9
48TAL21.3571.000×17.5941.000×84.1±1.471.8±2.0

TAL outperforms both Iterative RAG variants across all four video budgets. At 48 videos, TAL achieves 71.8% held-out accuracy, compared with 52.4% and 56.7% for four- and eight-round Iterative RAG, respectively. Notably, the eight-round variant consumes 1.406× as many construction tokens as TAL with nearly identical inference usage (17.467K vs. 17.594K tokens per query), yet remains 15.1 percentage points below TAL. These results suggest that iterative structured-memory refinement without TAL’s diagnostic QA harvesting does not achieve comparable performance, even with greater construction budgets and similar inference usage. We will include this comparison in the revised manuscript.

AuthorsOpenReview ↗

Author Rebuttal (Part2)

W2 / Q2. Generalization Beyond the Concepts Used in Memory Consolidation.

Thank you for raising this important question. Following your suggestion, we conduct a leave-one-opening-out evaluation to assess generalization to openings excluded from memory construction. Each fold constructs the tutorialbook using videos and diagnostic questions only from the other eleven openings. We repeat this procedure across all twelve openings, evaluating only the held-out questions corresponding to each excluded opening, thereby covering all 110 test questions across the folds at each video budget.

Table 2 compares cross-opening transfer with the original same-opening TAL and Vanilla baselines. Cross-opening accuracy increases from 59.09% to 64.85% as the learning corpus grows from 11 to 44 tutorials drawn from the other eleven openings. At the largest budget, TAL outperforms Vanilla (57.6%) by 7.25 percentage points, despite having no learning videos or diagnostic supervision from the target opening. These results provide evidence that TAL can generalize beyond the opening-specific knowledge used during memory consolidation.

Table 2: Leave-one-opening-out evaluation: held-out accuracy (%). Columns indicate tutorials per available opening. Gemini 3.5 Flash backbone; temperature =0.7; mean±SD over 3 seeds.

Setting1234
Vanilla57.6 ± 2.357.6 ± 2.357.6 ± 2.357.6 ± 2.3
TAL (target opening excluded)59.09 ± 1.2960.61 ± 2.3961.52 ± 1.1364.85 ± 2.27
TAL (target opening included)64.5 ± 2.064.8 ± 1.166.1 ± 0.471.8 ± 2.0

Beyond cross-opening transfer, we evaluate TAL on all six tasks of Continual Learning Bench (CL Bench) [1] using Claude Sonnet 4.6. In this text-only setting, task descriptions, interaction trajectories, and environment feedback serve as instructional materials, replacing the video-based inputs while preserving TAL’s two-stage memory consolidation and TOC retrieval mechanisms. Following the official evaluation protocol, we report aggregate normalized reward and gain across the six tasks. Table 3 presents the five highest-ranked systems by aggregate normalized reward.

Table 3: CL Bench results across six tasks (mean ± 95% CI). Baseline results are from the original CL Bench paper [1].

SystemModelAggregate reward ↑Aggregate gain ↑
TALClaude Sonnet 4.60.3535 ± 0.06720.3813 ± 0.0684
ICLClaude Sonnet 4.60.2231 ± 0.04100.2544 ± 0.0360
ICLGPT-5.40.2006 ± 0.09100.2006 ± 0.0910
Claude CodeClaude Sonnet 4.60.1901 ± 0.07100.2390 ± 0.0570
Mem0GPT-5.40.1505 ± 0.06400.2024 ± 0.0590

TAL ranks first among the 13 compared systems in both aggregate normalized reward (0.3535) and gain (0.3813), outperforming the strongest competing system, ICL with the same Claude Sonnet 4.6 backbone, by 0.1304 and 0.1269, respectively.

Together, these experiments provide complementary evidence that TAL can generalize beyond the specific opening knowledge used during memory consolidation and that its learning framework is applicable to broader sequential learning environments. We will include both experiments and their findings in the revised manuscript.

W3. Self-Generated Diagnostic Supervision and Generalization Beyond the Question Bank.

Thank you for raising this important concern. TAL’s QA harvesting provides self-generated diagnostic training supervision rather than externally predefined training questions. In Stage I, TAL generates tutorial-grounded questions, verifies their answers against source evidence, and retains those the frozen solver cannot answer correctly without memory. These questions identify agent-specific knowledge gaps that Stage II uses to guide and evaluate memory revisions.

Importantly, while Stage II reuses the harvested question bank during memory refinement, generalization is evaluated independently of these questions. The tutorialbook is fixed before evaluation on 110 held-out questions from 24 unseen videos, which do not participate in memory construction, refinement, or revision acceptance. Our stage ablation shows that QA-guided refinement improves held-out accuracy from 56.36% for the initial tutorialbook to 71.82% for full TAL (+15.46 pp). This demonstrates that the benefits of diagnostic supervision extend beyond the questions used during memory optimization.

The cross-opening and CL Bench evaluations detailed in W2/Q2, together with downstream chess gameplay, further support the generalizability of TAL’s acquired knowledge beyond the harvested question bank. We will clarify the distinction between self-generated training supervision and independent generalization evaluation in the revised manuscript.

References.

[1] P. Asawa et al. Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments. arXiv:2606.05661, 2026.

Dear Reviewer VgiK,

Thank you very much for your time and effort in reviewing our paper. Your constructive comments have been very helpful in improving the quality of our work, and your positive feedback has meant a lot to us.

As the author–reviewer discussion period is approaching its end, we would be grateful to hear whether our responses have addressed your concerns or if any questions remain. We would be happy to provide further clarification or additional evidence if needed.

Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_Q7LBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Tutorial Agentic Learning (TAL), a continual learning framework that enables frozen LLM agents to accumulate knowledge from instructional videos through external memory. Experiments on chess and Photoshop tutorials demonstrate improvements over retrieval and memory baselines, with evidence of cross-model memory transfer.

Critical strengths
  1. An interesting, human-inspired continual learning paradigm that improves frozen models through structured external memory.

  2. Strong empirical gains, with useful ablations and evidence of memory transfer across models.

Critical weaknesses
  1. Evaluation is limited to chess and Photoshop, leaving broader generalization unclear.

  2. The video-specific contribution needs clearer justification: the workflow relies heavily on subtitles and sparse frames, with limited temporal modeling. Looks like the pipeline can apply to pure-text scenario as well.

Questions for authors
  1. Were web search and external tools disabled during QA evaluation?

  2. Could TAL be evaluated on a broader sequential benchmark, such as Continual Learning Bench (https://continual-learning-bench.com/)?

  3. What are the full construction and per-query costs using the models in the main experiments?

3 comments
AuthorsOpenReview ↗

Author Rebuttal (Part1)

Thank you for the thoughtful and constructive review, and for recognizing TAL’s coherent framework design and strong empirical results. We appreciate your valuable feedback and address your questions and concerns through additional experiments and clarifications below.

W1 / Q2. Broader Generalization and Sequential Evaluation on Continual Learning Bench.

Following your suggestion, we evaluate TAL on all six tasks of Continual Learning Bench (CL Bench) [1] to assess knowledge accumulation and reuse across sequential task instances.

We use Claude Sonnet 4.6 as TAL’s backbone and evaluate it in the official CL Bench task environments. In this text-only setting, task descriptions, interaction trajectories, and environment feedback serve as instructional materials, replacing the video-based inputs while preserving TAL’s two-stage memory consolidation and TOC retrieval mechanisms. Stage I identifies knowledge gaps from these materials through diagnostic QA harvesting, and Stage II uses the resulting feedback to guide targeted revisions of the corresponding tutorialbook chapters.

Following CL Bench’s official evaluation protocol, we report aggregate normalized reward and gain, capturing overall performance and improvements relative to stateless baselines, respectively. Table 1 presents the top five systems ranked by aggregate normalized reward.

Table 1: CL Bench results across six tasks (mean ± 95% CI). Baseline results are from the original CL Bench paper [1].

SystemModelAggregate reward ↑Aggregate gain ↑
TALClaude Sonnet 4.60.3535 ± 0.06720.3813 ± 0.0684
ICLClaude Sonnet 4.60.2231 ± 0.04100.2544 ± 0.0360
ICLGPT-5.40.2006 ± 0.09100.2006 ± 0.0910
Claude CodeClaude Sonnet 4.60.1901 ± 0.07100.2390 ± 0.0570
Mem0GPT-5.40.1505 ± 0.06400.2024 ± 0.0590

TAL ranks first among the 13 compared systems in both aggregate normalized reward (0.3535) and gain (0.3813), outperforming the strongest competing system, ICL with Claude Sonnet 4.6, by 0.1304 and 0.1269, respectively. These results provide further evidence of the generalizability of TAL’s learning framework to broader sequential learning environments beyond instructional-video QA. We will include this additional experiment in the revised manuscript.

W2. Video-Specific Contribution.

Thank you for highlighting this point. TAL’s video agent jointly analyzes sampled frames and subtitles. Through timestamp alignment, each frame is paired with its surrounding subtitle window (± 10 s), linking the lecturer’s explanation to the demonstrated visual state and providing complementary evidence for tutorialbook construction.

To examine the contribution of visual information beyond textual narration, we compare full TAL with an otherwise matched transcript-only variant on the same 110 held-out questions. The transcript-only variant retains TAL’s two-stage consolidation and retrieval pipeline but excludes visual inputs during tutorialbook construction.

Table 2: Vision-in-construction ablation: held-out accuracy (%) on 110 questions. Columns indicate learning videos. Gemini 3.5 Flash answer model; temperature =0.7; mean±SD over three seeds.

Memory built from12243648
Vanilla (no memory)57.6 ± 2.357.6 ± 2.357.6 ± 2.357.6 ± 2.3
Transcript-only TAL (no visual input)58.8 ± 1.458.5 ± 2.360.9 ± 5.562.1 ± 1.9
Full TAL (frames + transcripts)64.5 ± 2.064.8 ± 1.166.1 ± 0.471.8 ± 2.0

As shown in Table 2, full TAL achieves higher mean held-out accuracy than transcript-only TAL across all video budgets. With 48 videos, transcript-only TAL improves accuracy from the Vanilla baseline of 57.6% to 62.1%, while full TAL reaches 71.8%, yielding a further 9.7 percentage-point improvement. These results show that while TAL can improve performance using textual instruction alone, incorporating visual evidence provides substantial additional benefits under the same learning framework.

We also appreciate the reviewer’s observation regarding temporal modeling and agree that developing more video-centric mechanisms to leverage richer temporal visual information, while preserving TAL’s fine-tuning-free nature, is an important direction for future research, as discussed in our Limitations section. We will further clarify the role of timestamp-aligned visual evidence in the Method section and emphasize the visual ablation findings in the revised manuscript.

AuthorsOpenReview ↗

Author Rebuttal (Part2)

Q1. Access to Web Search and External Tools During QA Evaluation.

Thank you for your question. We confirm that web search and external tools were disabled for both TAL and all baselines during QA evaluation. We will clarify this experimental setting in the revised manuscript.

Q3. Tutorialbook Construction and Per-Query Inference Costs.

Thank you for this question. In our cross-model experiments, constructing the complete tutorialbook from all 48 training videos using GPT-5.2 costs approximately \63.15(63.15 (\\1.32 per video), based on API rates of \1.75permillioninputtokensand1.75 per million input tokens and \\14.00 per million output tokens. Given an average video duration of 18.6 minutes, this corresponds to approximately \$0.07 per minute of video. Importantly, the tutorialbook is constructed only once and can subsequently be reused across different LLM backbones without additional construction costs.

For inference with the 48-video tutorialbook and top-3 TOC retrieval, the average per-query costs, including retrieval and answer generation, are \0.079forGPT−5.2,0.079 for GPT-5.2, \\0.051 for Gemini 3 Flash, \0.028forKimiK2.5,and0.028 for Kimi K2.5, and \\0.101 for Claude Sonnet 4.6. All costs are calculated from token usage using OpenRouter’s listed API pricing. We will include these cost details in the revised manuscript.

References.

[1] P. Asawa et al. Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments. arXiv:2606.05661, 2026.

Dear Reviewer Q7LB,

Thank you very much for your time and effort in reviewing our paper. Your constructive comments have been very helpful in improving the quality of our work, and your positive feedback has meant a lot to us.

As the author–reviewer discussion period is approaching its end, we would be grateful to hear whether our responses have addressed your concerns or if any questions remain. We would be happy to provide further clarification or additional evidence if needed.

Thank you again for your time and consideration.

Best regards, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

We sincerely thank all reviewers for their constructive feedback and thoughtful assessments of our work. We appreciate their recognition of the importance of the research problem and the empirical contributions of TAL.

In response to the reviewers' comments, we have revised the manuscript to clarify our methodological formulation and provide additional experimental evidence addressing the key concerns. The main revisions are summarized below:

  1. Clarification of Agent-Level Learning (Section 3). We clarify the formulation of nonparametric learning at the agent level, where a frozen LLM operates within a stateful agent harness that manages persistent memory, retrieval, and context construction.

  2. Additional Computation-Controlled Baseline (Section 4.3 and Appendix D). We introduce an Iterative RAG baseline that constructs and refines structured tutorial memory using the same video evidence but without TAL's hard-question adversarial loop. We compare construction and inference token budgets to investigate whether TAL's improvements can be explained by additional LLM computation.

  3. Cross-Opening Generalization (Section 4.3, Table 3, and Appendix E). We conduct leave-one-opening-out evaluations across all twelve chess openings to examine whether TAL can generalize beyond the opening-specific knowledge used during memory consolidation.

  4. Broader Continual Learning Evaluation (Section 4.3 and Appendix F). We extend TAL to all six tasks of Continual Learning Bench, evaluating its applicability to continual learning from textual task descriptions, interaction trajectories, and environment feedback.

  5. Additional Methodological Clarifications and Evaluation Details (Sections 3–4 and Appendices B–C). We clarify the use of temporally aligned visual and textual evidence, further discuss the contribution of visual information, specify the tool restrictions during QA evaluation, and report construction and per-query inference costs.

These revisions preserve the core TAL framework while strengthening the empirical evaluation and clarifying the scope of our contributions. We thank the reviewers again for their valuable feedback and consideration.

Dear Reviewers,

Thank you again for your time and effort in reviewing our paper. We greatly appreciate your constructive feedback, which has helped us improve our work.

As the author–reviewer discussion period approaches its end, we would be grateful to hear whether our responses and additional experimental results have addressed your concerns, or if any questions remain. We would be happy to provide further clarification if needed.

Thank you again for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.