acceptodds
Withdrawn from ICLR 2027

AgentBrew: Lifelong Knowledge Brewing from Strong Teachers to Weak LLM Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deploying LLM agents typically requires a compact test-time student, even if a stronger teacher is available during training. We study knowledge brewing: distilling a teacher’s interactive experience into a persistent external memory for the student. Crucially, this requires no weight updates, expert demonstrations, ground-truth labels, or test-time teacher access. This setting poses two challenges: environments provide only sparse, binary feedback, and teacher-authored notes must be inherently tailored to be concretely executable by a substantially weaker student. To address these hurdles, we propose AgentBrew, comprising two coupled components. First, a failure-triggered teacher–Ralph Loop mitigates sparse feedback by transforming student failures into environment-validated notes. Second, student-aware synthesis calibrates teacher knowledge to the weak executor’s operational granularity, yielding model-specific, actionable guidance. Extensive evaluations and comprehensive ablations across coding, math, and tool-use tasks demonstrate that this asymmetric, training-free brewing paradigm produces highly capable yet deployable LLM agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent, Knowledge Distillation, Continual learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YFQsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AgentBrew, a lifelong knowledge distillation framework designed for weaker LLM agents. It transforms the experience of a strong model into actionable notes for a weaker model. The core approach involves a strong "teacher" model analyzing the "student's" failure trajectories to generate actionable error-correction notes; these notes are iteratively validated through the Student's re-execution in the environment and, once confirmed effective, are stored in long-term memory for retrieval in future tasks. The key idea is that transferred knowledge should not only be correct, but also executable by the weaker student. Experiments on reasoning, coding, and tool-use benchmarks show consistent improvements over other baselines.

Critical strengths
  • The problem definition is practical. It accounts for the capability gap between the strong teacher model and weak student model, and achieves promising results by leveraging strong models to guide weak ones.
  • Notes introduced is verified in real environments, which relatively validates the effectiveness of the notes.
  • The system is easy to maintain and requires no model parameter updates, conferring advantages in API-only settings.
  • The work covers mathematics, code generation, tool use, and Terminal-Bench, yielding a relatively broad research scope.
Critical weaknesses
  • (Crucial) The method's innovation stems primarily from the combination of existing components; similar methods already exist (e.g., ReasoningBank[1], Agent KB[2], Memento-Skills[3], AutoSkill[4], Trace2Skill[5], etc.), so the novelty of the core technology is somewhat limited. Only compared with baseline methods, with no discussion or comparison against recent similar methods.
  • Success on a single task fails to straight establish the causal effect of notes; it cannot, further, demonstrate their cross-task effectiveness.
  • Issues such as long-term knowledge accumulation, training overhead, and negative transfer have not yet been sufficiently validated.
  • This work does not evaluate the cross-domain effectiveness of notes, which limits the scalability of the this method.
[1] Reasoningbank: Scaling agent self-evolving with reasoning memory.
[2] AGENT KB: A hierarchical memory framework for cross-domain agentic problem solving.
[3] Memento-skills: Let agents design agents.
[4] Autoskill: Experience-driven lifelong learning via skill self-evolution.
[5] Trace2skill: Distill trajectory-local lessons into transferable agent skills.
Questions for authors
  • The paper does not sufficiently compare with related methods (see above). Please add discussions on differences from similar approaches and corresponding performance comparisons.
  • The student model itself involves sampling randomness; thus, success upon re-execution might not be attributable to the newly added note. Have you evaluate this through no-note retry or with-note retry experiments?
  • As the note database grows, will erroneous retrieval, knowledge conflicts, or outdated insights gradually impact efficiency and performance?
  • The cross-task analysis in A.4 only reports performance for a single note over 1,200 tasks. Could you provide a more systematic analysis to evaluate the real effectiveness of the AgentBrew system on held-out tasks?
Reviewer_7x3sOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

AgentBrew improves a fixed, weaker student by accumulating knowledge from a stronger teacher in external memory. It generates actionable guidance from student failures, validates it through successful re-execution, and retrieves stored guidance at test time. Experiments and ablations across mathematics, coding, and tool use evaluate performance and component contributions.1. AgentBrew improves a fixed student without requiring the teacher at deployment. Reported gains over the base student on held-out tasks across mathematics, coding, and tool use support the practical value of external knowledge when weight updates are infeasible.

Critical strengths
  1. AgentBrew improves a fixed student without requiring the teacher at deployment. Reported gains over the base student on held-out tasks across mathematics, coding, and tool use support the practical value of external knowledge when weight updates are infeasible.
Critical weaknesses

W1. The distinct contribution over existing distillation and harness optimization methods remains unclear. On-policy distillation (OPD) updates student weights using teacher feedback on student trajectories. Harness optimization (e.g., Meta-Harness) uses a strong proposer’s feedback on execution traces to refine an external harness for a fixed model. The paper insufficiently clarifies these connections and distinctions, making it difficult to assess the conceptual novelty or practical advantages of brewing.

W2. The independent effects of student-specific guidance and execution-based validation are not established. The student replacement experiment also changes the failure experiences and memory content, while removing the validation loop alters opportunities for retries and refinement, as well as memory size. These confounds prevent attributing the gains to student-specific knowledge transfer rather than additional search or differences in memory construction.

Questions for authors
  1. How does AgentBrew relate to OPD, and what distinguishes it from Meta-Harness?

  2. Are there existing results comparing generic and student-specific guidance using identical student failure data, matched execution and memory budgets, and the same validation procedure?

Reviewer_93MEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

AgentBrew is a framework that improves weaker LLM agents using natural language knowledge distilled from a stronger teacher model. When a student fails a task, the teacher generates a corrective note from the failed trajectory. If the weaker LLM passes with the corrective note, the note is collected into a persistent knowledge base. The main claimed contributions are student-aware knowledge synthesis (the teacher gives explicit instructions on the specific failure), and Ralph Loop validation (only accept notes if the student is successful when using them).

Critical strengths
  1. The paper introduces a method to improve agents without needing fine-tuning by learning memory experientially from failed tasks.
  2. Measuring knowledge usability by checking if the student model succeeds after using the knowledge is interesting. This attempts to ensure the knowledge being learnt is calibrated for the student.
  3. Evaluation experiments use separate brewing and test sets. They cover MATH, AppWorld, etc. Improvements in coding and tool-use tasks.
Critical weaknesses
  1. Modest Results:
    1.1. Improvement over vanilla student is 2-4 %, except in MBPP and tool use where it's 10%. However, improvement relative to other teacher-augmented, and non-teacher methods are very small and also worse in some cases.
    1.2. If I understand correctly from Figure 1 and Table 1, in a weak student + weak teacher setting both MATH and GSM8K are worse compared to baseline. Baseline being no knowledge weak model - weak student only (taken from Table 1 ReAct row). In strong teacher + strong student MATH is better and GSM8K is worse than baseline (weak student only). Strong teacher + weak student are better overall. This points that a stronger teacher is necessary, and that the results may be sensitive to run-to-run variance.

  2. Experiments.
    2.1 If the system relies on a stronger teacher model, it is unclear how applicable the method is to frontier models, which are usually used in real-world deployed agents. Its asymmetric teacher-student formulation and student-side validation might not be an advantage compared to existing procedural-memory approaches (some are already referenced, other methods like BREW arXiv:2511.20297, and related experiential learning work arXiv:2603.24093).
    2.2 Missing confidence intervals or run-to-run variance estimates. While the paper reports statistical significance for some comparisons, I believe the small margin of improvements warrants those.
    2.3 Cost comparison to operate this method vs other existing solutions not discussed.
    2.4 I also believe the benchmarks/ datasets used are a bit too outdated for modern LLM agents. While the method might generalise, the experiments are not really indicative of the complex tasks current LLMs (even small open source ones) succeed or fail at.

  3. The ralph loop does not establish causal recovery. A note is accepted if the student succeeds on a subsequent attempt with that note. However, LLM agents are stochastic, and a student may succeed on retrying a previously failed task even without additional guidance. This becomes particularly important when multiple retries and note revisions are permitted. The paper should compare retries with and without the proposed notes on the same failed tasks, using similar rollout budgets. Independent revalidation of accepted notes would also help establish whether the mechanism reliably identifies useful knowledge.

  4. The method generates knowledge specifically for the weaker student. However, its comparison between brewing with strong and weak students changes both the intended recipient of the knowledge and the failure trajectories used to generate it. A more systematic experiment would hold the failure trajectories fixed and compare generic teacher-generated notes against explicitly student-calibrated notes.

Questions for authors
  1. Can the authors compare generic teacher-generated notes against student-calibrated notes while keeping the failure trajectories fixed? This would help isolate the actual contribution of student-aware synthesis.

  2. How frequently does the student recover from failed tasks simply by retrying without any notes? Can the authors compare this against note-assisted retries under the same rollout budget?

  3. Given the relatively small improvements over existing methods, can the authors report results with confidence intervals or variance estimates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.