acceptodds
Under review as a conference paper at ICLR 2027

Teaching Is More Than Solving: Compiling Executable Lessons for Agent-to-Agent Transfer

Abstract

Language-agent systems routinely ask one agent to summarize experience for another, yet they evaluate the sender as a solver rather than as a teacher. We separate these capabilities by defining teaching utility: the improvement that a budgeted lesson causes in a frozen student's decisions on disjoint held-out states. We introduce TeachWorld, a controlled testbed of 144 latent decision policies spanning six semantic domains, three rule structures, and 124 distinct family/structure signatures, and evaluate 432 teacher-concept artifacts across three independently served model families. We then present LessonSmith, a training-free lesson compiler that pairs a teacher's candidate rule with branch-covering executions from an exact verifier and makes every confirmation or correction explicit. Agent-written lessons improve frozen students by 44.6 percentage points over no lesson. Given an executable verifier and a branch-coverage mechanism, LessonSmith adds 4.3 points (95% concept-clustered CI [2.8, 5.9]). Two full-matrix controls identify why. Given the same executed evidence, explicit receipts outperform an execution-informed rule rewrite by 2.7 points ([1.2, 4.2]). When teachers instead choose four probes from eight fresh unlabeled cases, they cover only 2.40 of four policy branches and trail branch-covered lessons by 6.6 points ([4.8, 8.3]). A 12-item solving battery predicts teaching () but still misorders 27.5% of comparable teacher pairs. Teaching is therefore a measurable transfer capability: solving quality matters, while executable coverage and the form in which evidence is communicated determine what the next agent can use.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language agents, teaching utility, agent-to-agent transfer, executable verification, machine teaching, lesson compilation, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_Q1wEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper evaluates the value of transferred agent experience through performance gains on novel problems. LessonSmith augments teacher-authored rules with execution-verified examples and correction records. Across multiple policies and model pairings in TeachWorld, this format improves recipient performance over standard lessons.

Critical strengths

S1. Controlled experiments demonstrate the benefit of transferring execution evidence. Holding rules and diagnostic examples fixed, replacing teacher predictions with execution-derived ground truth and correction records improves recipient performance. This supports the utility of retaining verified experience in the transferred material.

Critical weaknesses

W1. The distinct contribution beyond existing harness optimization remains unclear. For example, Meta-Harness [1] uses execution traces to optimize context construction and execution logic. LessonSmith could likewise be viewed as a specific strategy within such a harness. The current experiments do not sufficiently establish what novel principles or practical advantages this strategy offers beyond existing approaches.

W2. The benefit of accurate examples is not isolated from that of the receipt design. No comparison removes only the correction annotations while retaining identical rules and ground-truth examples. Alternatives that revise only the rules also differ in the amount of information conveyed. Thus, the added value of preserving the original rules and appending correction records remains unclear.

[1] Lee, Y., Nair, R., Zhang, Q., Lee, K., Khattab, O., & Finn, C. (2026). Meta-harness: End-to-end optimization of model harnesses. COLM, 2026

Questions for authors
  1. In which settings is LessonSmith particularly advantageous over existing harness optimization approaches? Existing evidence of additional value under the constraint of no student access would help address W1.

  2. Are there existing comparisons that remove only correction annotations while retaining identical rules and ground-truth examples, or revise the rules under the same communication budget? Evidence that the proposed format retains its advantage in these comparisons would help address W2.

Reviewer_ZgRgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper measures agent-to-agent teaching through improvements in a frozen student’s behavior on unseen states. In synthetic policy tasks, LESSONSMITH preserves a teacher rule and compiles verified diagnostic cases into confirmation or correction receipts. An adaptive version selects cases using observed actions without latent branch labels. The experiments show that these teaching materials can improve student performance in the controlled setting.

Critical strengths

1 The paper asks whether the student learns correct behavior rather than whether the lesson reads well. Freezing the student and evaluating on unseen states gives teaching utility a direct and interpretable target. 2 The statistical analysis respects the structure of the data. Paired analyses by concept and checks for structural similarity and missing records avoid treating correlated decisions as independent evidence. 3. The adaptive condition removes access to latent branch labels while retaining a positive teaching effect. This reduces concern that the method depends on hidden task structure and makes the result more convincing than the branch-informed version alone.

Critical weaknesses
  1. The teacher already has correctly labeled history covering the task branches. Keeping the same rule and selecting history examples by observed action would produce a lesson covering the available actions. The current comparisons do not establish that new verification adds value beyond better selection of existing cases. 2.Receipt encoding is confounded with the amount of correct information. A rule with receipts and a short rule rewrite provide different concrete cases, while the no-execution condition also changes label correctness and confirmation status. Holding the rule and correct cases fixed while removing teacher predictions and receipt markers would clarify whether the encoding itself helps. 3.The manuscript's methods needs legal-state generation, access to the correct action, and a known action set for its coverage rule. General tool environments may not provide these. Action coverage may also be inadequate when branches share actions or outcomes are imbalanced, so the present conclusions are best confined to controlled policy teaching.
Questions for authors
  • How does the student perform with the same teacher rule and action-balanced examples selected from existing correct history? Can selection be prespecified or compared across different choices?
  • With the rule, correct cases, and approximate length held fixed, how does student performance change when only teacher predictions and confirmation/correction markers are removed?
  • What prior information do the correct-action interface and coverage stopping rule require? Do any existing results cover branches that share an action?
2 comments

The authors seem not to engage with the rebuttal yet. I would reconsider the final judgement on the manuscript if the authors tackle my proposed concerns.

As the rebuttal period will end soon, the authors have not engaged with the discussion. I recommend the authors present some new evidence as suggested by the reviewers, to help the reviewers re-judge the final score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.