- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper evaluates the value of transferred agent experience through performance gains on novel problems. LessonSmith augments teacher-authored rules with execution-verified examples and correction records. Across multiple policies and model pairings in TeachWorld, this format improves recipient performance over standard lessons.
Critical strengths
S1. Controlled experiments demonstrate the benefit of transferring execution evidence. Holding rules and diagnostic examples fixed, replacing teacher predictions with execution-derived ground truth and correction records improves recipient performance. This supports the utility of retaining verified experience in the transferred material.
Critical weaknesses
W1. The distinct contribution beyond existing harness optimization remains unclear. For example, Meta-Harness [1] uses execution traces to optimize context construction and execution logic. LessonSmith could likewise be viewed as a specific strategy within such a harness. The current experiments do not sufficiently establish what novel principles or practical advantages this strategy offers beyond existing approaches.
W2. The benefit of accurate examples is not isolated from that of the receipt design. No comparison removes only the correction annotations while retaining identical rules and ground-truth examples. Alternatives that revise only the rules also differ in the amount of information conveyed. Thus, the added value of preserving the original rules and appending correction records remains unclear.
[1] Lee, Y., Nair, R., Zhang, Q., Lee, K., Khattab, O., & Finn, C. (2026). Meta-harness: End-to-end optimization of model harnesses. COLM, 2026
Questions for authors
-
In which settings is LessonSmith particularly advantageous over existing harness optimization approaches? Existing evidence of additional value under the constraint of no student access would help address W1.
-
Are there existing comparisons that remove only correction annotations while retaining identical rules and ground-truth examples, or revise the rules under the same communication budget? Evidence that the proposed format retains its advantage in these comparisons would help address W2.
The authors seem not to engage with the rebuttal yet. I would reconsider the final judgement on the manuscript if the authors tackle my proposed concerns.