- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Heterotelic Learning, which is implemented as Learned Objective Transformation (LOT). In specific, LOT train a temporary compiler to shape the gradients produced by next-token-prediction loss, and online adaptation can serve an entirely separate goal, such as improving QA performance. Comprehensive experiments demonstrate the effectiveness of the method.
Critical strengths
The authors includes massive experiments to validate the effectiveness of the proposed HL method.
Critical weaknesses
The distinction between LOT and prior meta-learned update rules is not sufficiently established. For the related methods, Meta-SGD and Meta-Curvature learn update directions or gradient transformations, while CaMeLS uses a downstream QA objective to shape NTP-based online adaptation. To my understanding, C-only BCA explicitly transforms the LoRA-writer gradient, making its core operation closely related to a learned gradient map. However, the paper does not include a matched learned-optimizer or preconditioner baseline to show that the temporary compiler provides a distinct capability.
Questions for authors
- How is LOT conceptually distinct from meta-learned gradient transformations or preconditioners, as described above?
- Does “the same online objective” mean only that the NTP targets and loss form are unchanged?