- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Learning to Self-Evolve (LSE), which trains a context editor through single-step reinforcement learning while keeping the downstream action model frozen. At evaluation time, the editor uses labeled feedback to revise instructions, and UCB-guided search selects contexts using a development set. Experiments on BIRD and SuperGPQA show improvements over untrained editors and prompt optimization baselines, with an additional transfer experiment on another action model.
Critical strengths
- Keeping the action model and evolution protocol fixed provides useful evidence that explicitly training the editor improves context adaptation. The gains are particularly encouraging on BIRD.
- The trained editor improves a different action model without additional training, providing preliminary evidence that the learned editing policy can transfer across models.
Critical weaknesses
- Limited methodological novelty and insufficient validation of the framework’s advantages. Prior work such as PRewrite [PRewrite: Prompt Rewriting with Reinforcement Learning] already trains prompt rewriters through RL using downstream performance. LSE’s single-step policy-gradient formulation and pre-edit reward baseline do not, by themselves, establish a substantial methodological advance. Although the results support training the editor, comparisons with other trained editing approaches are insufficient to establish the advantages of this particular framework.
- The test-time search ablation uses only the untrained editor, leaving the benefit of UCB search within the full LSE system unverified.
- The evaluation depends on labeled target-domain supervision, and the data protocol is insufficiently specified. The editor receives ground-truth answers for adaptation batches, and search repeatedly uses labels from a fixed target-domain development set. This is closer to supervised target-domain prompt optimization than test-time adaptation without target-domain labels; the availability of this supervision and the resulting scope of the claims need clarification. Although dev/test disjointness is stated, the adaptation pool’s source, size, and separation from dev/test remain unclear, as does whether RL reward evaluation or checkpoint selection uses evaluation-domain data.
Questions for authors
- Please clarify the relationship between the adaptation problem pool, dev set, and test set and clarify whether RL reward evaluation or checkpoint selection accesses any evaluation-domain data.
- Please discuss prior methods that train models to generate or refine prompts, such as PRewrite, and provide broader experimental comparisons to substantiate the advantages of the proposed RL framework. The current comparison with a GRPO variant alone is insufficient to establish these advantages.
- Please report the total cost of the test-time evolution process per domain, including feedback collection, context editing, and repeated development-set evaluation. How does the performance–cost trade-off compare with the baselines, and how is this adaptation cost amortized over subsequent queries?