- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
- This paper proposes LANPO, which combines intra-sample reward-agnostic reflection and inter-sample relevant abstraction to guide exploration, while numerical rewards drive policy optimization.
- Experiments demonstrate improved zero-shot performance across mathematical reasoning and coding benchmarks, with further gains from feedback-augmented inference.
Critical strengths
- The paper presents a practical framework that uses language feedback to guide exploration and numerical rewards to optimize the policy. Experiments across multiple model sizes and both math and coding tasks show consistent improvements in average zero-shot performance.
- The framework mixes feedback-conditioned and from-scratch rollouts during training, allowing the resulting policy to solve problems independently while also benefiting from feedback at inference time.
Critical weaknesses
- The benefits of inter-sample feedback may depend on the coverage and density of related problems in the training data, a dependence that is not sufficiently characterized. Meanwhile, intra-sample feedback, which relies less on cross-problem data composition, yields only modest zero-shot gains on math and coding tasks. Given the additional computational cost and pipeline complexity, the current evidence does not clearly establish a compelling cost–benefit trade-off.
- The ablations do not sufficiently isolate the source of the gains. Inter-sample feedback combines experience summarization, relevance filtering, structured prompting, and a format reward, but the existing ablations primarily examine filtering. It therefore remains unclear whether the improvements arise from transferable knowledge in the summaries or from the other components.
Questions for authors
Q1: Table 10 reports a comparable-training-time comparison between LANPO-Both and GRPO. Do you also have results comparing LANPO-Intra with GRPO under the same training-time budget?
Q2: How does the effectiveness of inter-sample feedback depend on the availability of closely related training problems? Do the gains persist when highly relevant experiences are scarce?
Thank you for the authors' responses. Most of my concerns are solved, and I have increased my score. One concerns still remain:
Although the retrieval helps in the settings the authors test, I’m still unsure whether it prevents behavior collapse. A model could comment on the retrieved example and then solve the problem the same way it would have without it. That alone could increase edit distance.