- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether advice from smaller language models provides genuine incremental value when larger language model agents revise failed programming solutions. The authors propose a paired evaluation framework comparing self-revision (SELF), task-matched peer advice (MATCHED), task-mismatched advice (MISMATCHED), and advisor-free review (NO ADVICE), using the same archived checkpoints. Experiments on 300 CodeContests tasks show that an 8B advisor improves a 35B MoE recipient's success rate from 52.0% to 64.3%, although mismatched advice achieves comparable performance. Conversely, an 8B recipient benefits substantially more from matched than mismatched advice. On a 150-task subset, matched advice shows no detectable advantage over advisor-free review. Additional exploratory experiments suggest that concrete repair guidance can further improve revision outcomes. The main contribution is an empirical evaluation framework that distinguishes revision-workflow gains, task-matching effects, and the incremental value of peer advice beyond advisor-free review.
Critical strengths
Meaningful research question and well-motivated evaluation framework. The paper addresses an important problem in multi-agent LLM systems: improvements from collaboration do not necessarily imply that peer advice contributes useful information. The proposed paired evaluation framework distinguishes workflow improvements, task-matching effects, and genuine advisor contributions through controlled comparisons. In particular, introducing the NO ADVICE baseline helps avoid incorrectly attributing revision gains to peer advice, providing a valuable perspective for evaluating multi-agent collaboration.
Critical weaknesses
- Limited coverage and substantial heterogeneity in the key baseline experiments. The crucial NO ADVICE baseline is evaluated only on the first 150 sorted tasks, limiting the generalizability of the conclusions. More importantly, Appendix L reports substantial heterogeneity in the 8B recipient's matching effects between the two task halves (1.68 vs. 23.68 percentage points). Since NO ADVICE is not evaluated on the second half, it remains unclear whether matched advice provides genuine incremental value over advisor-free review on the full benchmark.
- Insufficient novelty relative to closely related work. The proposed approach substantially overlaps with ScriptSmith (AAAI 2025), which already investigates smaller-model peer review for larger-model-generated code and compares peer review against self-reflection. However, this closely related work is not discussed. While the present paper introduces additional controls, its methodological novelty beyond existing peer-refinement frameworks remains insufficiently justified. Furthermore, although several ablation-style comparisons are provided, the individual contributions of the review-instruction components are not systematically isolated.
Questions for authors
- Could the authors extend the NO ADVICE baseline to the remaining 150 tasks for both recipients? In particular, given the substantial heterogeneity reported in Appendix L, does matched advice outperform advisor-free review on the second half of the benchmark?
- Could the authors clarify the methodological novelty of this work compared with ScriptSmith (AAAI 2025), particularly regarding smaller-model peer review and self-reflection for code refinement? Additionally, could the authors provide a more fine-grained ablation or analysis to distinguish the contributions of individual review-instruction components?