acceptodds
Under review as a conference paper at ICLR 2027

Can Smaller Models Help Larger Agents? Measuring the Incremental Value of Peer Advice

Abstract

Whether smaller models can provide more effective corrections than self-revision for the failed tasks of large language model agents remains a critical question, one that distinguishes a model's independent problem-solving capability from cross-model collaborative value. This paper conducts fair comparisons between pure self-revision and peer-assisted revision under unified model checkpoints and fixed revision budgets, evaluating all outcomes on hidden test sets across 300 competitive-programming tasks to quantify the gains and risks of peer advice. Advice from an 8B model raises a 35B MoE agent's success from 52.0% to 64.3%, and the gain survives assigning each packet to a different task; in the reverse direction, collaboration depends critically on the task–advice assignment, separating 63.7% under matched advice from 54.0% under mismatched advice. On shared tasks,the two matching effects differ by 15.2 percentage points, a difference reproduced across seeds. Yet a partner-free review that keeps the revision instruction but removes all advice leaves matched advice with no detectable advantage on a fixed 150-task subset, so part of the collaborative gain reflects a productive revision workflow rather than advice content; adding verified repair steps to the same diagnosis still improves the 35B agent by 6.4–14.0 points across three seeds.Accordingly, this paper establishes a paired evaluation framework that separates workflow gains, task-matching value, and content value,measures the increment of peer advice over partner-free review, and quantifies both its benefits and risks, offering new insights for lightweight collaborative optimization of large language model agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Agents, Peer Advice, Self-Revision, Incremental Performance Gain, Model Cooperative Value

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_zgyVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether advice from smaller language models provides genuine incremental value when larger language model agents revise failed programming solutions. The authors propose a paired evaluation framework comparing self-revision (SELF), task-matched peer advice (MATCHED), task-mismatched advice (MISMATCHED), and advisor-free review (NO ADVICE), using the same archived checkpoints. Experiments on 300 CodeContests tasks show that an 8B advisor improves a 35B MoE recipient's success rate from 52.0% to 64.3%, although mismatched advice achieves comparable performance. Conversely, an 8B recipient benefits substantially more from matched than mismatched advice. On a 150-task subset, matched advice shows no detectable advantage over advisor-free review. Additional exploratory experiments suggest that concrete repair guidance can further improve revision outcomes. The main contribution is an empirical evaluation framework that distinguishes revision-workflow gains, task-matching effects, and the incremental value of peer advice beyond advisor-free review.

Critical strengths

Meaningful research question and well-motivated evaluation framework. The paper addresses an important problem in multi-agent LLM systems: improvements from collaboration do not necessarily imply that peer advice contributes useful information. The proposed paired evaluation framework distinguishes workflow improvements, task-matching effects, and genuine advisor contributions through controlled comparisons. In particular, introducing the NO ADVICE baseline helps avoid incorrectly attributing revision gains to peer advice, providing a valuable perspective for evaluating multi-agent collaboration.

Critical weaknesses
  1. Limited coverage and substantial heterogeneity in the key baseline experiments. The crucial NO ADVICE baseline is evaluated only on the first 150 sorted tasks, limiting the generalizability of the conclusions. More importantly, Appendix L reports substantial heterogeneity in the 8B recipient's matching effects between the two task halves (1.68 vs. 23.68 percentage points). Since NO ADVICE is not evaluated on the second half, it remains unclear whether matched advice provides genuine incremental value over advisor-free review on the full benchmark.
  2. Insufficient novelty relative to closely related work. The proposed approach substantially overlaps with ScriptSmith (AAAI 2025), which already investigates smaller-model peer review for larger-model-generated code and compares peer review against self-reflection. However, this closely related work is not discussed. While the present paper introduces additional controls, its methodological novelty beyond existing peer-refinement frameworks remains insufficiently justified. Furthermore, although several ablation-style comparisons are provided, the individual contributions of the review-instruction components are not systematically isolated.
Questions for authors
  1. Could the authors extend the NO ADVICE baseline to the remaining 150 tasks for both recipients? In particular, given the substantial heterogeneity reported in Appendix L, does matched advice outperform advisor-free review on the second half of the benchmark?
  2. Could the authors clarify the methodological novelty of this work compared with ScriptSmith (AAAI 2025), particularly regarding smaller-model peer review and self-reflection for code refinement? Additionally, could the authors provide a more fine-grained ablation or analysis to distinguish the contributions of individual review-instruction components?
Reviewer_6BCYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The central question addressed in this paper is whether smaller language models genuinely contribute useful information when advising larger programming agents. The authors introduce an evaluation framework that distinguishes gains arising from the revision workflow, task-specific advice relevance, and the incremental contribution of an advisor over partner-free review. Through experiments on programming tasks, the authors demonstrate that the performance gains of a 35B MoE agent remain comparable whether it receives relevant advice, mismatched advice, or no advice from an 8B model. These findings highlight that improvements attributed to multi-agent collaboration may instead arise from the revision process itself.

Critical strengths
  1. Problem is well motivated: The analysis of whether the advice is really helpful or not tries to answer an important question.

  2. Intuitive experimental design: The four evaluation conditions SELF, MATCHED, MISMATCHED, and NO ADVICE provide a simple and intuitive way to assess the contribution of peer advice to collaborative performance.

Critical weaknesses
  1. Paper not easy to follow: For instance, the introduction reports several experimental results without establishing why it is necessary to study this problem. Section 3 also lacks coherence; multiple details are presented without clearly connecting them. In Section 4, the authors mention A is the this model, and B is this model, but what is A and B is not connected. A coherent presentation would help the readability of the paper

  2. Experimental scope is limited: The paper proposes a hypothesis, but the scale of the experiment is not sufficient to justify any conclusion drawn from it. The tasks and the model family need expansion to actually draw useful insight from it.

  3. Limited analysis of advice utilization : The paper evaluates the effect of different advice conditions on final task success but the paper doesn't analyse why behind this issue. This analysis would help understand the failure mode of advice framework.

Questions for authors
  1. Why is the MATCHED versus NO ADVICE comparison restricted to 150 tasks, and do the conclusions hold on the full 300-task benchmark?

  2. What explains the difference in sensitivity to matched and mismatched advice between the two recipient configurations?

  3. Have the authors analyzed whether recipients follow, ignore, or modify the received advice, and how these behaviors affect task success?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.