- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents PAN, a multi-agent framework that uses research papers to guide program evolution. Agents explore different approaches, share findings, and adapt their search based on progress and available resources.
Critical strengths
The idea of using existing scientific knowledge to guide program search is interesting, and the framework is evaluated on a diverse set of 23 tasks. The results are promising and suggest that this approach has potential for automated program optimization.
Critical weaknesses
1. The core novelty is not clearly articulated or adequately validated.
After reading the paper multiple times and discussing it with AI to check my understanding, I still struggle to see how this system differs from a standard multi-agent system. If I understand correctly, PAN consists of multiple coding agents that start from different initial methods, work persistently, and are managed by a controller while collaborating indirectly through shared records. Could a similar result be achieved by simply extending an existing program evolution framework with paper retrieval, multiple independent search branches, a shared blackboard, and adaptive scheduling? Furthermore, although the main design includes literature guidance, persistent agents, inter-agent report exchange, and adaptive scheduling, the main experiment (Table 1) only compares the complete PAN system against four existing frameworks. There are no sufficient ablation studies to establish the individual contributions of these design choices.
2. The performance improvements in Table 1 are often very small.
The performance gaps between the proposed method and existing methods are quite small on many tasks, with differences sometimes only visible at the fourth decimal place. By my count, although I may have miscounted, approximately 14 out of 23 tasks have score differences below 0.01. Such small improvements seem to have limited practical significance. Moreover, why are some results in Table 1 reported to four decimal places while others are reported to only three? This inconsistent precision makes the comparisons harder to interpret.
3. The unusually large performance gap on Adversarial Splay needs clarification.
I am not very familiar with the Adversarial Splay task, but the proposed method appears to outperform the other approaches by an enormous margin (e.g., 0.527 vs. 0.006). Please clarify whether there could be any data issues or unfairness in the comparison, particularly regarding the task setup, artifact interfaces, and evaluation protocol.
Questions for authors
I have no further questions beyond the concerns already raised in the Weaknesses section.
Thank you for the direct questions. Your reading of PAN is accurate, and we ran the experiment you propose.
W1. Could an existing framework with paper retrieval, branches, a blackboard and scheduling do the same? We built that baseline: OpenEvolve (islands, migration, MAP-Elites archive, scheduler) with the same three papers' method indexes in its prompt, 3 runs per task, same model and cap.
It is lower than PAN on 5 of 8 tasks, clearly on signal (−27%) and scheduling (−16%), never clearly higher, and the papers make OpenEvolve itself worse on 7 of 8. We also ablated each design choice (3 runs, 8 tasks; clear = all runs outside Full's range):
So the answer is partly yes: report exchange, persistence and scheduling show no clear effect at this budget, and we will present the simpler system. What matters is that each agent implements, measures and revises one paper's method: removing that costs most, and the same papers in a mutation prompt do not substitute for it. We restate the contribution in these terms.
W2. Small improvements; precision. We agree many gaps are small and no longer claim a general lead. An absolute 0.01 threshold misleads where scores are near 0.03; PAN is within 1% of the best baseline on 7 of 21 tasks. A Wilcoxon signed-rank test over the 21 tasks puts PAN significantly above ShinkaEvolve (p = 0.011) only, not CodeEvolve (0.060), OpenEvolve (0.089) or AdaEvolve (0.56), so we describe PAN as level with the strongest framework. Table 1 now uses one precision per row.
W3. Adversarial splay. Indeed PAN generated the whole sequence while baselines edited a seed artifact, so 0.527 vs 0.006 reflected the interface. We removed splay and sorting networks; all 21 tasks now use one interface for every system.