acceptodds
Under review as a conference paper at ICLR 2027

Paper Agent Network: Literature-Grounded Multi-Agent Program Discovery

Abstract

Using a research paper in program search requires turning its methods into executable candidates and adapting them to a target task. A paper can initialise an approach whose implementation continues to develop through evaluation and findings from other approaches. We introduce the Paper Agent Network (PAN), a literature-grounded multi-agent system that organises this process around persistent coding agents. Each agent starts from a selected paper and develops a revisable method route through tool use, evaluator feedback, and accumulated implementation history. A shared result board and queued reports connect these routes; a progress-based controller allocates work and introduces reserve agents. The design separates admitting an improved artifact from sharing a finding that may inform later work. We evaluate the complete system on 23 tasks spanning mathematics, systems, engineering, planning, scientific code, and prediction against four open program-evolution frameworks. The reported comparison contains twelve best or tied-best results. Two process analyses make method and information use concrete. A denoising construction appears in executable artifacts, and a peer’s scheduling report motivates a change to a route that improves through later work. The 17-task literature comparison shows that the positive denoising case does not imply a general benefit from literature access. These observations establish method and information use within the implemented system; interaction’s independent performance contribution remains unresolved. Persistent paper-informed routes provide a concrete setting for studying how published methods are implemented, exchanged, and adapted during computational problem solving.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM agents, automated algorithm discovery, multi-agent systems, program synthesis, evolutionary search, sample efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_Lxt1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents PAN, a multi-agent framework that uses research papers to guide program evolution. Agents explore different approaches, share findings, and adapt their search based on progress and available resources.

Critical strengths

The idea of using existing scientific knowledge to guide program search is interesting, and the framework is evaluated on a diverse set of 23 tasks. The results are promising and suggest that this approach has potential for automated program optimization.

Critical weaknesses

1. The core novelty is not clearly articulated or adequately validated.

After reading the paper multiple times and discussing it with AI to check my understanding, I still struggle to see how this system differs from a standard multi-agent system. If I understand correctly, PAN consists of multiple coding agents that start from different initial methods, work persistently, and are managed by a controller while collaborating indirectly through shared records. Could a similar result be achieved by simply extending an existing program evolution framework with paper retrieval, multiple independent search branches, a shared blackboard, and adaptive scheduling? Furthermore, although the main design includes literature guidance, persistent agents, inter-agent report exchange, and adaptive scheduling, the main experiment (Table 1) only compares the complete PAN system against four existing frameworks. There are no sufficient ablation studies to establish the individual contributions of these design choices.

2. The performance improvements in Table 1 are often very small.

The performance gaps between the proposed method and existing methods are quite small on many tasks, with differences sometimes only visible at the fourth decimal place. By my count, although I may have miscounted, approximately 14 out of 23 tasks have score differences below 0.01. Such small improvements seem to have limited practical significance. Moreover, why are some results in Table 1 reported to four decimal places while others are reported to only three? This inconsistent precision makes the comparisons harder to interpret.

3. The unusually large performance gap on Adversarial Splay needs clarification.

I am not very familiar with the Adversarial Splay task, but the proposed method appears to outperform the other approaches by an enormous margin (e.g., 0.527 vs. 0.006). Please clarify whether there could be any data issues or unfairness in the comparison, particularly regarding the task setup, artifact interfaces, and evaluation protocol.

Questions for authors

I have no further questions beyond the concerns already raised in the Weaknesses section.

1 comment

Thank you for the direct questions. Your reading of PAN is accurate, and we ran the experiment you propose.

W1. Could an existing framework with paper retrieval, branches, a blackboard and scheduling do the same? We built that baseline: OpenEvolve (islands, migration, MAP-Elites archive, scheduler) with the same three papers' method indexes in its prompt, 3 runs per task, same model and cap.

TaskPANOpenEvolve+ papers
Heilbronn (triangle)0.03350.03230.0323
Heilbronn (convex)0.02730.02820.0247
Circle packing2.63122.62772.6275
Signal processing0.7280.6580.531
LLM–SQL0.6760.7210.701
Scheduling394737433331
Compression0.2810.3790.348
ResNet bit-flip0.9880.9060.988

It is lower than PAN on 5 of 8 tasks, clearly on signal (−27%) and scheduling (−16%), never clearly higher, and the papers make OpenEvolve itself worse on 7 of 8. We also ablated each design choice (3 runs, 8 tasks; clear = all runs outside Full's range):

RemovedMean vs FullLower onClearly lower
Literature + plurality−16.3%7/8signal
Plurality (one agent)−3.2%5/8triangle, convex, scheduling
Literature+4.0%3/8signal
Adaptive scheduling−1.0%2/8—
Blackboard reports+2.0%3/8—
Persistent agents+5.0%1/8—

So the answer is partly yes: report exchange, persistence and scheduling show no clear effect at this budget, and we will present the simpler system. What matters is that each agent implements, measures and revises one paper's method: removing that costs most, and the same papers in a mutation prompt do not substitute for it. We restate the contribution in these terms.

W2. Small improvements; precision. We agree many gaps are small and no longer claim a general lead. An absolute 0.01 threshold misleads where scores are near 0.03; PAN is within 1% of the best baseline on 7 of 21 tasks. A Wilcoxon signed-rank test over the 21 tasks puts PAN significantly above ShinkaEvolve (p = 0.011) only, not CodeEvolve (0.060), OpenEvolve (0.089) or AdaEvolve (0.56), so we describe PAN as level with the strongest framework. Table 1 now uses one precision per row.

W3. Adversarial splay. Indeed PAN generated the whole sequence while baselines edited a seed artifact, so 0.527 vs 0.006 reflected the interface. We removed splay and sorting networks; all 21 tasks now use one interface for every system.

Reviewer_cyaZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PAN searches for programs using coding agents initialized from selected papers. Agents retain their histories and code, exchange reports and evaluated artifacts, and receive work based on progress and cost.

Critical strengths

PAN demonstrates that published methods can yield competitive programs in automated search. In signal denoising, it implements the source paper’s algorithm and outperforms the no-paper configuration (0.728 vs. 0.630; §5.2).

Critical weaknesses
  1. Paper access does not consistently improve performance: full-paper and no-paper PAN each win seven tasks, with three ties (Table 2). Persistent histories and report exchange are not separately tested, and the scheduling trace shows report use without establishing a performance benefit (§5.2–5.3; B.1).

  2. Tools, token budgets, and computation are not matched across systems, so higher scores cannot be attributed to PAN's agent organization alone (§4; B.2). In splay, PAN can write code to generate the entire 4,096-number sequence, while baselines edit the number list directly (§5.1).

  3. The aggregate lead lacks uncertainty estimates: each task has only three PAN runs and two baseline runs. After excluding the two tasks with unequal interfaces, PAN retains the average-rank lead (2.10 versus 2.52), but its normalized-score lead is only 0.938 versus 0.934 (§4–5.1).

  4. Actual run configurations, implementation snapshots, and complete task–paper selections are missing, preventing reconstruction of the experiments (Table 5; B.1; C.1). Evaluation instances and prediction-data splits are also unspecified, including whether final scoring uses data already used during search (§4).

Questions for authors
  1. What was the “uncounted feedback” mentioned on p.10, and which reported runs received it? Did it influence candidate selection beyond the 30 recorded evaluations?

Minor comments

  1. Suggested related-work discussion: how PAN's paper-specific routes differ from CORAL's long-running agents with shared persistent memory and DeepEvolve's combination of research retrieval and algorithm evolution.
2 comments

Thank you for the precise review. We take each point in turn.

W1. Paper access; persistence and report exchange. We now test every component (3 runs per configuration on the 8 literature-comparison tasks, against Full run in the identical setting; a gap is clear only when all three runs fall outside Full's range):

RemovedMean vs FullLower onClearly lower
Papers + plurality−16.3%7/8signal
Plurality (one agent)−3.2%5/8triangle, convex, scheduling
Paper access+4.0%3/8signal
Adaptive allocation−1.0%2/8—
Report exchange+2.0%3/8—
Persistent histories+5.0%1/8—

Paper access clearly helps only on signal denoising (0.728 vs 0.609 with 3 runs each; your 0.630 came from 2 runs) and its effect shows mainly with plurality removed (7 of 8 tasks drop). Persistence and report exchange show no clear effect, confirming your reading of the scheduling trace; the revision says so and presents the simpler system.

W2. Unmatched resources; splay. Systems are matched in scored candidates, not tools or tokens, which we state. To test whether gains come from PAN's organisation rather than the papers, we gave OpenEvolve (islands, migration, archive, scheduler) the same papers' method indexes, 3 runs per task: it is lower than PAN on 5 of 8 tasks, clearly on signal (−27%) and scheduling (−16%), never clearly higher. PAN reads more input tokens than the baselines (4.4M vs 0.2–1.3M per run) but, with 95% cached, costs about the same at equal prices (≈$0.25 vs $0.22 for ShinkaEvolve and $0.68 for CodeEvolve; at least $0.17 and $0.11 for AdaEvolve and OpenEvolve, lower bounds as they log no usage). On splay you are right; we removed the two unequal-interface tasks (splay, sorting networks), leaving 21 tasks with one interface.

W3. Uncertainty of the aggregate lead. We agree. On the 21 tasks PAN's normalised score is 0.934 vs AdaEvolve's 0.932 (bootstrap interval of the gap −0.082 to +0.085; Wilcoxon p = 0.56). PAN keeps the best average rank (2.19 vs 2.52) and is significantly above ShinkaEvolve (p = 0.011) only. We drop the claim of a lead; baselines still have 2 runs, which we state. Table 1 now uses one precision per row.

W4. Reproducibility and leakage. Anonymised repository with run commands, every configuration of the ablation, task preparation, and the paper pools and selections for 15 of the 21 tasks (the remaining six will follow): https://anonymous.4open.science/r/PaperAgentNetwork-32AB/. scripts/run_ablations.sh runs every configuration above. Instances are each benchmark's own (fixed seeds for signal, three workloads for scheduling, MLE-Bench's split graded on private answers, hidden sequences for compression). No candidate can read answers, but, for every system, the search score is the reported score. We therefore scored every recorded best ablation program on fresh instances (new seeds; new workloads with the same transaction shapes):

SystemSignal: search → freshScheduling gain over seed: search → fresh
PAN0.728 → 0.7031.41× → 1.41×
No papers0.609 → 0.6131.34× → 1.31×
OpenEvolve + papers0.531 → 0.5311.19× → 1.20×

Rankings carry over across all eight configurations (Spearman ρ = 0.98 and 0.83).

Q. "Uncounted feedback" (p. 10). It referred to an earlier set of runs; none of the runs in the revised paper received it, the sentence is removed, and every reported run selects only from its 30 recorded evaluations.

Minor. We add CORAL (long-lived agents sharing a persistent memory; our ablation finds persistence and exchange add little at this budget) and DeepEvolve (one agent alternating deep research and evolution; PAN gives each paper its own agent and selects among measured results).

Thank you for the careful response and for running the component ablations. I appreciate that the results are reported openly. The repository and the scores on fresh instances also help with reproducibility.

However, the new results mostly confirm my concerns. Paper access clearly helps only on signal denoising, and without paper access the mean score is 4.0% higher than the full system. Persistent histories and report exchange show no clear effect, and the lead over AdaEvolve is no longer claimed (0.934 vs 0.932). After these changes the main claims are quite different from the submitted version, and I think the revised paper would need a new full review. The baselines still have only two runs, tools and tokens are still not matched, and the paper selections for six tasks are not released yet. I therefore keep my rating.

Reviewer_h5HaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Paper Agent Network (PAN), a literature-grounded multi-agent framework that employs persistent, paper-conditioned coding agents to iteratively develop executable solutions through evaluation feedback, information sharing, and adaptive resource allocation. Experiments across 23 diverse tasks demonstrate competitive performance against four program-evolution baselines, achieving the best or tied-best results on 12 tasks.

Critical strengths
  1. The framework presents an interesting approach to integrating literature-grounded methodological exploration with persistent multi-agent collaboration, enabling agents to iteratively refine and exchange research-inspired solutions.
  2. The evaluation covers a diverse set of 23 tasks, and the additional analyses of literature utilization and inter-agent information exchange provide useful insights into the framework's behavior and practical effectiveness.
Critical weaknesses
  1. The full-paper configuration does not consistently outperform the no-paper variant, raising concerns about the effectiveness of the paper-grounded design.
  2. Differences in tool access and computational budgets raise concerns about comparison fairness, while missing ablations leave the contributions of agent specialization, information exchange, and adaptive allocation unclear.
Questions for authors
  1. Could the authors further clarify why the full-paper configuration does not consistently outperform the no-paper variant? What evidence supports the effectiveness of literature grounding?
  2. Could the authors clarify how differences in tool access and computational budgets affect the comparisons? Additionally, is there any evidence or ablation analysis that could help isolate the contributions of agent persistence, information exchange, and adaptive allocation?
2 comments

Thank you for the review and the two direct questions, which we answer with new experiments.

Q1. Why does full-paper not consistently beat no-paper, and what supports literature grounding? With 3 matched runs per configuration on 8 tasks, papers clearly help where they describe a method the seed lacks: on signal denoising Full scores 0.728 vs 0.609 without papers, and Full's programs implement the paper's partial-circulant SVD construction. Without papers bit-flip is also less reliable (one run falls to 0.700). Elsewhere No papers stays within Full's run-to-run range, so papers do not help uniformly, and we now say so. The stronger evidence is the interaction with plurality: removing papers and using one agent lowers the score on 7 of 8 tasks (−16.3% on average), more than either removal alone. How papers are used also matters: the same papers in OpenEvolve's prompt make it worse on 7 of 8 tasks.

Q2. Contributions of persistence, exchange and allocation. We ablated each (3 runs per configuration on 8 tasks; a gap is clear only when all three runs fall outside Full's range):

RemovedMean vs FullLower onClearly lower
Papers + plurality−16.3%7/8signal
Specialisation / plurality (one agent)−3.2%5/8triangle, convex, scheduling
Papers+4.0%3/8signal
Adaptive allocation−1.0%2/8—
Information exchange+2.0%3/8—
Persistence+5.0%1/8—

Persistence, exchange and adaptive allocation show no clear effect at this budget, so the revision presents the simpler system the evidence supports: several agents, each starting from a different paper, sharing one leaderboard.

Q2. Tool access and budgets. Systems are matched in scored candidates (30 per run), not tools or tokens. We removed the two tasks where PAN's tools let it write whole artifacts while baselines edited (adversarial splay, sorting networks); all 21 tasks now share one interface. Per run, PAN reads more input tokens than the baselines (4.4M, 95% cached, vs 0.2–1.3M) but costs about the same at equal prices: ≈$0.25, against ShinkaEvolve $0.22, CodeEvolve $0.68, and at least $0.17 and $0.11 for AdaEvolve and OpenEvolve (lower bounds estimated from stored prompts, as they log no usage). To separate organisation from tools and papers, OpenEvolve with the same papers and model is lower than PAN on 5 of 8 tasks, clearly on signal denoising (−27%) and scheduling (−16%).

The new ablations show a clear benefit from literature grounding on signal denoising, but the benefit remains inconsistent across tasks. Removing the two tasks with different artifact interfaces improves the comparison, though tool access and token budgets still differ. I retain my rating of 2 (Weak rejection).

Reviewer_QgtEOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces Paper Agent Network (PAN), a literature-grounded multi-agent system for program evolution. Each persistent coding agent starts from research paper and develops a task-specific approach through implementation, evaluation, accumulated history, and reports from other agents. PAN combines paper-conditioned routes, a shared result board, report exchange, and adaptive work allocation. Experiments on 23 tasks show that PAN achieves 12 best or tied-best results, while a 17-task comparison indicates that the benefit of literature access is highly task-dependent.

Critical strengths
  • Meaningful research problem. The paper studies how research papers can initialize and guide persistent program-evolution processes, an aspect that is less explored in existing program-search frameworks.
  • Clear motivation and coherent design. The roles of persistent agents, shared results, report exchange, and adaptive allocation are clearly motivated. The distinction between artifact admission and information sharing is particularly useful.
  • Concrete process analysis and transparent discussion. The paper traces how a paper-derived method and a peer report enter later implementations. It also clearly acknowledges that the independent contributions of persistence, interaction, and allocation remain unresolved.
Critical weaknesses
  • Insufficient ablation studies. The paper does not separately evaluate the shared result board, report exchange, persistence, paper access, and adaptive allocation. Therefore, it is unclear which components are responsible for PAN’s reported improvements.
  • Unstable task-level performance. PAN leads the aggregate comparison, but its performance varies substantially across tasks. The literature-access comparison also shows that full-paper agents can perform worse than no-paper agents, which weakens the claim of a generally effective method.
  • Potential interface confounds. As the authors note, PAN may have an action-space advantage on tasks such as adversarial splay and sorting networks, where it can generate complete artifacts while some baselines mainly edit existing ones. This makes it difficult to determine whether the gains come from PAN’s method or from differences in task interfaces.
  • Incomplete computation-cost comparison. Figure 3 compares progress under the same 30-candidate evaluation cap, but token usage, API cost, caching, and offline computation are not matched across systems. Thus, faster convergence under the candidate budget does not establish better computational efficiency.
Questions for authors

See above.

1 comment

Thank you for the careful review. We ran the ablations you asked for and revised the paper's claims to match them.

W1. Ablations. We ablate each component on the 8 literature-comparison tasks, 3 runs per configuration, against Full run in the identical setting (revised Table 2). A gap is clear only when all three runs fall outside Full's three-run range. "No relay" withholds all peer reports and reference programs, so agents share only the board's score bar.

RemovedMean vs FullLower onClearly lowerClearly higher
Papers + plurality−16.3%7/8signal—
Plurality (one agent)−3.2%5/8triangle, convex, schedulingsignal, compression
Paper access+4.0%3/8signal—
Adaptive allocation−1.0%2/8——
Report exchange+2.0%3/8——
Persistence+5.0%1/8—compression

Papers and plurality are what matter, and they help on different tasks: one agent is clearly worse on the search-heavy constructions and scheduling, no papers on signal denoising, and removing both lowers the score on 7 of 8 tasks. Report exchange, persistence and adaptive allocation show no clear effect at this budget; we say so and present the simpler system. Compression and LLM–SQL vary too much between Full's own runs to conclude from.

W2. Unstable task-level performance. We agree and no longer claim a generally better method. Over Table 1's 21 tasks, a Wilcoxon signed-rank test on normalised scores puts PAN significantly above ShinkaEvolve (p = 0.011) but not CodeEvolve (0.060), OpenEvolve (0.089) or AdaEvolve (0.56); PAN and AdaEvolve are level (0.934 vs 0.932). On full-paper agents scoring below no-paper agents: with 3 matched runs, papers clearly help only on signal denoising and are within run-to-run spread elsewhere, so the revision presents literature as task-dependent and complementary to plurality.

W3. Interface confounds. We removed the two tasks where PAN generated whole artifacts while baselines edited existing ones (adversarial splay, sorting networks). Table 1 now has 21 tasks with one artifact interface for every system.

W4. Computation cost. We agree that equal candidate caps do not show computational efficiency and remove that reading of the trajectory figure. Tokens and tools are not matched, which we state. Measured per run (PAN: the 24 matched Full runs; baselines: their Table 1 runs on 15 tasks; AdaEvolve and OpenEvolve log no usage, so theirs are estimated from stored prompts and are lower bounds), with cost at the same per-token prices:

SystemInput tokensOutput tokensCost/run
PAN4.4M (95% cached)73k≈$0.25
ShinkaEvolve0.22M127k≈$0.22
CodeEvolve1.32M299k≈$0.68
AdaEvolve≈0.30M≈79k≥$0.17 (lower bound)
OpenEvolve≈0.23M≈49k≥$0.11 (lower bound)

PAN reads far more input tokens, but prompt caching keeps its cost per run within the baselines' range.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.