- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a multi-agent system for autonomous neural operator discovery. Concretely, it instantiates 16 "labs", each consisting of a LLM planner for code generation, a numerical worker for code verification and training, and a reviewer for peer review. The communication between labs is based on citation, and for each iteration there exists an eviction rule based on whether the lab's work is "cited" recently. The experiments cover three problems: Darcy, Navier-Stokes and Allen-Cahn; and through ablations on whether community is needed, whether eviction is needed and whether coding is needed, it shows improvements on these components. However, the coding models are restricted to Qwen2.5-Coder 7B, Qwen3-Coder-30B, and gemma3:12b, which are far behind current model capabilities, hindering the effectiveness of their results. Overall, I think it is a reasonable and experimentally-justified framework for neural operator discovery, although having limited novelty and hindered by the authors' selection of models.
Critical strengths
- The contribution of this paper is a framework. And the authors did a great job on ablating each component in the system, and offering insights into why each component helps with the overall performance. In particular, I find the analysis connecting why community is better with recovery from invalid programs insightful.
Critical weaknesses
-
Using open-source models with less than 30B parameters, especially not the latest generation of models hinders the applicability of the findings. Especially considering that from the author's analysis, a major failure case for single-thread is failure to execute the written program, which with more advanced models the probability of this happening is much smaller.
-
The gain from citation-based community versus a simple failure-recovery baseline is incremental. These experiment results makes the claim of citation-based systems not well-justified, and I would like to see further proof on why this citation-based rule is necessary.
Questions for authors
-
The coding versus no-coding comparison does not seem fair. Can the authors provide additional experiment results with matched planner model, objectives, budgets and review policy? I believe a controlled experiment that only varies the architectural representation (i.e. fixed genomes versus code programs) can clear up this confusion by a lot.
-
Could the authors provide further explanation on why the gain from citation-based community over a simpler failure-recovery strategy seems incremental?
-
How does the findings hold for more up-to-date models?
2 comments
Thank you for your efforts in addressing my concerns. My concern in the fairness in evaluation is largely resolved, however I still find the citation rule's necessity as well as the lack of results demonstrated on frontier models (which can be called through openrouter APIs therefore absolutely runnable on consumer hardware) critical. I will maintain my score.
Response to Reviewer d5E4
We thank the reviewer for the review and for the precise experimental request in Q1, which we ran. Below, we address each point.
1. Controlled comparison of code and genomes (Q1)
Reviewer Comment: "Can the authors provide additional experiment results with matched planner model, objectives, budgets and review policy? [...] only varies the architectural representation (i.e. fixed genomes versus code programs)"
What we did. A new genome arm, exactly as requested: same planner (Qwen2.5-Coder 7B), reviewer, review policy, harness, fitness and budget (16 labs x 16 iterations) as the blocks arm. The only change is that the planner chooses a genome over the block vocabulary and a fixed compiler writes the program; invalid genomes go through the same repair loop.
Results. Held-out relative L2 error, mean over search seeds (Tables 2 and 13):
Code wins 8 of 9 pairs on the three problems (geometric mean ratio 0.53, p=0.008). The genome community is even worse than random search over the same vocabulary (0.0640 on Darcy): the planner keeps proposing conservative, repeated genomes, whereas in code it rewires the same blocks into graphs with gates and parallel paths. This controlled comparison now carries the claim about coding (Sec. 5.2, "Code against genomes"); the earlier comparison with the campaign without coding is kept only as the ablation of LLM agency.
2. Is the citation rule necessary? (W2, Q2)
Reviewer Comment: "I would like to see further proof on why this citation-based rule is necessary."
We absolutely agree that the reported gains are small. Replacing the LLM's citations by the fitness ranking, by random citations, or by the fitness ranking without any reviewer gives marginal gains. We expect that further work, using more sophisticated LLMs, will make these gains larger. Our original intent in using open-weight models was to make the approach accessible and reproducible by the broader community using consumer GPUs. Over ten search seeds, with the decision rule fixed before the runs: citations 0.0538, fitness 0.0435, random 0.0478; citations win 3 and 5 of 10 pairs (p=0.50 and 0.64); no reviewer 0.0525 on five seeds (Table 12). The gain over a single lab comes from refounding failed labs from working programs in a population of 16: reset recovers 80% of it on its own, and the remaining 20% is the breadth of the population (every community variant is below reset in mean error). We have removed the claim that the citation rule is necessary, and we state the size of the remaining gain.
3. More recent models (W1, Q3)
Reviewer Comment: "How does the findings hold for more up-to-date models?"
What we did. As said above, our intent in this first communication was to make the approach accessible and reproducible on consumer GPUs. We fully agree with the criticism that these models may not be sophisticated enough. To investigate the effect of a larger model, we extended the 30B comparison to the three problems and six search seeds (18 pairs), with Qwen3-Coder-30B as planner (Table 13):
As anticipated, the 30B planner produces fewer programs that fail to run, but its winning operators are not more accurate on held-out seeds: at this budget the bottleneck is not program validity. We have no API access to frontier models and state this as a limitation. The framework is planner-agnostic: the frozen harness and the held-out retraining, which make the recorded numbers trustworthy (the search's own error understates the held-out error by 1.1x in the median and up to 2.0x), do not depend on the model.
We hope that our responses have adequately addressed your concerns. We respectfully ask you to consider whether they warrant a revision of your score. We welcome any follow-up questions that you might have.