- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents NanoResearch, a multi-agent system for personalized research automation across idea generation, experimental implementation, and manuscript preparation. Its proposed tri-level co-evolution combines a skill bank of reusable procedural rules, memory of user- and project-specific experience, and a planner adapted to natural-language feedback through SDPO with user-specific LoRA updates. An orchestrator retrieves relevant skills and memories, coordinates specialized agents, and updates accumulated knowledge throughout the research workflow. The evaluation compares existing research systems on 20 tasks across seven domains and includes ten real research tasks supplied by PhD researchers. Reported validation covers execution success, task-native predictive performance, preference alignment, novelty, writing quality, component ablations, and changes in performance and cost over three rounds.
Critical strengths
-
A substantial system contribution with a carefully controlled static comparison. NanoResearch connects literature retrieval, experiment planning, code execution and debugging, manuscript production, procedural skills, contextual memory, and an adaptable planner into one research workflow. Appendix C explicitly gives every system the complete researcher profile and matches model backends by shared role, inference settings, tools, and task budgets. Consequently, the Round-1 comparison in Table 1 provides meaningful evidence about orchestration: the reported benchmark E2E success is 1.000 versus at most 0.900 for the baselines, while objective task performance is 0.6844 versus the strongest baseline's 0.6238. The improvement is therefore broader than a subjective preference score. This is a useful empirical contribution even if the learning mechanisms require stronger attribution.
-
The human evaluation goes beyond simulated users and separates preference satisfaction from independent quality assessment. Appendix N describes ten PhD researchers contributing ten frozen, real research tasks with heterogeneous constraints, including reproducibility, interpretability, compute efficiency, and patient-wise generalization. Task owners provide feedback and rate Alignment; three independent, domain-matched reviewers per task assess Novelty and Writing under anonymization, format normalization, and randomized presentation. Review assignments are correctly averaged within task rather than counted as independent experimental replicates. Table 1 reports an already strong human-study Round 1, followed by increases in Alignment from 8.1 to 9.1 and objective performance from 0.8012 to 0.8564. These design choices make the evidence more persuasive than an evaluation relying exclusively on an LLM feedback-and-judging loop.
Critical weaknesses
-
The central ablation configurations and aggregation rules are unresolved. Table 2's “Nano w/o Planner Model” row is numerically identical on all five shared metrics to Table 6's “Planner Model Only” row, including E2E = 0.835 and Perf. = 0.642, although the labels describe opposite planner configurations. A labeling error could explain this, but the actual component masks and experimental conditions need to be identified before these results can establish component contributions. E2E is defined as a fraction of completed runs, yet values such as 0.849, 0.935, and 0.835 cannot arise from one binary outcome for each of twenty tasks. Repeated runs or another aggregation could explain them, but the success counts, denominators, and aggregation rules are not supplied. The full-system ablation row also reproduces Round-1 results without clearly specifying the adaptation state of each variant. These ambiguities are consequential because the ablations provide the main evidence for the proposed three-layer design; they prevent a reliable interpretation of its incremental benefits.
-
The incremental contribution of persistent parameter learning is not adequately separated from feedback-conditioned revision. NanoResearch combines retrieved procedural skills, user/project memory, and a planner updated using existing SDPO. Its learning contribution therefore depends on establishing what the parameter updates and their interaction with retrieval add. Table 6's Planner+SDPO versus Planner Only comparison removes skill/memory retrieval, which is informative, but does not establish that feedback messages, retained context, execution history, and experiment budgets are matched. Appendix Q acknowledges the missing exact same-feedback frozen-planner control. The specific benefit of parameter persistence therefore remains uncertain. Component removal also does not by itself establish the proposed directional co-evolution between skills, memory, and policy. R1–R3 revisit the same task, and Appendix Q explicitly leaves cross-project transfer untested. The results provide useful evidence of feedback-driven project improvement, while the broader claims about persistent preferences and reusable cross-project knowledge need stronger support. This scope limitation compounds the unresolved ablation evidence; it would not alone determine my recommendation.
Questions for authors
-
Clarification: What exact component masks, initialization states, and evaluated rounds generated Tables 2 and 6? Please resolve the identical “w/o Planner Model” and “Planner Model Only” results and identify which labels and values are correct. When, relative to each evaluated artifact, are feedback and SDPO updates applied? This determines what the planner and preference-alignment ablations measure.
-
Clarification: For each ablation E2E value, what are the success count, number of runs, and aggregation rule? How are failed tasks included in Perf., and what are the paired task-level differences or uncertainty estimates for the principal component comparisons? These details are necessary to distinguish robust improvements from aggregation effects or stochastic variation.
-
Clarification and validation of current claims: In Planner+SDPO versus Planner Only, are feedback messages, feedback retained in context, execution history, and inference/experiment budgets matched? Under otherwise matched feedback and research history, what improvement remains when only the learned adapter differs? This would directly affect my assessment of persistent parameter learning. Please also distinguish demonstrated within-task adaptation from the proposed cross-project reuse and preference persistence that Appendix Q leaves untested.