- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes JANUS, a framework for autonomous research that combines pre-execution proposal evaluation with experience-based reflection. Research candidates and their experimental outcomes are maintained in an evolving directed acyclic graph. An LLM-based world model compares proposals according to value improvement, information gain, and exploration utility, guiding proposal selection, refinement, and pruning. Execution feedback is subsequently distilled into research skills, prediction and exploration skills, and coordination protocols. The framework is evaluated on the 30 tasks of MLS-Bench-Lite and three additional research tasks involving language-model training, attention-kernel optimization, and genotype-to-phenotype prediction. The reported results show improvements over several research-agent baselines, while ablations suggest that adaptive exploration and experience reuse contribute to performance. My main reservations concern the connection between the formal world-model formulation and its implemented judgments, and whether the experiments substantiate the stronger efficiency and scaling claims.
Critical strengths
-
The framework addresses a consequential bottleneck in autonomous research. Evaluating and refining proposals before expensive execution is a useful direction, particularly when experiments require substantial training or compilation time. Combining this with a structured record of successful and failed experiments provides a coherent approach to allocating research effort.
-
The evaluation demonstrates practical promise across different research settings. Table 1 includes common-backbone comparisons and reports both development and test performance. For example, with DeepSeek-V4 Flash, JANUS achieves a test score of 41.07 versus 38.91 for Codex. The additional tasks and component ablations strengthen the evidence that the framework has utility beyond a single benchmark configuration.
Critical weaknesses
-
The central world-model and co-evolution claims are not sufficiently connected to measurable predictive behavior. Sections 3.1–3.4 introduce expected cumulative reward, information gain, and exploration utility, but Appendix A primarily supplies their semantic definitions for LLM pairwise judgments. The manuscript does not establish how accurately these judgments predict realized experimental gains, or whether reflection improves prediction quality over time. Moreover, the “w/o World Model” ablation also removes evolution, so that comparison does not isolate forecasting itself. The results support the usefulness of the overall system more clearly than the claimed predictive mechanism.
-
The evidence supports finite-budget performance improvements more strongly than an evolving scaling law or a higher attainable ceiling. Figure 3 and Table 2 measure progress primarily against experiment count. Experiments can differ in execution cost, and proposal scoring, debate, and reflection incur additional inference costs. The fixed five-hour MLS-Bench-Lite comparison is valuable, but it does not resolve the resource interpretation of the additional-task curves. The reported search trajectories also lack sufficiently documented run-level uncertainty to establish stable plateau differences. Consequently, the efficiency and ceiling claims are stronger than the demonstrated evidence.
Questions for authors
-
Implementation clarification: What are the exact procedures for proposal-pool sizing, pair selection, adjustment of and , and skill updating? Which operations are explicit algorithms, and which are determined through LLM prompting?
-
Mechanism validation: Do you have measurements of proposal-ranking accuracy or prediction error before and after world-model reflection? Can these distinguish improved predictive judgment from improved proposal generation and richer memory?
-
Evaluation clarification: How many independent research runs underpin Figures 3–5 and Table 2? Can you provide uncertainty estimates and performance comparisons against total elapsed time or total resource cost, including prediction and reflection overhead?