- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper demonstrates the effectiveness of several agent components through a series of ablation studies, including empirical guardrails against over-configuration in automated parallelization, reference-guided self-reflection, and self-reflection with lightweight probes. It also compares TritonDFT with the LLM-based framework DREAMS and rule-based parameter-selection baselines derived from Atomate and AiiDA, showing advantages in the reported evaluations.
However, the paper’s characterization of DREAMS is inaccurate. Section 5.3 contrasts a “fixed prompt template” with task-dependent planning and execution, whereas DREAMS already supports dynamic planning, systematic convergence testing, and feedback-driven error recovery. Its execution examples show the planner adding missing calculation steps and revising the workflow after failures. A fixed system-prompt template does not imply a fixed execution plan. Furthermore, the manuscript does not clearly describe the comparison protocol, including whether the official DREAMS implementation and prompts were used, which underlying model was selected, and whether DREAMS could readjust its parameters for downstream SCF and DOS calculations. These omissions weaken the persuasiveness of the comparison.
Moreover, several existing studies have proposed highly similar agent frameworks, including DREAMS [1], VASPilot [2], and AutoDFT [3]. These systems already combine task planning, domain-specific tools, parameter selection, execution monitoring, and feedback-driven recovery. Although the manuscript cites these works, it does not adequately address their architectural overlap with TritonDFT or establish clear methodological differences through direct comparisons. This substantially limits the novelty of the proposed architecture.
References
[1] Wang, Z., et al. (2025). DREAMS: Density Functional Theory Based Research Engine for Agentic Materials Simulation. arXiv:2507.14267.
[2] Liu, J., et al. (2025). VASPilot: MCP-Facilitated Multi-Agent Intelligence for Autonomous VASP Simulations. arXiv:2508.07035.
[3] Yang, P., et al. (2026). AutoDFT: A Closed-Loop Multi-Agent Framework for Autonomous DFT Calculations. arXiv:2605.26179.
Critical strengths
- The paper presents extensive ablation studies demonstrating the effectiveness of several components, including reference-guided self-reflection, lightweight execution probes, and expert-informed guardrails for parallelization, which improve parameter-selection accuracy and computational efficiency.
Critical weaknesses
-
Limited novelty. Several existing frameworks, such as DREAMS, VASPilot, and AutoDFT, adopt similar designs involving task planning, tool use, and feedback-driven refinement. The paper does not sufficiently distinguish its architecture from these approaches or demonstrate the contribution of its specific design choices.
-
Insufficient experimental detail. The paper does not consistently report the effective test-set size and pass-rate denominator for each experiment. The comparison with DREAMS also lacks key details, including the underlying model, whether the official implementation and prompts were used, and whether both methods had comparable computational budgets and opportunities to adjust parameters. These omissions make the results difficult to reproduce and the fairness of the comparison difficult to assess.
Questions for authors
-
Could a stronger model, equipped with an automated execution loop as in Codex or Claude Code, achieve comparable results with appropriate prompting, without the proposed framework?
-
Are the observed improvements statistically significant given the current test-set sizes? How sensitive are these results to variations in prompt wording?
We thank the reviewer for the detailed feedback and for recognizing the effectiveness of improvement.
Weakness 1: Novelty and Distinction from Existing Agent Frameworks
We agree that existing frameworks, including DREAMS, VASPilot, and AutoDFT, have demonstrated the potential of LLM agents for automated DFT workflows. Our contribution focuses on additional challenges in scientific computing rather than workflow automation alone. In particular, TritonDFT introduces Pareto-aware parameter inference to balance numerical accuracy and computational cost, lightweight probing for feedback-driven refinement, and hardware-aware parallelization. DFTBench further evaluates these capabilities across 67 materials with diverse physical and computational characteristics. These dimensions remain not fully explored in existing agentic DFT studies.
We will clarify these distinctions and better position our contributions relative to prior work in the revision.
Weakness 2: Experimental Details and DREAMS Comparison
Thank you for highlighting these issues. We acknowledge that our description of DREAMS as using a fixed prompt template could be misleading. DREAMS supports dynamic planning, and we will revise this characterization.
Our comparison focuses on the numerical accuracy of configurations selected by each framework and their suitability for downstream calculations. We will provide additional details on the DREAMS implementation, underlying LLM, prompts, computational budgets, and opportunities for parameter refinement. We will also clarify the effective test-set sizes and pass-rate denominators across experiments to improve reproducibility.
Question 1: Comparison with General-Purpose Coding Agents
General-purpose coding agents may face challenges in scientific computing without sufficient guidance from domain experts, particularly in selecting physically appropriate parameters and balancing numerical accuracy with computational cost.
We will include additional baselines using general-purpose coding agents, such as Codex and Claude Code, in the revision to further evaluate their capabilities on DFTBench and compare them with TritonDFT.
Question 2: Statistical Significance and Prompt Sensitivity
We thank the reviewer for raising this concern. DFTBench includes 67 materials across 10 categories and three accuracy thresholds. Our experiments demonstrate consistent improvements from reference guidance and lightweight probing across different material types.
In the revision, we will clarify the sample sizes and evaluation protocols. We will also expand the statistical analysis and examine sensitivity to prompt variations to better characterize the robustness of our findings.
Thank you again for your constructive feedback.