- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies autonomous research agents under ambiguous or underspecified user prompts, where planning often disconnects from execution and scientific validation criteria are missing. The authors introduce IntentSci-Graph, a training-free framework for pre-execution scientific program synthesis. Five asymmetric roles, the Main Agent, Expert, Critic, Validator, and Condenser, interact over a shared state, with the Main Agent implemented as non-LLM control logic that coordinates the other four roles. Before the downstream executor runs, these roles iteratively draft, critique, and verify the original research request (p, W, C) and condense it into an explicit program interface saved as EXP.md. Paired experiments on the 60 project-level tasks of ASI-Bench evaluate four prompt completeness levels from B1 to B4 across four executor and model combinations. While EXP.md reduces degradation on the underspecified B3 and B4 levels and reaches an average gain of +7.00 on B4, it consistently hurts performance on B1 where complete expert procedures are already given, with a reported macro average change of −7.92. Averaged across all 16 configuration and level cells, the net gain is only +0.21.
Critical strengths
The main strengths are:
-
Important research problem. How autonomous research agents should handle ambiguous or underspecified research intent is an important and timely question, and it becomes increasingly relevant as these agents are applied to realistic scientific tasks.
-
Clear presentation. The paper is well written and easy to follow, and its figures convey the overall pipeline and the motivating case intuitively.
-
Intuitive core idea. Making the scientific plan explicit before execution, rather than leaving the executor to infer it while acting, is a natural and well-motivated idea.
Critical weaknesses
The main weaknesses are:
-
The necessity of the multi-role design is not demonstrated. The paper attributes its gains to "self-play-inspired role-based deliberation" (L194–195) but provides no ablation studies for any module.
-
The experimental evaluation is narrow in both benchmarks and baselines. All experiments rely on a single 60-task benchmark, despite citing other benchmarks such as ResearchClawBench by Xu et al. 2026. Moreover, the raw prompt is the only baseline compared. None of the "Auto-research Baselines" in Figure 1, such as AI-Researcher by Tang et al. 2025, is evaluated in Sec. 4.
-
Gains are weak and unstable, and the reported numbers contain internal contradictions. The paper notes that "the average over all 16 configuration–level cells changes only from 23.12 to 23.33 (+0.21)" (L322–323), which does not support the abstract's claim that IntentSci-Graph "effectively improves autonomous research performance" (L027–028). Every configuration drops on B1, B2 shows mixed signs, and CodeBuddy with GLM-5.3-Flash drops by 3.89 on B3 and averages −2.45 overall. Although L312 defines a run index s, the number of runs per task and variances are never reported.
Several numbers in Sec. 4 also contradict one another. First, averaging the four B1 rows in Table 1 yields 38.29, 30.27, and −8.02 rather than the reported 38.92, 31.00, and −7.92, and the downstream figures in Sec. 4.2, including 25.80, 10.88, 14.92, and 23.12, all rely on that incorrect 38.92. Second, L362 states "All four B1 differences are negative (−7.77, −7.09, −11.25, and −5.55)," whereas Table 1 lists −8.17 for CodeBuddy with DeepSeek. Third, Table 2 uses the same configuration, but its B2–B4 task-weighted means disagree with Table 1, such as 17.80 / 28.69 versus 18.67 / 29.56 on B4. -
The terminology overstates the actual implementation. The formulation S = (O, H, M, E, V, X) includes "testable hypotheses H" and "evidence requirements E" (L152–156), yet the Expert's PlanDoc at L230–233 has no fields for either. Similarly, the pipeline is called self-play-inspired despite running "without a symmetric game, competitive rewards, or online parameter updates" (L196–197), and the Main Agent is counted as one of five roles despite being "graph-control logic rather than an additional LLM call" (L205).
-
Computational overhead is not reported. Each synthesis invokes multiple LLM calls across roles and revision loops, yet the paper reports no revision counts, token usage, latency, or cost. The EXP setting thus uses extra test-time compute without a compute-matched baseline.
Questions for authors
Regarding Weakness 1-5, please answer the following:
- Could you provide ablation results for the synthesis pipeline?
- Could you compare IntentSci-Graph experimentally against other existing auto-research baselines and evaluate it on additional benchmarks beyond ASI-Bench?
- How many runs were conducted per task, and what are the corresponding variances or confidence intervals? Please also explain the calculation error in the B1 macro average of Table 1 and reconcile the discrepancies between Table 1, Table 2, and L362.
- Where are testable hypotheses H and evidence requirements E stored in the PlanDoc schema ?
- What is the average synthesis overhead per task in revision rounds, tokens, and time, and how does EXP compare against a compute-matched Raw baseline?