acceptodds
Under review as a conference paper at ICLR 2027

IntentSci-Graph: A Scientific Reasoning Layer for Autonomous Research Agents

Abstract

Autonomous research agents are increasingly capable of executing scientific workflows, but reliable scientific discovery cannot be reduced to execution capability alone. Reliable scientific discovery also depends on whether an agent can reason over an underspecified research intent, organize a scientifically coherent plan, and determine what evidence is needed to support a conclusion. Existing research agents can organize this reasoning through planning and execution workflows, but the resulting organization is often incomplete or insufficiently verified. A method may not be clearly connected to the evidence required for the intended claim, or the final outputs may not be aligned with the criteria used to validate the result. To address this gap, we introduce IntentSci-Graph, a scientific reasoning layer for autonomous research. IntentSci-Graph casts autonomous research as a pre-execution scientific program synthesis problem, transforming research intent into an explicit and verifiable scientific program. Rather than asking a downstream executor to infer scientific structure while acting, IntentSci-Graph produces a structured interface that links research objectives, methods, required evidence, and validation criteria. The program is refined through structured synthesis and verification, making the intended study inspectable before execution and reusable across different downstream agents. On ASI-Bench, IntentSci-Graph effectively improves autonomous research performance, particularly under underspecified research prompts, demonstrating that explicit and verifiable scientific programs can provide a practical foundation for reliable autonomous research.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Agent Self-Evolution, Auto Research, Human–Agent Co-Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_6DmBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies autonomous research agents under ambiguous or underspecified user prompts, where planning often disconnects from execution and scientific validation criteria are missing. The authors introduce IntentSci-Graph, a training-free framework for pre-execution scientific program synthesis. Five asymmetric roles, the Main Agent, Expert, Critic, Validator, and Condenser, interact over a shared state, with the Main Agent implemented as non-LLM control logic that coordinates the other four roles. Before the downstream executor runs, these roles iteratively draft, critique, and verify the original research request (p, W, C) and condense it into an explicit program interface saved as EXP.md. Paired experiments on the 60 project-level tasks of ASI-Bench evaluate four prompt completeness levels from B1 to B4 across four executor and model combinations. While EXP.md reduces degradation on the underspecified B3 and B4 levels and reaches an average gain of +7.00 on B4, it consistently hurts performance on B1 where complete expert procedures are already given, with a reported macro average change of −7.92. Averaged across all 16 configuration and level cells, the net gain is only +0.21.

Critical strengths

The main strengths are:

  1. Important research problem. How autonomous research agents should handle ambiguous or underspecified research intent is an important and timely question, and it becomes increasingly relevant as these agents are applied to realistic scientific tasks.

  2. Clear presentation. The paper is well written and easy to follow, and its figures convey the overall pipeline and the motivating case intuitively.

  3. Intuitive core idea. Making the scientific plan explicit before execution, rather than leaving the executor to infer it while acting, is a natural and well-motivated idea.

Critical weaknesses

The main weaknesses are:

  1. The necessity of the multi-role design is not demonstrated. The paper attributes its gains to "self-play-inspired role-based deliberation" (L194–195) but provides no ablation studies for any module.

  2. The experimental evaluation is narrow in both benchmarks and baselines. All experiments rely on a single 60-task benchmark, despite citing other benchmarks such as ResearchClawBench by Xu et al. 2026. Moreover, the raw prompt is the only baseline compared. None of the "Auto-research Baselines" in Figure 1, such as AI-Researcher by Tang et al. 2025, is evaluated in Sec. 4.

  3. Gains are weak and unstable, and the reported numbers contain internal contradictions. The paper notes that "the average over all 16 configuration–level cells changes only from 23.12 to 23.33 (+0.21)" (L322–323), which does not support the abstract's claim that IntentSci-Graph "effectively improves autonomous research performance" (L027–028). Every configuration drops on B1, B2 shows mixed signs, and CodeBuddy with GLM-5.3-Flash drops by 3.89 on B3 and averages −2.45 overall. Although L312 defines a run index s, the number of runs per task and variances are never reported.
    Several numbers in Sec. 4 also contradict one another. First, averaging the four B1 rows in Table 1 yields 38.29, 30.27, and −8.02 rather than the reported 38.92, 31.00, and −7.92, and the downstream figures in Sec. 4.2, including 25.80, 10.88, 14.92, and 23.12, all rely on that incorrect 38.92. Second, L362 states "All four B1 differences are negative (−7.77, −7.09, −11.25, and −5.55)," whereas Table 1 lists −8.17 for CodeBuddy with DeepSeek. Third, Table 2 uses the same configuration, but its B2–B4 task-weighted means disagree with Table 1, such as 17.80 / 28.69 versus 18.67 / 29.56 on B4.

  4. The terminology overstates the actual implementation. The formulation S = (O, H, M, E, V, X) includes "testable hypotheses H" and "evidence requirements E" (L152–156), yet the Expert's PlanDoc at L230–233 has no fields for either. Similarly, the pipeline is called self-play-inspired despite running "without a symmetric game, competitive rewards, or online parameter updates" (L196–197), and the Main Agent is counted as one of five roles despite being "graph-control logic rather than an additional LLM call" (L205).

  5. Computational overhead is not reported. Each synthesis invokes multiple LLM calls across roles and revision loops, yet the paper reports no revision counts, token usage, latency, or cost. The EXP setting thus uses extra test-time compute without a compute-matched baseline.

Questions for authors

Regarding Weakness 1-5, please answer the following:

  • Could you provide ablation results for the synthesis pipeline?
  • Could you compare IntentSci-Graph experimentally against other existing auto-research baselines and evaluate it on additional benchmarks beyond ASI-Bench?
  • How many runs were conducted per task, and what are the corresponding variances or confidence intervals? Please also explain the calculation error in the B1 macro average of Table 1 and reconcile the discrepancies between Table 1, Table 2, and L362.
  • Where are testable hypotheses H and evidence requirements E stored in the PlanDoc schema ?
  • What is the average synthesis overhead per task in revision rounds, tokens, and time, and how does EXP compare against a compute-matched Raw baseline?
Reviewer_mrGaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces IntentSci-Graph, a framework that makes underspecified research request into a strctured scientific programs before execution. This framework designs multiple roles, including Expert, Critic, Validator, and main agent to make it a multi-agent system. The resulting program is then passed to downstream research agents.

Critical strengths
  1. practical design: separating scientific program synthesis with downstream execution is a reasonable architecture design.

  2. evaluation across different information levels: The experiments shows how the framework performs under varying amount of methodology guidance. This is a useful design to show and analyze the performance.

Critical weaknesses
  1. malformed citations: Some in-text citations, such as those in Lines 58 and 73, are not rendered correctly.

  2. limited imporvement: The overall improvement is only 0.21 points. The largest improvement occurs on B4 while B3 shows a much smaller gain. It is unclear whether the improvement comes from method selection or simply more effective filtering and restructuring. Need more ablation studies on it.

  3. lack of baselines: The paper lacks comparisons with competitive existing research agents. It should compare with AutoResearchClaw, The AI-Scientist v2 and AI-researcher and AgentLaboratory. The proposed framework shares substantial similarities with existing multi-agent research systems, but the paper does not sufficiently demonstrate its advantages over prior approaches. Comparisons with relevant research-agent or planning baselines under comparable experimental settings are needed to establish its novelty and effectiveness.

  4. lack of ablations: There are no ablation studies on the functions of multi-agent design. For example, deleting critic or validator, how much points would it drop. If they do not use the shared state, how much would it drop?

Questions for authors

Can you compare IntentSci-Graph against a single agent with structured planner or prompt-rewriting baseline?

Does IntentSci-Graph improve the actual success rate of scientific research tasks, or does it mainly improve partial scores without increasing the number of successfully completed tasks?

Reviewer_YQdTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents IntentSci-Graph, a scientific reasoning layer for autonomous research. It is meant to be used before calling a general-purpose autoresearch system like Codex or CodeBuddy. Intentsci-Graph organizes an underspecified research intent into an explicit "scientific program" (a structured representation) that connects the research objective with the intended methodology, required evidence, validation criteria, and execution contract. The goal is to make the intended study inspectable before execution and reusable across different downstream agents.

The paper evaluates IntentSci-Graph on ASI-Bench with three different executor architectures (i.e. autoresearch systems) --- Codex, CodeBuddy, ReAct. ASI-Bench provides a way to evaluate scientific program synthesis on progressively underspecified initial research prompts.

IntentSci-Graph is found to improve autoresearcher performance, particularly under underspecified research prompts.

Critical strengths

I like the underlying idea behind the work --- recasting the specification of autonomous research as a scientific program synthesis problem. It allows the formal specification of research intent as an explicit "scientific program" that can be automatically modified and verified. (The paper really needs to define "scientific program" at the beginning of the intro somewhere.)

Critical weaknesses

In spite of the formal specification of research intent that IntentSci-Graph allows, the paper does not make clear or concrete how the system operates. In the end, I think that all of the components are implemented as natural language prompts to an LLM. So many of the potential benefits of enforcing a formal structure, e.g. employing formal verification techniques to validate the "program", are lost.

Ultimately, not enough information was provided on how everything works underneath for me to believe that the complex tasks described in the stated contributions are really being accomplished. It's really hard to get a sense of the what the inputs and outputs look like at each step or how the various, quite complex, tasks associated with each role were actually performed. I realize that the case study (section 4.4) along with Figure 3 are supposed to help with this but they didn't. I didn't see in the case study, for example, how "the [execution] steps connect input parsing and period search to harmonic regression, parameter estimation, and file generation". The steps do provide an ordered list but don't at all make the relationships among the listed elements explicit. How the role-playing modules "jointly organize and verify the relationships among scientific objectives, methods, evidence, validation criteria, and execution requirements" (108-110) was particularly unclear.

Questions for authors
  1. How are the components of the role-playing system implemented? Is each a fixed prompt to an LLM? I wanted to see this information at the beginning of section 3 as well as pointers to an appendix that showed the prompts.

  2. How realistic are the ASI-BENCH tasks? How feasible is it for real users to specify tasks in their required format?

  3. ESP, EXP: these are used without defining?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.