acceptodds
Under review as a conference paper at ICLR 2027

More Than Can Be Said: A Benchmark and Framework for Pre-Question Scientific Ideation

15 views

Abstract

AI research agents have shown strong potential in automating literature search and manuscript refinement, yet most operate only after a research question has already been made explicit, assuming a clear and actionable initial input. Human research, however, often starts earlier, with an inarticulate sense of friction between intuition and existing explanations, before any question can be posed. We introduce InciteResearch, a multi-agent framework that helps a researcher externalize this pre-question friction into an inspectable, revisable profile, and then uses that profile to reframe the underlying problem before drafting a proposal. Concretely, InciteResearch (1) elicits a five-dimensional researcher-profile representation, comprising an explicitly logged friction point together with motivation, constraints, preference, and a refined topic, through structured multi-turn dialogue; (2) identifies a hidden assumption underlying the initial framing by jointly scoring candidate assumptions on removability and reframing potential, and produces a reframed problem statement with a seven-stage causal derivation trace linking assumption, insight, claim, prediction, and method; and (3) checks whether each proposed method component is a logically required consequence of that derivation rather than a plausible add-on. We introduce TF-Bench to evaluate InciteResearch. TF-Bench is a benchmark for tacit-to-explicit research assistance spanning prediction, discovery, attribution, causality four scientific modes and domain-related, domain-unrelated two ambiguity regimes, now comprising 120 inspiration examples. Relative to a matched AgentLaboratory and prompt-based baselines, InciteResearch improves mean novelty and impact from 3.617 / 3.810 and 3.695 / 3.827 to 4.221 / 4.368, respectively, showing consistent gains across all four domains, which are further corroborated by expert evaluations validating the reliability of the judges. These results support assumption-breaking dialogue as a measurable, specific contributor to proposal quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scientific ideation, research agents, tacit knowledge elicitation, assumption breaking, hypothesis generation, LLM-as-a-judge, human-AI collaboration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kvEUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces InciteResearch, a multi-agent framework for transforming vague research intuitions into structured scientific proposals through Socratic Elicitation, assumption-breaking reframing, and Necessity Checking. It also introduces TF-Bench, containing 120 inspiration examples across four scientific domains. Experiments demonstrate improvements in novelty and impact over prompt-based and AgentLaboratory baselines, supported by component ablations and LLM–human judge agreement analysis.

Critical strengths
  1. The paper addresses pre-question scientific ideation, a relatively underexplored stage of AI-assisted research. The EVN framework provides an interpretable approach to transforming implicit intuitions into structured proposals.
  2. TF-Bench covers four scientific domains and two ambiguity regimes. InciteResearch consistently improves novelty and impact over the evaluated baselines. The ablation studies further support the importance of the E and V operators.
Critical weaknesses
  1. Limited ecological validity of the benchmark. TF-Bench primarily contains LLM-generated, human-revised inspiration examples rather than authentic interactions with researchers.
  2. Insufficient validation of proposal quality and methodological contributions. The main improvements rely on LLM-as-a-Judge evaluations, while human experiments primarily establish judge agreement rather than independently demonstrating proposal-quality gains.
Questions for authors
  1. Are any TF-Bench examples derived from authentic researcher interactions rather than LLM-generated inspirations?
  2. Can the authors report direct human-expert comparisons of proposal novelty, feasibility, and impact using the existing expert evaluations?
  3. Were inference budgets and model calls matched across the proposed framework and baselines? How much improvement can be attributed specifically to the EVN design?
Reviewer_9KYxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes InciteResearch, a multi-agent framework that handles pre-question scientific ideation. The framework is called EVN, including socratic elicitation, validity and problem reframing, and necessity checking. Also, TF-Bench benchmark is proposed to evaluate the pre-question scientific ideation. The InciteResearch showed superiority compared to the prompt-based LLM.

Critical strengths

None

Critical weaknesses
  • Human evaluation was conducted, but the comparison scores are not reported: Appendix C presents the human evaluation protocol, but there are no comparison scores (i.e., whether the experts rated IncitResearch above the baseline). All main experimental results (Tables 2, 3) are based on LLM judges. However, LLM judge alignment with experts is much lower than human-human alignment. Reporting the comparison of expert scores would be helpful.
  • No real researcher is involved in the evaluation: The paper is motivated by helping a researcher express a feeling they cannot put into words, and the E stage is described as a multi-round dialogue with that researcher. However, all TF-Bench inputs were written by an LLM, and the evaluation replaces the researcher’s replies with two fixed paragraphs. Without a study involving real researchers, the paper does not show that pre-question assistance addresses a real need.
Questions for authors

Why did the authors never reported the comparison scores of human evaluation?

Reviewer_ZbDEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Research often starts with vague intuitions or dissatisfaction with existing methods. This paper proposes InciteResearch, a multi-agent framework that turns such inputs into structured research proposals. It first uses a two-round dialogue to identify the researcher’s concerns, motivation, constraints, preferences, and topic. It then identifies a hidden assumption to reframe the problem, and builds a reasoning chain connecting the problem to method design. A reviewer agent checks whether the proposed method is justified by this reasoning. The authors also introduce TF-Bench, a benchmark of 120 inspiration examples across four scientific domains, generated by an LLM and revised by humans. Evaluation uses three LLM judges, with expert assessments on a subset. Compared with prompt-based generation and AgentLaboratory, the method receives higher novelty and potential impact scores but lower feasibility scores.

Critical strengths
  1. A well-motivated research question with a concrete framework to address it. The paper studies how AI can help researchers turn vague intuitions into research questions, addressing an important stage before the explicit task formulation often assumed by scientific agents.
  2. A concrete framework and an initial benchmark to address it. The EVN pipeline (elicitation, assumption violation, necessity checking) decomposes the task into explicit steps, each producing an auditable intermediate output. The authors also construct TF-Bench filling a gap left by existing datasets that presuppose well-formed research questions.
Critical weaknesses
  1. Mismatch between the intended task and the evaluation setting. The paper aims to help researchers clarify vague intuitions through dialogue, but TF-Bench uses two prewritten input paragraphs generated by an LLM and revised by humans. The evaluation does not directly assess whether the system’s questions elicit useful new information or whether the resulting research questions reflect the researcher’s intentions. This provides limited evidence for the claimed benefit of dialogue in turning initially vague ideas into clear research questions.
  2. The intended functions of the modules are insufficiently validated. The ablations show changes in final proposal scores, but do not establish whether E faithfully extracts user intentions, V identifies valid assumptions worth challenging, or N detects and corrects reasoning errors.
Questions for authors
  1. Benchmark construction. Could you provide the prompts used to generate TF-Bench, the criteria for selecting examples, and details of the human revision process? How did you check whether the final examples reflect the vague ideas researchers have before forming a clear research question?
  2. Elicitation in TF-Bench. How is the second prewritten paragraph used to answer the follow-up questions generated by the E stage? Could you include a small evaluation with researchers to check whether these questions help clarify their initial ideas and whether the resulting research questions reflect their intentions?
  3. A complete scientific case. Could you provide an end-to-end example showing the initial input, E’s researcher profile, V’s assumption and reframing, and N’s review?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.