acceptodds
Withdrawn from ICLR 2027

Arxidea: An Agentic System for Personalized Research Ideation and Evidence-Backed Verification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language-model systems can propose research ideas at scale, but they leave two gaps for individual researchers. The proposals are rarely conditioned on the re- searcher’s own publication record or on papers appearing that week, and the result- ing ideas are typically handed over without verification. Existing systems often evaluate personalization using simulated users or model-based judges rather than the intended researcher. We present ARXIDEA, a personalized, human-in-the- loop agentic system that closes these gaps. Its four stages are Recommendation, Ideation, Desk Check, and Experiment Check. Recommendation ranks the week’s relevant papers using a profile derived from the researcher’s publications. Ideation proposes ideas from them. The two checks form a two-level idea checker that re- turns an evidence-backed feasibility verdict. We evaluate relevance directly with the profiled researcher. We construct a week-sized evaluation pool by sampling 2,574 new papers uniformly across twelve weeks of arXiv, and the researcher grades every paper for relevance. The resulting pipeline returns 100 papers of which 84 carry the researcher’s top relevance grade, spread across subject areas. Adding a reranker over the retrieved candidates reduces recommendation quality according to the researcher’s grades. A frontier agent (Claude Opus 5.5) that reads the week-sized sample in place of retrieval only matches it while spending 15 to 29 million tokens per week. Retrieval and ranking therefore operate without a gen- erative model. In the first level, the Desk Check, every candidate idea undergoes premise, theory, and novelty checks, after which it is retained with supporting ev- idence or eliminated with a recorded reason. The idea the researcher selects then enters the second level, the Experiment Check, a bounded experiment–review loop whose persistent memory is a versioned research specification. The loop records its verdict and hands surviving ideas back to the researcher with findings and next steps. We report three runs—one that passes, one that fails, and one that needs the researcher’s input—that expose the checker’s verdicts and supporting evidence. We will release the code, prompts, and researcher-provided relevance grades.

Primary area:
foundation or frontier models, including LLMs
Keywords:
personalized research ideation, research agents, literature retrieval, idea verification, agentic AI, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ARRYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Arxidea, a human-in-the-loop agentic system that takes one researcher's publication record and one week of arXiv postings and produces checked research ideas. It has four stages, coordinated by orchestrator agents through a file-based memory.

Critical strengths
  1. A fully graded, researcher-specific evaluation pool, used carefully. Most personalization studies in scholarly recommendation rely on clicks, citation proxies or simulated users. Here the profiled researcher graded every paper in a week-sized uniform sample (2,574 papers), including those no method retrieved. This allows real recall to be computed and every design to be compared on equal terms (l. 237–248). The analysis is careful:

    • selection among 1,799 configurations uses a fixed rule and is reported cross-fitted across odd and even weeks (l. 305–312);
    • bootstrap intervals are paired;
    • profile builds are repeated five times;
    • grade consistency is checked with 90 duplicates.

    The resulting negative finding is informative for practitioners. A cross-encoder and two LLM rerankers each make the list worse, replacing about 40 relevant papers with 21–24 relevant ones (l. 369–377, Table 3), and LLM runs are unstable (66–70 of 100 picks shared). Releasing the pool and grades would be a useful resource.

  2. A design that records evidence and separates roles. The system is organized so that no agent judges its own work:

    • numbers enter the specification only through a program-maintained registry;
    • a model-free program replays tests and searches before a verdict is recorded (l. 208–210, 225–226);
    • eliminated ideas keep their recorded reasons.

    Given documented failure modes of autonomous research agents (model judges rating weak proposals favorably, fabricated data, component substitution; l. 100–103), this is a sensible architecture, and the three runs show it can return negative and flagged verdicts, not only passes.

Critical weaknesses
  1. The paper's main claims, personalized ideation and evidence-backed verification, are not evaluated in a way that supports them.

    • The ideas are never assessed by the researcher. The abstract criticizes prior work for evaluating personalization "using simulated users or model-based judges rather than the intended researcher" (l. 16–18). Yet the ideas are evaluated only by a model-based judge (GPT-6 Sol; §5.1, App. E). R1 graded all 2,574 papers but none of the 79 ideas. There is no evidence that the ideas are personalized (relevant to R1's threads), novel or useful to R1, which is what the title claims.
    • The Desk Check evaluation is partly circular. The grader's case files include "registered premise checks, executed test outputs, a written derivation, and the closest existing work found by literature search" (l. 1118–1120). This is the same evidence on which the Desk Check based its decisions. Removing only the explicit verdict field does not make the grader independent: if a premise test failed or close prior work was found, both the Desk Check and the grader will downgrade the idea. The 0.79-point gap may therefore show consistency between two readers of the same evidence rather than correct filtering. A grader that sees only the idea text (and seed abstract), or a human expert blind to the evidence, is needed.
    • No baselines and weak filtering. There is no comparison with simpler filters: a single-call LLM critique, self-refinement, random elimination at the same rate, or the system without the replay program. Even under the model grader, 40% of retained ideas are Fatal or Flawed and only 13% are Good or better (Fig. 5). False negatives (good ideas eliminated, 18% graded Fair or better) are not analyzed.
    • The Experiment Check is shown, not evaluated. It is supported by one review round from each of three runs (§5.2, App. F), described in paraphrase. There is no measure of whether pass, fail or flag verdicts are correct, how often the auditors catch numerical errors, or how reviewer votes relate to expert judgment. The cost is also substantial: 203M and 748M tokens for the fail and flag runs (Table 10), with no comparison to the researcher's own time.
    • Missing related work on idea evaluation. The paper does not engage with large-scale human evaluations of LLM-generated research ideas, e.g., Si et al. (ICLR 2025, 100+ NLP researchers judging novelty and feasibility), or with ideation systems that ground ideas in retrieved literature, e.g., ResearchAgent (Baek et al., 2025), Scideator and Chain of Ideas. These are the natural baselines and evaluation protocols for §5.
  2. The recommendation evaluation rests on a single researcher, and parts of it are built into the grades.

    • N = 1. All configuration selection and the headline numbers (84/100, the reranker result) come from R1's grades. R1 appears closely tied to the project: R1 confirmed the profile, answered Experiment Check points and asked for four papers to be omitted from the judge's copy (l. 999–1000, 506). If R1 is an author, the grades are not independent of system design. The paper should state this.
    • Some grades are assigned by a category rule that matches a deployed signal. 380 of the 2,574 grades (15%) are set to 0 by a rule based on arXiv primary categories (l. 746–750). The deployed category signal ranks papers by the same arXiv categories (l. 170–171). This builds part of the category signal's success into the ground truth. It may also explain why the category signal is among the most useful additions (Fig. 3) and why random baselines are low.
    • The comparison with the reading agent is overstated. The paper says the agent "only matches" retrieval (abstract, l. 28–29; l. 468–469). On R1's own grades, the one-sentence agent's list holds 91 top-grade papers against retrieval's 84, with mean grade 3.89 vs. 3.78 (Table 7). For R2, retrieval is significantly worse than both agent lists (−0.44 and −0.45 mean grade; Table 6), and for R3 it is worse than the profile-input agent (−0.16). Evidence for R2 and R3 also comes only from an LLM judge with Spearman 0.55 against R1 (l. 1022–1024). This judge counts 22 of 176 papers as worth reading where R1 counts 151 (l. 1075–1076), and it agrees with itself only 65–74% of the time across sessions (l. 1028–1029). The conclusion that "ranking the week therefore calls no language model" is a cost argument, not a quality result, and should be stated as such.
    • Scope. All three researchers are in applied mathematics, and the Experiment Check runs are cheap statistical studies (uncertainty quantification, survival analysis, adaptive testing). Whether the design works for fields where experiments need significant compute or data is untested.
Questions for authors
  1. Researcher assessment of ideas. Did R1 (or other researchers) assess the candidate ideas, retained and eliminated, for relevance, novelty and value? If not, can you collect such grades, ideally blind to the Desk Check verdict, and report how the retained/eliminated split looks under them?
  2. Grader independence. Can you re-run the §5.1 grading with case files that contain only the idea and seed abstract, without the Desk Check's premise outputs, derivations and closest-work search? How much of the 0.79-point gap remains?
  3. Baselines for the checker. How does the Desk Check compare with a single LLM critique call, or with a self-refine baseline, at the same elimination rate? What does the model-free replay program catch on its own?
  4. Relationship of R1 to the authors. Is R1 an author or otherwise involved in the system's design? Did R1 know the system's signals when grading?
  5. Rule-assigned grades. How do the results in §4.2 and Fig. 3 change if the 380 rule-assigned grades are excluded, or graded individually? In particular, how much of the category signal's contribution remains?
  6. Experiment Check correctness. For the three runs, can you provide the full specifications, registries and reviewer outputs? Did a human expert check whether the verdicts were right? How often did auditors find mismatched numbers across all runs, not just these three?
  7. Personalization of ideas. How much do the generated ideas depend on the profile? E.g., if the same seed papers are given with a different researcher's profile, do the ideas change, and does R1 prefer the ones made with their own profile?
Reviewer_DuQpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ARXIDEA is a human-in-the-loop agentic system intended to personalize weekly research intake and turn newly posted arXiv papers into checked research ideas using a researcher’s publication record. I particularly like the topic and find the research direction compelling. However, I strongly encourage the authors to devote more attention to experimental design and provide more rigorous evidence to support their central claims.

Critical strengths

Important and practical research problem. The paper addresses two meaningful limitations of existing AI research systems: the lack of personalization to individual researchers and the limited verification of generated research ideas.

Interesting system design. Arxidea integrates personalized paper recommendation, idea generation, preliminary verification, and experimental validation into a coherent human-in-the-loop workflow. The separation between idea generation and independent verification is particularly valuable.

Critical weaknesses

Insufficient evaluation of the central contribution. The experimental evaluation is heavily concentrated on paper recommendation, while the proposed evidence-backed verification pipeline receives much less rigorous validation. Desk Check is evaluated on 79 ideas, and Experiment Check is demonstrated through only three cases. These experiments are insufficient to establish whether Arxidea can reliably identify and verify promising research ideas.

Missing baselines and ablation studies for the verification pipeline. The paper does not compare its full verification pipeline against simpler alternatives, such as direct LLM-based idea evaluation, Desk Check alone, or experimental evaluation without multi-agent review.

Questions for authors

Can the authors provide evaluations by independent domain experts to establish whether Desk Check retains scientifically sound and novel ideas while rejecting invalid ones?

What is the incremental benefit of each verification component? How does the full pipeline compare with a single strong LLM reviewer, Desk Check alone, or an experiment-based pipeline without the reviewer panel?

Can the authors provide end-to-end evaluations involving multiple real researchers, measuring idea quality, research time saved, and the additional scientific value relative to the computational cost?

Reviewer_uZctOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose ARXIDEA, a personalized, human-in-the-loop agentic system for automated research idea generation and evidence-based verification. The system consists of four stages: (1) Paper Recommendation: Given a researcher profile built from their published papers, the system retrieves and recommends recent arXiv papers. (2) Ideation: The system then generates research ideas grounded in the selected papers. (3) Desk Check: The system evaluates the premises, theoretical soundness, and novelty of these ideas for quality assessment. (4) Experiment Check: For research ideas selected by the researcher from those that pass the Desk Check, the system conducts experiments and uses independent review agents to assess their feasibility. To validate the system, the authors evaluate the recommendation stage using 2,574 arXiv papers graded by a researcher, reporting that 84 of the top 100 recommended papers received the highest relevance grade and that conventional retrieval outperformed LLM-based reranking. For idea verification, the Desk Check eliminated 49 of 79 candidate ideas, and the results show that retained ideas received higher average quality scores from an external LLM judge. In conclusion, this work provides a personalized and traceable workflow for generating and evaluating research ideas.

Critical strengths
  1. This paper provides a human-grounded evaluation of its recommendation module, using relevance judgments provided by the target researcher. This provides a more direct assessment of personalized recommendation quality than relying solely on LLM-based evaluations.
  2. This paper also presents a concrete research idea verification mechanism. It introduces a two-level verification process that takes multiple aspects into consideration, such as theoretical soundness, novelty, and experimental feasibility, addressing the important challenge of ensuring the reliability of LLM-generated research ideas.
Critical weaknesses
  1. Limited Technical Novelty. The framework integrates four components for automated research, but their differences from existing methods and the new capabilities enabled by their integration are not clearly established. The contribution beyond combining paper recommendation, ideation, and agentic verification remains unclear.

  2. Insufficient End-to-End Validation. Most quantitative experiments focus on paper recommendation, leaving limited evidence that the complete system improves the quality, novelty, and feasibility of research ideas. Desk Check relies on an external LLM judge without independent expert assessment, and the judge’s access to system-generated verification records may introduce bias. Experiment Check is demonstrated through only three case studies. Strong system-level baselines and component ablations are also missing, making the benefits of the two-level verification mechanism and independent review agents unclear.

Questions for authors
  1. Given the reliance on LLM judges in Section 5.1 and Appendix D, and the reported Spearman correlation of only 0.55 with human ratings, what evidence supports the reliability of these evaluations, particularly the conclusions about the effectiveness of the Desk Check?
  2. Can the authors provide quantitative evidence that the complete ARXIDEA pipeline improves research idea quality or verification reliability compared with simpler baselines, rather than only demonstrating the effectiveness of its recommendation module?
  3. In the Experiment Check's pass case, only one of the four predefined claims is satisfied, yet the idea still passes verification. How do the authors justify this outcome as evidence of successful idea verification, rather than merely reflecting the review agents' acceptance decisions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.